I have a cluster where GPFS is running over the Gb Ethernet. To make it so that GPFS goes over InfiniBand Ido the following:
mmshutdown -a
mmchnode --daemon-interface=compute001-ib0 -N compute001
That did it for one node. To do it for all nodes:
for i in $(nodels compute); do mmchnode --daemon-interface=$i-ib0 -N $i; done
Alternatively, I could have just made a spec file called /tmp/foo with the contents:
s01 --daemon-interface=s01-ib0
s02 --daemon-interface=s02-ib0
Then run:
mmchnode -S /tmp/foo
Once finished, I run mmlscluster and see that all the nodes that I wanted are now communicating over the InfiniBand. GPFS is so easy!
Then, start things back up:
mmstartup -a
And you're off to the races.