Compiling NWChem

Today I compiled NWChem on my IBM Intel InfiniBand cluster.  I have no idea if my performance is optimal, but I do know that it works.

Here is the secret to my success:

First, you have to set some path variables to get things running.  This is set in my home directory:

cat ~/.bashrc

INTELCCROOT=/home/appls/compilers/intel/11.0/083
INTELFCROOT=/home/appls/compilers/intel/11.0/081

PGROOT=/home/appls/compilers/pgi
PGCC=$PGROOT/linux86-64/8.0-5/bin
PGFLEXLM=$PGROOT/linux86-64/8.0/bin
LM_LICENSE_FILE=$LM_LICENSE_FILE:$PGROOT/license.dat

#MPI_HOME=/home/appls/openmpi/gcc
#MPI_HOME=/home/appls/openmpi/pgi
MPI_HOME=/home/appls/openmpi/intel

NWCHEM=/home/appls/QChem/NWChem/intel/bin

PATH=$PATH:$MPI_HOME/bin:$HOME/bin:$INTELCCROOT/bin/intel64:$INTELFCROOT/bin/intel64:$PGCC:$PGFLEXLM:$NWCHEM

LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$MPI_HOME/lib:$INTELCCROOT/lib/intel64:$INTELFCROOT/lib/intel64

Unfortunately, I’m leaving out a lot of the details here, but I used the Intel compilers and had previously built openmpi.  The way I built openmpi was:

./configure --prefix /home/appls/openmpi/intel CC=icc CXX=icpc F77=ifort FC=ifort

make -j8

make install

S
http://wiki.cse.ucdavis.edu/support:hpc:software:nwchem

http://www.mcsr.olemiss.edu/appssubpage.php?pagename=nwchem.inc

So, after I got that set up, I did the normal:

tar zxvf nwchem-5.1.tar.gz

Then I made a script that pretty much did all the work.  The script is called makeit.sh:

export TCGRSH=/usr/bin/ssh
export NWCHEM_TOP=/home/vallard/qchem/nwchem-5.1/
export NWCHEM_TARGET=LINUX64
export USE_MPI=y
export USE_MPIF=y
export MPI_LOC=/home/appls/openmpi/intel
export MPI_LIB=$MPI_LOC/lib
export LIBMPI="-L $MPI_LIB -lmpi -lopen-pal -lopen-rte -lmpi_f90 -lmpi_f77"
export MPI_INCLUDE=$MPI_LOC/include
export ARMCI_NETWORK=OPENIB
export LARGE_FILES
export NWCHEM_MODULES=all
export FC=ifort
export CC=icc
cd $NWCHEM_TOP/src
make CC=icc FC=ifort -j4

After kicking that off, it ran for almost 20 minutes compiling!  Forever!  I saw a lot of vector loop messages, but gave them no head, and fearlessly pressed forward.

After it was done compiling as root, I did the following:

[root@mgt vallard]# export NWCHEM_TOP=/home/vallard/qchem/nwchem-5.1/
[root@mgt vallard]# mkdir $NWCHEM/bin
[root@mgt vallard]# mkdir $NWCHEM/data
[root@mgt vallard]# cp /home/vallard/qchem/nwchem-5.1/bin/LINUX64/nwchem $NWCHEM/bin
^[[root@mgt vallard]# cp /home/vallard/qchem/nwchem-5.1/bin/LINUX64/depend.x $NWEM/bin/
[root@mgt vallard]# cd $NWCHEM_TOP/src/basis 
[root@mgt basis]# cp -r libraries $NWCHEM/data/
[root@mgt basis]# cd $NWCHEM_TOP/src/
[root@mgt src]# cp -r data $NWCHEM
[root@mgt src]# cd $NWCHEM_TOP/src/nwpw/libraryps
[root@mgt libraryps]# cp -r pspw_default  $NWCHEM/data/
[root@mgt libraryps]# cp -r paw_default/  $NWCHEM/data/
[root@mgt libraryps]# cp -r TM  $NWCHEM/data/
[root@mgt libraryps]# cp -r HGH_LDA  $NWCHEM/data/

That got everything in place.  When done, I went to my compute node and tried one of the examples:

cd /home/vallard/qchem/nwchem-5.1/examples/dirdyvtst/h3
mpirun -np 32 -machinefile machinefile nwchem h3tr2.nw

After that it all seemed to work.  Any optimization info would be great!  Thanks

  • http://www.open-mpi.org/ Jeff Squyres

    FWIW, you can set FC and CC to mpif77 and mpicc (respectively) in your makeit.sh and let Open MPI’s wrappers do the majority of the work for you (they specify many of the flags that you specified already — in particular, the values in LIBMPI may change in any future version of Open MPI — using mpifcc and mpif77 instead of setting LIBMPI will isolate/protect you from such changes).

    Good luck!

  • Anonymous

    Test my comments.

  • Pingback: Installing NWChem with OpenMPI, Intel Compilers and MKL and CentOS 5.x « Linux Cluster