Thang's Notes: Compiling OpenMPI 4

https://www.open-mpi.org/faq/?category=building
NOTE:
- Some applications require C++11, this is only supported on GCC 4.8 or newer, which is not always available on system, then newer GCC need to be installed before compiling Openmpi
- Make sure to build OpenMPI with 64-bit support. To check whether the currently available
OpenMPI do support 64-bit or not, type this:
ompi_info -a | grep 'Fort integer size'

If the output is 8, then it supports 64-bit. If output is 4, then it just supports 32-bit.* configuration for 64-bit support:

+ For Intel compilers use:
FFLAGS=-i8 FCFLAGS=-i8 CFLAGS=-m64 CXXFLAGS=-m64
+ For GNU compilers type:
FFLAGS="-m64 -fdefault-integer-8" FCFLAGS="-m64 -fdefault-integer-8" CFLAGS=-m64 CXXFLAGS=-m64

- must keep the source after compiling

Install UCX and libfabric:

https://www.blogger.com/blog/post/edit/4362825388287927017/5648027713769097400?hl=en

################

download OpenMPI

Installation OPTIONS in README.txt

or ./configure -h

tar xvzf openmpi-4.0.2.tar.gz
cd openmpi-4.0.2

mkdir build
cd build

I. OpenMPI-4.0.4 + Intel-2020xe (USC)

Note: Intel2019 on cluster has wrong path, cannot work

- "--with-verbs" to use InfiniBand

- Use infiniband may limit the number of nodes?

-without-verbs/--without-ucx... can be turned at run-time:

mpirun --mca btl ^tcp,openib -np 4 a.out

export OMPI_MCA_btl=^tcp,openib

export OMPI_MCA_btl=^tcp

So just compile the whole thing.

#--

# USC1

module load intel/compiler-xe19u5
module load compiler/gcc/9.1.0

check: icpc -v

Configure OpenMPI

../configure CC=icc CXX=icpc FC=ifort F77=ifort \
--with-sge --with-verbs --without-ucx --without-cma \
--prefix=/uhome/p001cao/local/app/openmpi/4.0.2-intelxe19u5-IB

make -j 8

make install

3. Test openMPI

the only thing that users need to do to use Open MPI is ensure that:

<prefix>/bin is in their PATH, and
<prefix>/lib is in their LD_LIBRARY_PATH.
Users may need to ensure to set the PATH and LD_LIBRARY_PATH in their shell setup files (e.g., .bashrc, .cshrc) so that non-interactive rsh/ssh-based logins will be able to find the Open MPI executables.

Create Module file:
.............

TEST: mpic++ -v

4. Running:
mpirun -np 2 ./app

Ref:
[1]. https://github.com/openucx/ucx/wiki/OpenMPI-and-OpenSHMEM-installation-with-UCX
[2]. https://software.intel.com/en-us/articles/performance-tools-for-software-developers-building-open-mpi-with-the-intel-compilers
[3]. Compiling OpenMPI 1.6.5 with Intel 12.1.5 on CentOS 6
[4]. http://www.diracprogram.org/doc/release-18/installation/int64/mpi64.html

#2.2. USC 2:

# use linker lld (include in Intel-bin, require GLIBC >2.15)

module load compiler/gcc-10.1.0

module load intel/compiler-xe19u5 # lld

#--

export myUCX=/home1/p001cao/local/app/tool_dev/ucx-1.8-intel

../configure CC=icc CXX=icpc FC=ifort F77=ifort LDFLAGS="-fuse-ld=lld -lrt" \
--with-sge --without-verbs --with-ucx=${myUCX} \
--prefix=/home1/p001cao/local/app/openmpi/4.0.4-intelxe19u5

## consider link to lib

export myIntel=/home1/p001cao/local/app/intel/xe19u5/compilers_and_libraries_2019.5.281/linux/compiler/lib

LDFLAGS="-L${myIntel}/intel64_lin -Wl,-rpath,${myIntel}/intel64_lin" \

##--

export PATH=/home1/p001cao/local/app/intel/xe19u5/compilers_and_libraries_2019.5.281/linux/bin/intel64:$PATH
export CC=icc export CXX=icpc export FORTRAN=ifort

II. install OpenMPI + GCC (USC)

NOTE:

* install libfabric, knem,... in UCX (openMPI 4.0,3 --> ucx-1.7 or older), or install with openMPI

* use UCX is recommended: --without-verbs

## consider lld linker:

module load llvm/llvm-gcc10-lld # to use lld

LDFLAGS="-fuse-ld=lld -lrt"

## gold linker:

module load tool_dev/binutils-2.32

LDFLAGS="-fuse-ld=gold -lrt"

Ref:
Q2: https://www.open-mpi.org/faq/?category=troubleshooting
https://www.open-mpi.org/faq/?category=building

https://raw.githubusercontent.com/open-mpi/ompi/v4.1.x/NEWS

###

https://raw.githubusercontent.com/open-mpi/ompi/v4.1.x/NEWS
tar xvf openmpi-4.1.1.tar.gz

cd openmpi-4.1.1

# 2.1. USC 1:

## 1. not use UCX

module load tool_dev/binutils-2.36 # gold, should use to avoid link-error

module load compiler/gcc-11.2
export myKNEM=/uhome/p001cao/local/app/tool_dev/knem-1.1.4

## IB cluster

mkdir build_eagle && cd build_eagle

../configure CC=gcc CXX=g++ FC=gfortran F77=gfortran LDFLAGS="-fuse-ld=gold -lrt" \
--with-sge --without-ucx --with-verbs --with-knem=${myKNEM} \
--prefix=/uhome/p001cao/local/app/openmpi/4.1.1-gcc11.2-noUCX-eagle

## noIB cluster

mkdir build_lion && cd build_lion

../configure CC=gcc CXX=g++ FC=gfortran F77=gfortran LDFLAGS="-fuse-ld=gold -lrt" \
--with-sge --without-ucx --without-verbs --with-knem=${myKNEM} \
--prefix=/uhome/p001cao/local/app/openmpi/4.1.1-gcc11.2-noUCX-lion

### 2. with ucx

# ucx Error: ib_md.c:329 UCX ERROR ibv_reg_mr(address=0x145cb580, length=263504, access=0xf) failed: Resource temporarily unavailable

## use the same procedure to compile on Lion and Eagle

module load tool_dev/binutils-2.35 # gold

module load compiler/gcc-10.2
export myUCX=/uhome/p001cao/local/app/tool_dev/ucx-1.9

export myKNEM=/uhome/p001cao/local/app/tool_dev/knem-1.1.4

../configure CC=gcc CXX=g++ FC=gfortran F77=gfortran LDFLAGS="-fuse-ld=gold -lrt" \
--with-sge --without-verbs --with-ucx=${myUCX} --with-knem=${myKNEM} \
--prefix=/uhome/p001cao/local/app/openmpi/4.1.1-gcc10.3-eagle

##2.2. USC 2:

## with UCX: on Tacheon, ucx gives better performance (but raise posibility of err)

cd openmpi-4.1.1

mkdir buildGCC && cd buildGCC

#--

module load tool_dev/binutils-2.35 # gold
module load compiler/gcc-10.3

export myUCX=/home1/p001cao/local/app/tool_dev/ucx-1.10 ## UCX

../configure CC=gcc CXX=g++ FC=gfortran F77=gfortran LDFLAGS="-fuse-ld=gold -lrt" \
--with-sge --without-verbs --with-ucx=${myUCX}  \
--prefix=/home1/p001cao/local/app/openmpi/4.1.1-gcc10.3

#not use

export myKNEM=/home1/p001cao/local/app/tool_dev/knem-1.1.4

--with-knem=${myKNEM}

## without UCX:

module load tool_dev/binutils-2.35 # gold
module load compiler/gcc-10.3

../configure CC=gcc CXX=g++ FC=gfortran F77=gfortran LDFLAGS="-fuse-ld=gold -lrt" \
--with-sge --with-verbs --without-ucx \
--prefix=/home1/p001cao/local/app/openmpi/4.1.1-gcc10.3-noUCX

##2.3. CAN:

module load gcc/gcc-7.4.0
check: g++ -v

2. Configuration

cd openmpi-4.0.2
mkdir build
cd build

../configure CC=gcc CXX=g++ FC=gfortran F77=gfortran \
--with-sge --without-verbs --without-ucx  \
--prefix=/home/thang/local/app/openmpi/4.0.2-gcc7.4.0

##2.4. CAN-GPU:

I. Install Cuda with Intel-xe19: (Runfile Installation)

https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html#redhat-installation

Download: wget http://developer.download.nvidia.com/compute/cuda/10.2/Prod/local_installers/cuda_10.2.89_440.33.01_rhel6.run

Install: (root acc)
1. disable the graphical target, to update Nvidia driver
systemctl isolate multi-user.target
modprobe -r nvidia-drm

module load compiler/gcc-7.4

sh cuda_10.2.89_440.33.01_rhel6.run --toolkitpath=/home/thang/local/app/cuda-10.2

2. after install Cuda, start the graphical environment again

systemctl start graphical.target

II. Install OpenMPI

# need binutils 2.22 or newer to link cuda

cd openmpi-4.1.1
mkdir build && cd build

Load compilers:

module load compiler/gcc-7.4 # cuda-10 only support to gcc-8
module load binutils-2.35

../configure CC=gcc CXX=g++ FC=gfortran F77=gfortran \
--with-sge --without-ucx \
--with-cuda=/home/thang/local/app/cuda-10.2 \
--prefix=/home/thang/local/app/openmpi/4.1.1-gcc7.4-cuda

Thang's Notes

menu

Friday, July 26, 2019

Compiling OpenMPI 4

If the output is 8, then it supports 64-bit. If output is 4, then it just supports 32-bit.* configuration for 64-bit support:

+ For Intel compilers use:
FFLAGS=-i8 FCFLAGS=-i8 CFLAGS=-m64 CXXFLAGS=-m64
+ For GNU compilers type:
FFLAGS="-m64 -fdefault-integer-8" FCFLAGS="-m64 -fdefault-integer-8" CFLAGS=-m64 CXXFLAGS=-m64

I. OpenMPI-4.0.4 + Intel-2020xe (USC)

II. install OpenMPI + GCC (USC)

##2.3. CAN:

module load gcc/gcc-7.4.0
check: g++ -v

2. Configuration

cd openmpi-4.0.2
mkdir build
cd build

../configure CC=gcc CXX=g++ FC=gfortran F77=gfortran \ --with-sge --without-verbs --without-ucx \ --prefix=/home/thang/local/app/openmpi/4.0.2-gcc7.4.0

##2.4. CAN-GPU:

cd openmpi-4.1.1
mkdir build && cd build

Load compilers:

module load compiler/gcc-7.4 # cuda-10 only support to gcc-8
module load binutils-2.35

No comments:

Post a Comment

menu

Friday, July 26, 2019

Compiling OpenMPI 4

If the output is 8, then it supports 64-bit. If output is 4, then it just supports 32-bit.* configuration for 64-bit support:

+ For Intel compilers use: FFLAGS=-i8 FCFLAGS=-i8 CFLAGS=-m64 CXXFLAGS=-m64 + For GNU compilers type: FFLAGS="-m64 -fdefault-integer-8" FCFLAGS="-m64 -fdefault-integer-8" CFLAGS=-m64 CXXFLAGS=-m64

I. OpenMPI-4.0.4 + Intel-2020xe (USC)

II. install OpenMPI + GCC (USC)

##2.3. CAN:

module load gcc/gcc-7.4.0 check: g++ -v

2. Configuration

cd openmpi-4.0.2mkdir buildcd build ../configure CC=gcc CXX=g++ FC=gfortran F77=gfortran \ --with-sge --without-verbs --without-ucx \ --prefix=/home/thang/local/app/openmpi/4.0.2-gcc7.4.0

##2.4. CAN-GPU:

cd openmpi-4.1.1mkdir build && cd build

Load compilers: module load compiler/gcc-7.4 # cuda-10 only support to gcc-8module load binutils-2.35

No comments:

Post a Comment

+ For Intel compilers use:
FFLAGS=-i8 FCFLAGS=-i8 CFLAGS=-m64 CXXFLAGS=-m64
+ For GNU compilers type:
FFLAGS="-m64 -fdefault-integer-8" FCFLAGS="-m64 -fdefault-integer-8" CFLAGS=-m64 CXXFLAGS=-m64

module load gcc/gcc-7.4.0
check: g++ -v

cd openmpi-4.0.2
mkdir build
cd build

../configure CC=gcc CXX=g++ FC=gfortran F77=gfortran \ --with-sge --without-verbs --without-ucx \ --prefix=/home/thang/local/app/openmpi/4.0.2-gcc7.4.0

cd openmpi-4.1.1
mkdir build && cd build

Load compilers:

module load compiler/gcc-7.4 # cuda-10 only support to gcc-8
module load binutils-2.35