Monday, July 29, 2019

Compiling LAMMPS 2020

-DPYTHON_EXECUTABLE=${myPY} -DPYTHON_LIBRARY=${myPYLIB} -DPYTHON_INCLUDE_DIR=${myPYINC} \https://tinyurl.com/ybt75yau
NOTE: a newer Cmake version may reduce the probability of error during compiling
CMake [-D OPTION_A=VALUE_A -D OPTION_B=VALUE_B ...] ../CMake make

1. Prerequisite:
- Assume that Intel compiler and corresponding OpenMPI & FFTW were be installed
- Also, the newer CMAKE version is updated already.
- Newer LAMMPS may be no longer compatible with an old openMPI, as well FFTW, so these two libs need to be updated too.
- Lammps does not yet support Python 3
- OpenMPI may the fastest
- There is no longer USER_ packages from Jul-2021

module display <module_name>
2. download: 
## git error (if any): Couldn't resolve host 'github.com' while ....
git config --global --unset http.proxy    
git config --global --unset https.proxy 
## local change git
git reset --hard

tar -xvf lammps-stable_7Aug2019
cd lammps-stable_7Aug2019
mkdir build
cd build

# or download use Git:
git clone https://github.com/lammps/lammps.git    lammps_master
cd lammps_master
git checkout master
git pull origin master

# Download specific TAG:  
$ git clone --branch <tag_name> <repo_url>
$ git checkout tagID
git clone --branch patch_20Nov2019 https://github.com/lammps/lammps.git lammps_patch_20Nov2019 cd lammps_patch_20Nov2019
git checkout patch_20Nov2019 --branch master


git clone --branch stable https://github.com/lammps/lammps.git    lammps_stable
cd lammps_stable
git checkout stable


3. Packages: 
add more package (UFM potential) - optional
cd lammps-folder/src/ git clone https://github.com/plrodolfo/FluidFreeEnergyforLAMMPS.git USER-FFE
copy new pair_ufm into /src
copy new pair_eam.cpp & pair_eam.h into /src and delete corresponding files in /src/MANYBODY


NOTEs:
include these OPTIONS in Cmake command, to build package-lib automatically:

1. PYTHON package
-D PYTHON_EXECUTABLE=/uhome/p001cao/local/Anaconda2/bin/python
-D PYTHON_INCLUDE_DIRS=/uhome/p001cao/local/Anaconda2/include
-D PYTHON_LIBRARY=/uhome/p001cao/local/Anaconda2/lib # libpython2.7.so

2. POEMS package, OPT package, 
-D PKG_OPT=yes

3. MSCG Package
-D PKG_MSCG=yes
-D DOWNLOAD_MSCG=yes

3. USER-SMD package
-D PKG_USER-SMD=yes
-D DOWNLOAD_EIGEN3=yes

4. VORONOI package
-D PKG_VORONOI=yes
-D DOWNLOAD_VORO=yes

5. USER-PLUMED package: need to install gsl library
-D PKG_USER-PLUMED=yes
-D DOWNLOAD_PLUMED=yes
-D PLUMED_MODE=static

or: pre-compile Plumed separately: see here
-D PKG_USER-PLUMED=yes
-D DOWNLOAD_PLUMED=no
-D PLUMED_MODE=static



6. KSPACE Package
* if use MKL for FFT, then need MKL library (if FTWW3 (prior ot choose, then dont need MKL_LIBRARY )
-D FFT=MKL  \
-D MKL_INCLUDE_DIRS=/uhome/p001cao/local/intel/xe2019/compilers_and_libraries_2019.5.281/linux/mkl/include  \
-D MKL_LIBRARY=/uhome/p001cao/local/intel/xe2019/compilers_and_libraries_2019.5.281/linux/mkl/lib/intel64  \

-D FFT=FFTW3
-D FFTW3_INCLUDE_DIRS=/uhome/p001cao/local/fftw/3.3.8-openmpi4.0.1-Intel2019xe-double/include \
-D FFTW3_LIBRARY=/uhome/p001cao/local/fftw/3.3.8-openmpi4.0.1-Intel2019xe-double/lib \

or use FFTW3 from intel_mkl: (not support long-double precision)
-D FFT=FFTW3 
-D FFTW3_INCLUDE_DIRS=/uhome/p001cao/local/intel/xe2018/compilers_and_libraries_2018.0.128/linux/mkl/include/fftw 

7. LAPACK & BLAS
module load intel/mkl
Use "intel/mkl" package, then LAPACK & BLAS will be found automatically



8. USER-OMP package
-D PKG_USER-OMP=no
-D BUILD_OMP=no                                #  use BUILD_OMP=yes may cause some bad performance
-D PKG_USER-INTEL=no

9. make no packages
-D PKG_GPU=no -D PKG_KIM=no -D PKG_LATTE=no -D PKG_MSCG=no -D PKG_KOKKOS=no \
-D DOWNLOAD_VORO=yes -D DOWNLOAD_EIGEN3=yes \
-D PKG_USER-ADIOS=no -D PKG_USER-NETCDF=no -D PKG_USER-OMP=no -D PKG_USER-INTEL=no \
-D PKG_USER-QUIP=no -D PKG_USER-SCAFACOS=no -D PKG_USER-QMMM=no -D PKG_USER-VTK=no \
-D PKG_USER-H5MD=no \

10. KOKKOS
https://lammps.sandia.gov/doc/Build_extras.html#kokkos
https://github.com/kokkos/kokkos/wiki/Compiling
For multicore CPUs using OpenMP, set these 2 variables.
-DKokkos_ARCH_WSM=yes                 # HOSTARCH = HOST from list above 
-DKokkos_ENABLE_OPENMP=yes 
-DBUILD_OMP=yes


#### self-build PLUMED: will need GSL to link LAPACK, BLAS (require MKL)
## open file: ../cmake/Modules/Packages/USER-PLUMED.cmake
* Configure Plumed to use Internal LAPACK&BLAS: (no need install BLAS&LAPACK or MKL+GSL)
  # find_package(LAPACK REQUIRED)
  # find_package(BLAS REQUIRED)
  # find_package(GSL REQUIRED)
  # list(APPEND PLUMED_LINK_LIBS ${LAPACK_LIBRARIES} ${BLAS_LIBRARIES} GSL::gsl)

* change lines: 
# URL http...... (line 65)
# URL_MD5
* into: 
GIT_REPOSITORY https://github.com/plumed/plumed2.git 
GIT_TAG master                            # hack-the-tree   v2.6.2   v2.7b

CONFIGURE_COMMAND <SOURCE_DIR>/configure  ....   
            --enable-modules=all --enable-asmjit --disable-external-blas --disable-external-lapack
* add this command after line 76 (inside ExternalProject_Add(...)): UPDATE_COMMAND "" 

#### USER-SMD package: require Eigen
* open file: ../cmake/Modules/Packages/USER-SMD.cmake
* change: 
URL http...... (line 12)
URL_MD5
* into: 
GIT_REPOSITORY https://github.com/eigenteam/eigen-git-mirror.git 
GIT_TAG  3.3.7
####---

##### MLIAP package
* require python >3.6

##### MOLFILE package
to dump PDB file, need install VMD-plugins
compatible with VMD 1.9 and 1.9.1
## Compile VMD

tar zxvf vmd-1.9.src.tar.gz
###1. compile plugins (just this is need for Lammps)
https://www.discngine.com/blog/2019/5/25/building-the-vmd-molfile-plugin-from-source-code
cd plugins
make LINUXPPC64
export PLUGINDIR=/uhome/p001cao/local/wSourceCode/vmd/vmd-1.9/plugins
make distrib

###2. compile VMD
cd vmd-1.9.4a51
module load compiler/gcc-10.3
export VMDINSTALLDIR=/uhome/p001cao/local/app/vmd
./configure LINUXPPC64 OPENGL SILENT PTHREADS
cd src
make

#################
# path in lib/molfile/Make.lammps: molfile_SYSPATH =-L/uhome/p001cao/local/wSourceCode/vmd/vmd-1.9/plugins/LINUXPPC64/molfile
-D MOLFILE_INCLUDE_DIR=path   # (optional) path where VMD molfile plugin headers are installed
-D PKG_MOLFILE=yes
##### PYTHON package (use 1 of following ways, same result)
- new numpy require higher GLIBC
### Module load --> do not need setting in Cmake
module load conda/py37Lammps
### use Python_ROOT_DIR (same as module load): --> will encounter the error: Anaconda environments prevent CMake from generating a safe runtime search path --> cannot be solved so far 
export pyROOT=/uhome/p001cao/local/app/miniconda3/envs/py37Lammps
-DPython_ROOT_DIR=${pyROOT}
### use Python_EXECUTABLE 
export pyEXE=/uhome/p001cao/local/app/miniconda3/envs/py37Lammps/bin/python
export pyINC=/uhome/p001cao/local/app/miniconda3/envs/py37Lammps/include/python3.7m
export pyLIB=/uhome/p001cao/local/app/miniconda3/envs/py37Lammps/lib/libpython3.7m.so  
-DPython_EXECUTABLE=${pyEXE} -DPython_INCLUDE_DIR=${pyINC} -DPython_LIBRARY=${pyLIB} 

https://cmake.org/cmake/help/latest/module/FindPython.html

A. OMPI + GCC

## "GCC + gold linker" is good now
## Dowload:
git clone -b master https://github.com/lammps/lammps.git lammps_master
cd lammps_master 
mkdir build   &&   cd build   

##### 1. USC1 (eagle)
- use different openmpi for Eagle vs Lion
- use external BLAS&LAPACK instead of MKL
- use FFTW instead of MKL
- Note: python>3.7.9 require GLIBC new
conda install python=3.7.5 pandas=1.0 numpy=1.19
- Use GCC-11 need also update GCC-conda = 11
conda install -c conda-forge libstdcxx-ng=11 libgcc-ng=11 libgfortran-ng=11

# Load modules 
# module load intel/mkl-xe19u5
# source mklvars.sh intel64
# module load tool_dev/gsl-2.6
# export myLAPACK=/uhome/p001cao/local/app/lapack-3.9/liblapack.a
# export myBLAS=/uhome/p001cao/local/app/blas-3.8/libfblas.a

module load tool_dev/binutils-2.36        # gold
module load tool_dev/cmake-3.21
module load fftw/fftw3.3.8-ompi4.1-gcc11.2
module load mpi/ompi4.1.1-gcc11.2-noUCX-eagle
##
export PATH=/uhome/p001cao/local/app/openmpi/4.1.1-gcc11.2-noUCX-eagle/bin:$PATH
export CC=mpicc
export CXX=mpic++
export FORTRAN=mpifort
## MOLFILE_plugins:
export PlugIncDIR=/uhome/p001cao/local/wSourceCode/vmd/vmd-1.9/plugins/include
## python (require py3) 
export pyROOT=/uhome/p001cao/local/app/miniconda3/envs/py37Lammps
cmake ../cmake -C ../cmake/presets/all_on.cmake \
-DCMAKE_EXE_LINKER_FLAGS="-fuse-ld=gold -lrt" \
-DPython_ROOT_DIR=${pyROOT} -DMOLFILE_INCLUDE_DIR=${PlugIncDIR} \
-DLAMMPS_EXCEPTIONS=yes -DBUILD_MPI=yes -DBUILD_OMP=yes -DLAMMPS_MACHINE=mpi \
-DPKG_OPENMP=yes -DPKG_INTEL=no -DPKG_GPU=no -DPKG_KOKKOS=no \
-DDOWNLOAD_EIGEN3=yes -DDOWNLOAD_VORO=yes \
-DPKG_KIM=no -DDOWNLOAD_KIM=no -DPKG_LATTE=no -DPKG_MSCG=no -DPKG_ATC=no -DPKG_MESONT=no  \
-DPKG_ADIOS=no -DPKG_NETCDF=no -DPKG_ML-QUIP=no -DPKG_SCAFACOS=no \
-DPKG_VTK=no -DPKG_H5MD=no \
-DFFT=FFTW3 \
-DPKG_PLUMED=yes -DDOWNLOAD_PLUMED=yes\
-DCMAKE_C_COMPILER=mpicc -DCMAKE_CXX_COMPILER=mpic++ -DCMAKE_Fortran_COMPILER=mpifort \
-DCMAKE_INSTALL_PREFIX=/uhome/p001cao/local/app/lammps/gccOMPI-master
make -j 1     # ,ust use 1 core
test:  mpirun -np 2 lmp_mpi
make install
## -DBLAS_LIBRARIES=${myBLAS} -DLAPACK_LIBRARIES=${myLAPACK}
##### 2. USC2:    
#module load mpi/ompi4.0.4-gcc10.1.0-lld
#module load llvm/llvm-gcc10-lld                    # to use lld 
##
module load mpi/ompi4.1.1-gcc10.3
module load tool_dev/binutils-2.35                # gold 
module load tool_dev/cmake-3.20.3
module load fftw/fftw3.3.8-ompi4.1-gcc10.3
##
export PATH=$PATH:/home1/p001cao/local/app/openmpi/4.1.1-gcc10.3/bin
export CC=mpicc
export CXX=mpic++
export FORTRAN=mpifort
## python (require py3) 
export myPy=/home1/p001cao/local/app/miniconda3/envs/py37Lammps/bin/python
cmake ../cmake -C ../cmake/presets/all_on.cmake \
-DCMAKE_EXE_LINKER_FLAGS="-fuse-ld=gold -lrt" \
-DPython_EXECUTABLE=${myPy} \
-DLAMMPS_EXCEPTIONS=yes -DBUILD_MPI=yes -DBUILD_OMP=yes -DLAMMPS_MACHINE=mpi \
-DPKG_OPENMP=yes -DPKG_INTEL=no -DPKG_GPU=no -DPKG_KOKKOS=no \
-DDOWNLOAD_EIGEN3=yes -DDOWNLOAD_VORO=yes \
-DPKG_KIM=no -DDOWNLOAD_KIM=no -DPKG_LATTE=no -DPKG_MSCG=no -DPKG_ATC=no -DPKG_MESONT=no  \
-DPKG_ADIOS=no -DPKG_NETCDF=no -DPKG_ML-QUIP=no -DPKG_SCAFACOS=no \
-DPKG_VTK=no -DPKG_H5MD=no \
-DFFT=FFTW3 \
-DPKG_PLUMED=yes -DDOWNLOAD_PLUMED=yes\
-DCMAKE_C_COMPILER=mpicc -DCMAKE_CXX_COMPILER=mpic++ -DCMAKE_Fortran_COMPILER=mpifort \
-DCMAKE_INSTALL_PREFIX=/home1/p001cao/local/app/lammps/gccOMPI-master
# NOTE: 
* must use: export CC=mpicc, export CXX=mpic++...to avoid miss matching compiler

# use FFTW instead of MKL
module load fftw/fftw3.3.8-ompi4.1-gcc10.2
-DFFT=FFTW3 \

# use Internal LAPACK&BLAS, then no need (GSL & MKL): open file: ../cmake/Modules/Packages/USER_PLUMED.cmake
comment out line 9-->12: find LAPACK, BLAS, GSL (Plumed build itself, no need GSL anymore)
--> then, do not need these:
module load tool_dev/gsl-2.6
module load intel/mkl-xe19u5
source mklvars.sh intel64
-DFFT=MKL \    # must set before Plumed

# load plumed separately (bad alloc)
module load plumed2/2.7htt-gcc
-DPKG_USER-PLUMED=yes -DDOWNLOAD_PLUMED=no -DPLUMED_MODE=shared \

# consider linker:
module load llvm/llvm-gcc10-lld                    # to use lld
-DCMAKE_EXE_LINKER_FLAGS="-fuse-ld=lld -lrt" \
module load tool_dev/binutils-2.35                # gold
-DCMAKE_EXE_LINKER_FLAGS="-fuse-ld=gold -lrt" \

#openKim: 
must create module file for openKim to add its PKG's path




######### OMPI3 
module load compiler/gcc/7.4.0 
module load mpi/gcc-7.4.0/ompi/3.1.4 
module load tool_dev/gsl-2.6 
module load cmake/3.16.2
#--
export PATH=gcc-7.4.0/ompi/3.1.4 /......../bin:$PATH
export CC=mpicc
export CXX=mpic++
export FORTRAN=mpifort





cmake ../cmake -C ../cmake/presets/all_on.cmake \
-DBUILD_MPI=yes -DLAMMPS_MACHINE=mpi \
-DBUILD_OMP=yes -DPKG_USER-OMP=yes -DPKG_USER-INTEL=no \
-DBUILD_LIB=yes -DBUILD_SHARED_LIBS=yes -DLAMMPS_EXCEPTIONS=yes \
-DPKG_GPU=no -DPKG_LATTE=no -DPKG_KOKKOS=no -DPKG_KIM=no -DPKG_MSCG=no \
-DDOWNLOAD_VORO=yes -DDOWNLOAD_EIGEN3=yes \
-DPKG_USER-ADIOS=no -DPKG_USER-NETCDF=no -DPKG_USER-QUIP=no -DPKG_USER-SCAFACOS=no \
-DPKG_USER-QMMM=no -DPKG_USER-VTK=no -DPKG_USER-H5MD=no \
-DPKG_USER-PLUMED=yes -DDOWNLOAD_PLUMED=yes\
-DCMAKE_C_COMPILER=mpicc -DCMAKE_CXX_COMPILER=mpic++ -DCMAKE_Fortran_COMPILER=mpifort \
-DCMAKE_INSTALL_PREFIX=/home1/p001cao/local/app/lammps/19Mar20-gcc3


module load mpi/ompi4.1-gcc7.4-cuda      # cuda-10 only support to gcc-8
module load cmake-3.20.3
module load fftw/fftw3.3.8-ompi4.1-gcc7.4
##
export PATH=$PATH:/home/thang/local/app/openmpi/4.1.1-gcc7.4-cuda/bin
export CC=mpicc
export CXX=mpic++
export FORTRAN=mpifort
## python (require py3) 
export myPy=/home/thang/local/app/miniconda3/envs/py37/bin/python
export CUDA_PATH=/home/thang/local/app/cuda-10.2
export bin2c=/home/thang/local/app/cuda-10.2/bin/bin2c
cmake ../cmake -C ../cmake/presets/all_on.cmake \
-DPython_EXECUTABLE=${myPy} \
-DLAMMPS_EXCEPTIONS=yes -DBUILD_MPI=yes -DBUILD_OMP=yes -DLAMMPS_MACHINE=mpi \
-DPKG_OPENMP=yes -D PKG_OPT=yes -DPKG_INTEL=no -DPKG_KOKKOS=no \
-DPKG_GPU=yes -DGPU_API=cuda -DGPU_ARCH=sm_60 -DBIN2C=${bin2c} -DGPU_PREC=double \
-DDOWNLOAD_EIGEN3=yes -DDOWNLOAD_VORO=yes \
-DPKG_KIM=no -DDOWNLOAD_KIM=no -DPKG_LATTE=no -DPKG_MSCG=no -DPKG_ATC=no -DPKG_MESONT=no  \
-DPKG_ADIOS=no -DPKG_NETCDF=no -DPKG_ML-QUIP=no -DPKG_SCAFACOS=no \
-DPKG_VTK=no -DPKG_H5MD=no \
-DFFT=FFTW3 \
-DPKG_PLUMED=yes -DDOWNLOAD_PLUMED=yes\
-DCMAKE_C_COMPILER=mpicc -DCMAKE_CXX_COMPILER=mpic++ -DCMAKE_Fortran_COMPILER=mpifort \
-DCMAKE_INSTALL_PREFIX=/home/thang/local/app/lammps/gccOMPI-master








#####################




KOKKOS (USC 2) - 05May20 (error tbb_malloc  --> change TBB folder in file TBB.cmake)



-DBUILD_OMP=yes -DKokkos_ARCH_WSM=yes -DKokkos_ENABLE_OPENMP=yes  \
-DLMP_KOKKOS_USE_ATOMICS=yes -DKokkos_ENABLE_HWLOC=yes \



## TBB lib



set     topdir          /home1/p001cao/local/wSourceCode/Tooldev/oneTBB-2020.2
setenv          TBBROOT                 $topdir/bin
prepend-path    INCLUDE          $topdir/include
prepend-path    LD_LIBRARY_PATH         $topdir/build/linux_intel64_gcc_cc9.2.0_libc2.12_kernel2.6.32_release










##-- edit /cmake/Modules/FindTBB_MALLOC.cmake





find_path(TBB_MALLOC_INCLUDE_DIR NAMES tbb.h PATHS $ENV{TBBROOT}/include/tbb)



find_library(TBB_MALLOC_LIBRARY NAMES tbbmalloc PATHS $ENV{TBBROOT}/lib/intel64/gcc4.7





                                                       $ENV{TBBROOT}/build/linux_intel64_gcc_cc9.2.0_libc2.12_kernel2.6.32_release)



https://github.com/kokkos/kokkos/blob/master/BUILD.md
##-- must use
https://stackoverflow.com/questions/52018092/how-to-set-rpath-and-runpath-with-gcc-ld#52020177
export myGCC=/home1/p001cao/local/app/compiler/gcc-9.2.0
-DCMAKE_CXX_LINK_FLAGS="-L${myGCC}/lib64 -Wl,-rpath,${myGCC}/lib64" \




##--
module load mpi/ompi4.0.3-gcc9.2.0 
module load tool_dev/gsl-2.6 
module load tool_dev/cmake-3.17.2

module load tool_dev/binutils-2.32    
module load tool_dev/tbb-2020.2 
 export TBB_MALLOC_LIBRARY 
 export TBB_MALLOC_INCLUDE_DIR
cmake ../cmake -C ../cmake/presets/all_on.cmake \
-DCMAKE_EXE_LINKER_FLAGS="-fuse-ld=gold -lrt" \
-DBUILD_MPI=yes -DLAMMPS_MACHINE=mpi \
-DBUILD_OMP=yes -DKokkos_ARCH_WSM=yes -DKokkos_ENABLE_OPENMP=yes  \
-DBUILD_SHARED_LIBS=yes -DLAMMPS_EXCEPTIONS=yes \
-DPKG_GPU=no -DPKG_LATTE=no -DPKG_KIM=no -DPKG_MSCG=no -DPKG_USER-INTEL=no\
-DDOWNLOAD_VORO=yes -DDOWNLOAD_EIGEN3=yes \
-DPKG_USER-ADIOS=no -DPKG_USER-NETCDF=no -DPKG_USER-QUIP=no -DPKG_USER-SCAFACOS=no \
-DPKG_USER-QMMM=no -DPKG_USER-VTK=no -DPKG_USER-H5MD=no \

-DCMAKE_INSTALL_PREFIX=/home1/p001cao/local/app/lammps/05May20-gcc
C1. OpenSHMEM + GCC
module load mpi/ompi4.1.0-gcc10.2
module load tool_dev/binutils-2.35                # gold 
module load tool_dev/cmake-3.18.0
module load fftw/fftw3.3.8-ompi4.1-gcc10.2
##
export PATH=$PATH:/home1/p001cao/local/app/openmpi/4.1.0-gcc10.2/bin
export CC=shmemcc
export CXX=shmemc++
export FORTRAN=shmemfort
cmake ../cmake -C ../cmake/presets/all_on.cmake \
-DCMAKE_EXE_LINKER_FLAGS="-fuse-ld=gold -lrt" \
-DLAMMPS_EXCEPTIONS=yes -DBUILD_MPI=yes -DBUILD_OMP=yes -DLAMMPS_MACHINE=mpi \
-DPKG_USER-OMP=yes -DPKG_USER-INTEL=no -DPKG_GPU=no -DPKG_KOKKOS=no \
-DPKG_USER-SMD=yes -DDOWNLOAD_EIGEN3=yes -DDOWNLOAD_VORO=yes \
-DPKG_KIM=no -DDOWNLOAD_KIM=no -DPKG_LATTE=no -DPKG_MSCG=no -DPKG_USER-ATC=no -DPKG_USER-MESONT=no  \
-DPKG_USER-ADIOS=no -DPKG_USER-NETCDF=no -DPKG_USER-QUIP=no -DPKG_USER-SCAFACOS=no \
-DPKG_USER-VTK=no -DPKG_USER-H5MD=no \
-DFFT=FFTW3 \
-DPKG_USER-PLUMED=yes -DDOWNLOAD_PLUMED=yes\
-DCMAKE_C_COMPILER=mpicc -DCMAKE_CXX_COMPILER=mpic++ -DCMAKE_Fortran_COMPILER=mpifort \
-DCMAKE_INSTALL_PREFIX=/home1/p001cao/local/app/lammps/gccSHMEM-master


D. Compile Lammps19 with openMPI-conda & MKL-conda

I. Require:

openMPI & MKL must be installed in conda

Note: not yet support ucx



conda install -c conda-forge cmake mkl mkl-include libjpeg-turbo libpng 

conda install -c conda-forge openmpi openmpi-mpicc openmpi-mpicxx openmpi-mpifort 



# infiniBand

conda install -c conda-forge libibverbs-cos6-x86_64





II: Load modules 



module load conda/py37ompi

III: Configure






cmake  -C ../cmake/presets/all_on.cmake \
-D CMAKE_INSTALL_PREFIX=/home1/p001cao/local/lammps/20Nov19conda  \
-D BUILD_MPI=yes -D LAMMPS_MACHINE=mpi \
-D BUILD_LIB=yes -D BUILD_SHARED_LIBS=yes -D LAMMPS_EXCEPTIONS=yes \
-D PKG_GPU=no -D PKG_KIM=no -D PKG_LATTE=no -D PKG_MSCG=no -D PKG_KOKKOS=no \
-D DOWNLOAD_VORO=yes -D DOWNLOAD_EIGEN3=yes \
-D BUILD_OMP=yes -D PKG_USER-OMP=yes -D PKG_USER-INTEL=no \
-D PKG_USER-ADIOS=no -D PKG_USER-NETCDF=no -D PKG_USER-QUIP=no -D PKG_USER-SCAFACOS=no \
-D PKG_USER-QMMM=no -D PKG_USER-VTK=no -D PKG_USER-H5MD=no \
-D PKG_USER-PLUMED=no -D DOWNLOAD_PLUMED=no -D PLUMED_MODE=shared \
-D FFT=MKL \
-D MKL_LIBRARY=/home1/p001cao/local/miniconda3/envs/py37ompi/lib \
-D CMAKE_C_COMPILER=mpicc -D CMAKE_CXX_COMPILER=mpic++ -D CMAKE_Fortran_COMPILER=mpifort \../cmake 











E. Compile Lammps19 with openMPI4.0.1-gcc7.4.0 on CAN

Load modules: 

module load mpi/openmpi4.0.2-gcc7.4.0

module load cmake-3.12



-D PKG_USER-ATC=no -D PKG_VORONOI=no -D PKG_USER-SMD=no -D PKG_USER-PLUMED=no

use KISS for FTT


Configure:


cmake  -C ../cmake/presets/all_on.cmake \
-D CMAKE_INSTALL_PREFIX=/home/thang/local/app/lammps/20Nov19 \
-D BUILD_MPI=yes -D LAMMPS_MACHINE=mpi \
-D BUILD_LIB=yes -D BUILD_SHARED_LIBS=yes -D LAMMPS_EXCEPTIONS=yes \
-D PKG_GPU=no -D PKG_KIM=no -D PKG_LATTE=no -D PKG_MSCG=no -D PKG_KOKKOS=no \
-D PKG_USER-ATC=no -D PKG_VORONOI=no -D PKG_USER-SMD=no \
-D BUILD_OMP=yes -D PKG_USER-OMP=yes -D PKG_USER-INTEL=no \
-D PKG_USER-ADIOS=no -D PKG_USER-NETCDF=no -D PKG_USER-QUIP=no -D PKG_USER-SCAFACOS=no \
-D PKG_USER-QMMM=no -D PKG_USER-VTK=no -D PKG_USER-H5MD=no \
-D PKG_USER-PLUMED=no -D DOWNLOAD_PLUMED=no -D PLUMED_MODE=shared \
-D CMAKE_C_COMPILER=mpicc  -D CMAKE_CXX_COMPILER=mpic++ -D CMAKE_Fortran_COMPILER=mpifort \
../cmake



















B. Lammps on USC2

4. OMPI-clang



OpenMP Compiler compatibility info: Some compilers do not fully support the default(none) directive 
and others (e.g. GCC version 9 and beyond) may implement OpenMP 4.0 semantics 



# run this 

cd /src/USER-OMP 
./hack_openmp_for_pgi_gcc9.sh





#NOTE: 

* The above incompatibility is solved with lammps/15Apr2020: https://github.com/lammps/lammps/pull/1651 
* Clang support openMP 4.5
* Ninja-cmake available from lammps/19Mar20
* from lammps/15Apr2020, lammps develop OpenMP with Clang 

##
git clone --branch patch_15Apr2020 https://github.com/lammps/lammps.git lammps_patch_15Apr2020 
cd lammps_patch_15Apr2020 
git checkout master
git pull origin master
mkdir build 
cd build



Load

module load mpi/ompi4.0.3-clang10 
module load plumed2/2.7htt-clang 
module load cmake-3.15.1





Configure




cmake ../cmake -C ../cmake/presets/all_on.cmake \
-DBUILD_MPI=yes -DLAMMPS_MACHINE=mpi \
-DBUILD_OMP=yes -DPKG_USER-OMP=yes -DPKG_USER-INTEL=no \
-DBUILD_LIB=yes -DBUILD_SHARED_LIBS=yes -DLAMMPS_EXCEPTIONS=yes \
-DPKG_GPU=no -DPKG_LATTE=no -DPKG_KOKKOS=no -DPKG_KIM=no -DPKG_MSCG=no \
-DDOWNLOAD_VORO=yes -DDOWNLOAD_EIGEN3=yes \
-DPKG_USER-ADIOS=no -DPKG_USER-NETCDF=no -DPKG_USER-QUIP=no -DPKG_USER-SCAFACOS=no \
-DPKG_USER-QMMM=no -DPKG_USER-VTK=no -DPKG_USER-H5MD=no \
-DPKG_USER-PLUMED=yes -DDOWNLOAD_PLUMED=no -DPLUMED_MODE=shared \
-DCMAKE_C_COMPILER=mpicc -DCMAKE_CXX_COMPILER=mpic++ -DCMAKE_Fortran_COMPILER=mpifort \
-DCMAKE_INSTALL_PREFIX=/home1/p001cao/local/app/lammps/15Apr20-clang









# USC1
-DCMAKE_INSTALL_PREFIX=/uhome/p001cao/local/app/lammps/19Mar20-clang





5. MVAPICH-GCC



module load mpi/mvapich2-2.3.2-gcc9.2.0
module load plumed2/2.7htt-mvapich
module load conda/py37mvapichSupp



Configure


cmake ../cmake -DCMAKE_EXE_LINKER_FLAGS="-fuse-ld=gold -lrt" \
-C ../cmake/presets/all_on.cmake \
-DBUILD_MPI=yes -DLAMMPS_MACHINE=mpi \
-DBUILD_OMP=yes -DPKG_USER-OMP=yes -DPKG_USER-INTEL=no \
-DBUILD_LIB=yes -DBUILD_SHARED_LIBS=yes -DLAMMPS_EXCEPTIONS=yes \
-DPKG_GPU=no -DPKG_LATTE=no -DPKG_KOKKOS=no -DPKG_KIM=no -DPKG_MSCG=no \
-DDOWNLOAD_VORO=yes -DDOWNLOAD_EIGEN3=yes \
-DPKG_USER-ADIOS=no -DPKG_USER-NETCDF=no -DPKG_USER-QUIP=no -DPKG_USER-SCAFACOS=no \
-DPKG_USER-QMMM=no -DPKG_USER-VTK=no -DPKG_USER-H5MD=no \
-DPKG_USER-PLUMED=yes -DDOWNLOAD_PLUMED=no -DPLUMED_MODE=shared \
-DCMAKE_C_COMPILER=mpicc -DCMAKE_CXX_COMPILER=mpic++ -DCMAKE_Fortran_COMPILER=mpifort \
-DCMAKE_INSTALL_PREFIX=/home1/p001cao/local/app/lammps/19Mar20-mva

https://github.com/lammps/lammps/blob/master/lib/message/cslib/src/STUBS_ZMQ/zmq.h




A. Compile Lammps19 with openMPI-4.0.2, Intel2019xe and MKL (w/wt FFTW-3.3.8) (USC)
II: Load modules
module load mpi/ompi4.0.2-Intel2019xe-noIB
module load intel/mkl-2019xe
module load gcc/gcc-7.4.0
module load plumed2/2.6htt
module load cmake-3.15.1
III: Compiling LAMMPS
* if occur error not found compiler, use this command to find path

find / -name icpc
find / -name ifort

find / -name icc

iii. Compile lammps
cd  lammps-folder
mkdir  build
cd  build

Step1: configuration

Note: write CMAKE command below on single line

cmake  -C ../cmake/presets/all_on.cmake \
-D CMAKE_INSTALL_PREFIX=/uhome/p001cao/local/lammps/20Nov19 \
-D BUILD_MPI=yes -D LAMMPS_MACHINE=mpi \
-D BUILD_LIB=yes -D BUILD_SHARED_LIBS=yes -D LAMMPS_EXCEPTIONS=yes \
-D PKG_GPU=no -D PKG_KIM=no -D PKG_LATTE=no -D PKG_MSCG=no -D PKG_KOKKOS=no \
-D DOWNLOAD_VORO=yes -D DOWNLOAD_EIGEN3=yes \
-D BUILD_OMP=yes -D PKG_USER-OMP=yes -D PKG_USER-INTEL=no \
-D PKG_USER-ADIOS=no -D PKG_USER-NETCDF=no -D PKG_USER-QUIP=no -D PKG_USER-SCAFACOS=no \
-D PKG_USER-QMMM=no -D PKG_USER-VTK=no -D PKG_USER-H5MD=no \
-D PKG_USER-PLUMED=yes -D DOWNLOAD_PLUMED=no -D PLUMED_MODE=shared \
-D FFT=MKL \
-D MKL_LIBRARY=/uhome/p001cao/local/intel/xe2019/compilers_and_libraries_2019.5.281/linux/mkl/lib/intel64 \
-D CMAKE_C_COMPILER=mpicc  -D CMAKE_CXX_COMPILER=mpic++ -D CMAKE_Fortran_COMPILER=mpifort \
../cmake

Step 2: compile ( in /build)
make -j 8
test: mpirun -np 2 lmp_mpi
LAMMPS (19 Jul 2019) 
Total wall time: 0:00:21

step 3: copy file 
make install

Step 4: create module file
 create file "7Aug19"
#######################################
# for Tcl script use only
set     topdir          /uhome/p001cao/local/lammps/7Aug19
set     version         7Aug19

module load  mpi/openMPI/4.0.2-Intel2018xe
module load  fftw/3.3.8/openmpi4.0.2-intel2018xe-double
module load  conda2-2019
module load  plumed2/2.6.0


setenv          LAMMPS                  $topdir

prepend-path    PATH                                    $topdir/bin
prepend-path    LD_LIBRARY_PATH         $topdir/lib64
prepend-path    INCLUDE                            $topdir/include/lammps
#######################################

save it in: /uhome/p001cao/local/share/lmodfiles/lammps
Ref:
https://lammps.sandia.gov/doc/Build_basics.html

#USC2:
Note: Kokkos may require TBB lib --> might only Intel can work
# Download specific TAG: git clone --branch <tag_name> <repo_url>
git clone --branch stable_3Mar2020 https://github.com/lammps/lammps.git lammps_stable_3Mar2020
cd lammps_stable_3Mar2020
mkdir build
cd build

##--- module load mpi/ompi4.0.3-intel19u5 module load intel/mkl-xe19u5 module load plumed2/2.7htt module load tool_dev/cmake-3.17.2


Configure
cmake ../cmake -C ../cmake/presets/all_on.cmake \ -DBUILD_MPI=yes -DLAMMPS_MACHINE=mpi \ -DBUILD_OMP=yes -DKokkos_ARCH_WSM=yes -DKokkos_ENABLE_OPENMP=yes \ -DBUILD_SHARED_LIBS=yes -DLAMMPS_EXCEPTIONS=yes \ -DPKG_GPU=no -DPKG_LATTE=no -DPKG_KIM=no -DPKG_MSCG=no -DPKG_USER-INTEL=no\ -DDOWNLOAD_VORO=yes -DDOWNLOAD_EIGEN3=yes \ -DPKG_USER-ADIOS=no -DPKG_USER-NETCDF=no -DPKG_USER-QUIP=no -DPKG_USER-SCAFACOS=no \ -DPKG_USER-QMMM=no -DPKG_USER-VTK=no -DPKG_USER-H5MD=no \ -DPKG_USER-PLUMED=yes -DDOWNLOAD_PLUMED=no -DPLUMED_MODE=shared \ -DCMAKE_C_COMPILER=mpicc -DCMAKE_CXX_COMPILER=mpic++ \ -DCMAKE_INSTALL_PREFIX=/home1/p001cao/local/app/lammps/05May20
B. with IMPI-2019xe + MKL:
Note: use intelMPI can run all both centos7 & centos6
# 1. USC 1:
Load modulesmodule load intel/compiler-xe19u5
module load mpi/impi-xe19u5
module load intel/mkl-xe19u5
module load plumed2/2.6httIMPI
module load conda/py37
module load cmake-3.15.1 

configuration
cd lammps-folder
mkdir build
cd build
cmake  -C ../cmake/presets/all_on.cmake \
-D CMAKE_INSTALL_PREFIX=/uhome/p001cao/local/app/lammps/20Nov19impi \
-D BUILD_MPI=yes -D LAMMPS_MACHINE=mpi \
-D BUILD_LIB=yes -D BUILD_SHARED_LIBS=yes -D LAMMPS_EXCEPTIONS=yes \
-D PKG_GPU=no -D PKG_KIM=no -D PKG_LATTE=no -D PKG_MSCG=no -D PKG_KOKKOS=no \
-D DOWNLOAD_VORO=yes -D DOWNLOAD_EIGEN3=yes \
-D BUILD_OMP=yes -D PKG_USER-OMP=yes -D PKG_USER-INTEL=no \
-D PKG_USER-ADIOS=no -D PKG_USER-NETCDF=no -D PKG_USER-QUIP=no -D PKG_USER-SCAFACOS=no \
-D PKG_USER-QMMM=no -D PKG_USER-VTK=no -D PKG_USER-H5MD=no \
-D PKG_USER-PLUMED=yes -D DOWNLOAD_PLUMED=no -D PLUMED_MODE=shared \
-D FFT=MKL \
-D MKL_LIBRARY=/uhome/p001cao/local/app/intel/xe19u5/compilers_and_libraries_2019.5.281/linux/mkl/lib/intel64_lin \
-D CMAKE_C_COMPILER=mpiicc -D CMAKE_CXX_COMPILER=mpiicpc -D CMAKE_Fortran_COMPILER=mpiifort \
../cmake
make -j 8
test:  mpirun -np 2 ./lmp_mpi
make install
Step 4: create module file
 create file "7Aug19-Impi"
############################################
module load intel/2019xe
module load mpi/impi-2019xe
module load plumed2/2.6.0-Impi
module load conda2-2019
setenv          LAMMPS                  $topdir

prepend-path    PATH                                    $topdir/bin
prepend-path    LD_LIBRARY_PATH         $topdir/lib64
prepend-path    INCLUDE                            $topdir/include/lammps
###############################################################


# 2. USC 2:
module load compiler/gcc-10.2              # must load before impi
module load intel/compiler-xe19u5           # intel include lld linker  require GLIBC 2.15
module load intel/mkl-xe19u5  
module load intel/impi-xe19u5           
source mpivars.sh release
module load tool_dev/cmake-3.18.0
module load tool_dev/gsl-2.6
module load tool_dev/binutils-2.32                # gold 
export PATH=$PATH:/home1/p001cao/local/app/intel/xe19u5/compilers_and_libraries_2019.5.281/linux/bin
export CC=mpiicc
export CXX=mpiicpc
export FORTRAN=mpiifort

Configure
cmake ../cmake -C ../cmake/presets/all_on.cmake \
-DCMAKE_EXE_LINKER_FLAGS="-fuse-ld=gold -lrt" \
-DLAMMPS_EXCEPTIONS=yes -DBUILD_MPI=yes -DBUILD_OMP=yes -DLAMMPS_MACHINE=mpi \
-DPKG_USER-OMP=yes -DPKG_USER-INTEL=yes -DPKG_GPU=no -DPKG_KOKKOS=no \
-DPKG_USER-SMD=yes -DDOWNLOAD_EIGEN3=yes -DDOWNLOAD_VORO=yes \
-DPKG_KIM=no -DDOWNLOAD_KIM=no -DPKG_LATTE=no -DPKG_MSCG=no -DPKG_USER-ATC=no \
-DPKG_USER-ADIOS=no -DPKG_USER-NETCDF=no -DPKG_USER-QUIP=no -DPKG_USER-SCAFACOS=no \
-DPKG_USER-VTK=no -DPKG_USER-H5MD=no \
-DFFT=MKL \
-DPKG_USER-PLUMED=yes -DDOWNLOAD_PLUMED=yes\
-DCMAKE_C_COMPILER=mpiicc -DCMAKE_CXX_COMPILER=mpiicpc -DCMAKE_Fortran_COMPILER=mpiifort \
-DCMAKE_INSTALL_PREFIX=/home1/p001cao/local/app/lammps/impi-master
#-- NOTE: Kokkos require TBB lib
module load intel/tbb-xe20u2
##-- edit /cmake/Modules/FindTBB_MALLOC.cmake
find_path(TBB_MALLOC_INCLUDE_DIR NAMES tbb.h PATHS $ENV{TBBROOT}/include/tbb)
find_library(TBB_MALLOC_LIBRARY NAMES tbbmalloc PATHS $ENV{TBBROOT}/lib/intel64/gcc4.8)     
##--
-DCMAKE_EXE_LINKER_FLAGS="-fuse-ld=lld -lrt" \
source compilervars.sh intel64
source mklvars.sh intel64
Posted by at No comments:
Labels: , ,

Sunday, July 28, 2019

Compiling GCC 10

NOTE:
- Some applications require C++11, this is only supported on GCC 4.8 or newer
- intel 2018 support gcc versions 4.3 - 6.3
https://software.intel.com/en-us/articles/intel-c-compiler-180-for-linux-release-notes-for-intel-parallel-studio-xe-2018

1. Download:
https://gcc.gnu.org/releases.html

check all availabe versions GCC :
svn ls svn://gcc.gnu.org/svn/gcc/tags | grep gcc | grep release
#or http://ftp.tsukuba.wide.ad.jp/software/gcc/releases

wget http://ftp.tsukuba.wide.ad.jp/software/gcc/releases/gcc-10.3.0/gcc-10.3.0.tar.gz
tar xvf gcc-10.3.0.tar.gz

2. Install
a. download prerequisites:
cd gcc-10.3
./contrib/download_prerequisites

b. Configure:
note: compile GCC out source-dir, to avoid modifying source code when compiling get fail


## USC1: (eagle)
## (if any) error: Couldn't resolve host 'github.com' while ....
git config --global --unset http.proxy    
git config --global --unset https.proxy 
####
git clone -b releases/gcc-11.2.0 https://github.com/gcc-mirror/gcc gcc-11.2.0
cd gcc-11.2
git checkout releases/gcc-11.2
./contrib/download_prerequisites
mkdir build && cd build
module load compiler/gcc-10.3         # to avoid:  uint64_t or int64_t not found 
../configure --enable-languages=c,c++,objc,obj-c++,fortran \
--enable-checking=release --enable-shared --disable-multilib --with-system-zlib \
--prefix=/uhome/p001cao/local/app/compiler/gcc-11.2
make    # note use -j to know what error

## USC 2
--prefix=/home1/p001cao/local/app/compiler/gcc-10.3
## CAN
--prefix=/home/thang/local/app/compiler/gcc-10.3
## CAN_GPU
--prefix=/home/thang/local/app/compiler/gcc-10.3
#configure: error: uint64_t or int64_t not found       --> need at least gcc-4.5
module load compiler/gcc-7.4

https://stackoverflow.com/questions/7412548/error-gnu-stubs-32-h-no-such-file-or-directory-while-compiling-nachos-source

c. Installing:
make -j 8
make install

check: g++ -v

 3.Make module file 
at directory: /uhome/p001cao/local/share/lmodfiles/GCC
create file "gcc-10.3"
# for Tcl script use only set topdir /home1/p001cao/local/app/compiler/gcc-10.3
set version gcc-10.3.0
setenv gcc $topdir

# gcc compiler
prepend-path    PATH                    $topdir/bin
prepend-path    INCLUDE         $topdir/include
prepend-path    LD_LIBRARY_PATH         $topdir/lib
prepend-path    LD_LIBRARY_PATH         $topdir/lib64
prepend-path    LD_LIBRARY_PATH         $topdir/libexec


ref:
https://m.blog.naver.com/PostView.nhn?blogId=shumin&logNo=220823075261&proxyReferer=https%3A%2F%2Fwww.google.com%2F
https://gcc.gnu.org/wiki/InstallingGCC
https://serverkurma.com/linux/how-to-install-newer-version-of-gcc-on-centos-6-x/
https://trilinos.github.io/install_gcc.html

Posted by at No comments:
Labels: ,

Saturday, July 27, 2019

Install FFTW, BLAS & LAPACK 



II. BLAS & LAPACK
https://ahmadzareei.github.io/azareei/linux/2016/04/08/configuring-blas-lapack.html
1. Dowloand LAPACK
https://thelinuxcluster.com/2012/04/09/building-lapack-3-4-with-intel-and-gnu-compiler/
wget http://www.netlib.org/lapack/lapack-3.9.0.tgz
tar -zxvf lapack-3.9.0.tgz
cd lapack-3.9.0
## Use gfortran 64-bits compiler
module load compiler/gcc-10.2
##  copy and create make.inc
cp INSTALL/make.inc.gfortran make.inc

Edit the make.inc

make -j 12
## copy lib to new place
mkdir -p /uhome/p001cao/local/app/lapack-3.9
cp liblapack.a /uhome/p001cao/local/app/lapack-3.9/liblapack.a
## Use:
export LAPACK=/uhome/p001cao/local/app/lapack-3.9/liblapack.a


2. Install BLAS
- should use BLAS source include in LAPACK source
cd lapack-3.9.0/BLAS
## Use gfortran 64-bits compiler
module load compiler/gcc-10.2
gfortran -O3 -std=legacy -m64 -fno-second-underscore -fPIC -c *.f
ar r libfblas.a *.o   # creates libblas.a
ranlib libfblas.a
## copy lib to new place
mkdir -p /uhome/p001cao/local/app/blas
cp libfblas.a /uhome/p001cao/local/app/blas/libfblas.a  
## Use: 
export BLAS=/uhome/p001cao/local/app/blas/libfblas.a





Posted by at No comments:
Labels: , , ,

Friday, July 26, 2019

Compiling OpenMPI 4

- Some applications require C++11, this is only supported on GCC 4.8 or newer, which is not always available on system, then newer GCC need to be installed before compiling Openmpi
- Make sure to build OpenMPI with 64-bit support. To check whether the currently available 
OpenMPI do support 64-bit or not, type this:
                ompi_info -a | grep 'Fort integer size'

If the output is 8, then it supports 64-bit. If output is 4, then it just supports 32-bit.* configuration for 64-bit support:

+ For Intel compilers use:
           FFLAGS=-i8 FCFLAGS=-i8 CFLAGS=-m64 CXXFLAGS=-m64
+ For GNU compilers type:
           FFLAGS="-m64 -fdefault-integer-8" FCFLAGS="-m64 -fdefault-integer-8" CFLAGS=-m64 CXXFLAGS=-m64

- must keep the source after compiling
Install UCX and libfabric:
################
Installation OPTIONS in README.txt
or ./configure -h

tar xvzf openmpi-4.0.2.tar.gz
cd openmpi-4.0.2
mkdir build
cd build

I. OpenMPI-4.0.4 + Intel-2020xe (USC)

Note: Intel2019 on cluster has wrong path, cannot work

- "--with-verbs" to use InfiniBand
- Use infiniband may limit the number of nodes?
-without-verbs/--without-ucx... can be turned at run-time:
mpirun --mca btl ^tcp,openib -np 4 a.out
export OMPI_MCA_btl=^tcp,openib
export OMPI_MCA_btl=^tcp


So just compile the whole thing.
#--
# USC1
module load intel/compiler-xe19u5
module load compiler/gcc/9.1.0

check: icpc -v
Configure OpenMPI
../configure CC=icc CXX=icpc FC=ifort F77=ifort \
--with-sge --with-verbs --without-ucx --without-cma \
--prefix=/uhome/p001cao/local/app/openmpi/4.0.2-intelxe19u5-IB
make -j 8
make install

3. Test openMPI
the only thing that users need to do to use Open MPI is ensure that: 
<prefix>/bin is in their PATH, and
<prefix>/lib is in their LD_LIBRARY_PATH.
Users may need to ensure to set the PATH and LD_LIBRARY_PATH in their shell setup files (e.g., .bashrc, .cshrc) so that non-interactive rsh/ssh-based logins will be able to find the Open MPI executables.

Create Module file:
.............

TEST: mpic++ -v

#2.2. USC 2: 
# use linker lld (include in Intel-bin, require GLIBC >2.15)
module load compiler/gcc-10.1.0
module load intel/compiler-xe19u5       # lld
#--
export myUCX=/home1/p001cao/local/app/tool_dev/ucx-1.8-intel  
../configure CC=icc CXX=icpc FC=ifort F77=ifort LDFLAGS="-fuse-ld=lld -lrt" \
--with-sge --without-verbs --with-ucx=${myUCX} \
--prefix=/home1/p001cao/local/app/openmpi/4.0.4-intelxe19u5
## consider link to lib
export myIntel=/home1/p001cao/local/app/intel/xe19u5/compilers_and_libraries_2019.5.281/linux/compiler/lib
LDFLAGS="-L${myIntel}/intel64_lin -Wl,-rpath,${myIntel}/intel64_lin" \
##--
export PATH=/home1/p001cao/local/app/intel/xe19u5/compilers_and_libraries_2019.5.281/linux/bin/intel64:$PATH
export CC=icc  export CXX=icpc  export FORTRAN=ifort

II. install OpenMPI + GCC (USC)

NOTE: 
* install libfabric, knem,... in UCX (openMPI 4.0,3 --> ucx-1.7 or older), or install with openMPI 
* use UCX is recommended: --without-verbs  

## consider lld linker: 
module load llvm/llvm-gcc10-lld                   # to use lld  
LDFLAGS="-fuse-ld=lld -lrt"    

## gold linker:
module load tool_dev/binutils-2.32                                         
LDFLAGS="-fuse-ld=gold -lrt"     

# 2.1. USC 1:
## 1. not use UCX
module load tool_dev/binutils-2.36                       # gold, should use to avoid link-error
module load compiler/gcc-11.2
export myKNEM=/uhome/p001cao/local/app/tool_dev/knem-1.1.4
    
## IB cluster
mkdir build_eagle && cd build_eagle 
../configure CC=gcc CXX=g++ FC=gfortran F77=gfortran LDFLAGS="-fuse-ld=gold -lrt" \
--with-sge --without-ucx --with-verbs --with-knem=${myKNEM} \
--prefix=/uhome/p001cao/local/app/openmpi/4.1.1-gcc11.2-noUCX-eagle
## noIB cluster
mkdir build_lion && cd build_lion
../configure CC=gcc CXX=g++ FC=gfortran F77=gfortran LDFLAGS="-fuse-ld=gold -lrt" \
--with-sge --without-ucx --without-verbs --with-knem=${myKNEM} \
--prefix=/uhome/p001cao/local/app/openmpi/4.1.1-gcc11.2-noUCX-lion

### 2. with ucx
# ucx Error: ib_md.c:329  UCX  ERROR ibv_reg_mr(address=0x145cb580, length=263504, access=0xf) failed: Resource temporarily unavailable

## use the same procedure to compile on Lion and Eagle
module load tool_dev/binutils-2.35                        # gold 
module load compiler/gcc-10.2                 
export myUCX=/uhome/p001cao/local/app/tool_dev/ucx-1.9
export myKNEM=/uhome/p001cao/local/app/tool_dev/knem-1.1.4
../configure CC=gcc CXX=g++ FC=gfortran F77=gfortran LDFLAGS="-fuse-ld=gold -lrt" \
--with-sge --without-verbs --with-ucx=${myUCX} --with-knem=${myKNEM} \
--prefix=/uhome/p001cao/local/app/openmpi/4.1.1-gcc10.3-eagle


##2.2. USC 2: 
## with UCX: on Tacheon, ucx gives better performance (but raise posibility of err)
cd openmpi-4.1.1
mkdir buildGCC && cd buildGCC
#--
module load tool_dev/binutils-2.35                        # gold
module load compiler/gcc-10.3
export myUCX=/home1/p001cao/local/app/tool_dev/ucx-1.10               ## UCX
../configure CC=gcc CXX=g++ FC=gfortran F77=gfortran LDFLAGS="-fuse-ld=gold -lrt" \
--with-sge --without-verbs --with-ucx=${myUCX}  \
--prefix=/home1/p001cao/local/app/openmpi/4.1.1-gcc10.3
#not use
export myKNEM=/home1/p001cao/local/app/tool_dev/knem-1.1.4
--with-knem=${myKNEM}

## without UCX: 
module load tool_dev/binutils-2.35                        # gold
module load compiler/gcc-10.3
../configure CC=gcc CXX=g++ FC=gfortran F77=gfortran LDFLAGS="-fuse-ld=gold -lrt" \
--with-sge --with-verbs --without-ucx \
--prefix=/home1/p001cao/local/app/openmpi/4.1.1-gcc10.3-noUCX

##2.3. CAN: 

module load gcc/gcc-7.4.0
check:   g++   -v

2. Configuration

cd openmpi-4.0.2
mkdir build
cd build
../configure CC=gcc CXX=g++ FC=gfortran F77=gfortran \
--with-sge --without-verbs --without-ucx  \
--prefix=/home/thang/local/app/openmpi/4.0.2-gcc7.4.0

##2.4. CAN-GPU: 

I. Install Cuda with Intel-xe19: (Runfile Installation)
Download:  wget http://developer.download.nvidia.com/compute/cuda/10.2/Prod/local_installers/cuda_10.2.89_440.33.01_rhel6.run

Install: (root acc)
1. disable the graphical target, to update Nvidia driver
systemctl isolate multi-user.target
modprobe -r nvidia-drm

module load compiler/gcc-7.4
sh cuda_10.2.89_440.33.01_rhel6.run --toolkitpath=/home/thang/local/app/cuda-10.2

2. after install Cuda, start the graphical environment again
systemctl start graphical.target

II. Install OpenMPI
# need binutils 2.22 or newer to link cuda

cd openmpi-4.1.1
mkdir build && cd build

Load compilers:
module load compiler/gcc-7.4   # cuda-10 only support to gcc-8
module load binutils-2.35 

../configure CC=gcc CXX=g++ FC=gfortran F77=gfortran \
--with-sge --without-ucx \
--with-cuda=/home/thang/local/app/cuda-10.2 \
--prefix=/home/thang/local/app/openmpi/4.1.1-gcc7.4-cuda



Posted by at No comments:
Labels: , , ,

Wednesday, July 10, 2019

Compiling Lammps/22Aug18 with Intel 14.0, OpenMPI 1.8.1 and FFTW 3.3.4 (USC)

I.  Assum that Intel compiler and correspoding OpenMPI & FFTW werer be installed

 module avail
 module display "module_name"

II:  Load modules

 module load   intel/intel-14.0
 module load   intel/mkl-14.0
 module load   mpi/intel-14.0/openmpi-1.8.1
 module load   fftw/3.3.4/openmpi-1.8.1/intel-14.0/double/fftw-3.3.4
check 
 module list

III: Compiling LAMMPS

tar -xzvf lammps*.tar.gz

Step 1: chose the packages. 

Go to src directory
cd lammps_folder
1a. Check which packages are included
make package-status
1b. Choose all the standard
make yes-standard

make yes-user-fep
make yes-user-misc
make yes-user-phonon
make yes-user-meamc
make yes-user-reaxc
1c. Exclude packages that are not required
 make no-voronoi
 make no-kim
 make no-gpu
 make no-kokkos
 make no-mscg

make no-latte
make no-message

Step2: Prepare libraries for the chosed packages

2a. lib/reax
make -f Makefile.ifort
2b. lib/meam
There are 2 ways to do:
way 1: manually
$ cd lib/meam
$ vi Makefile.lammps.ifort
      - - - as below - - - (in Vi press "i" to insert mode --> Esc --> :wq)

$ make -f Makefile.ifort
[Makefile.lammps.ifort modify] 
meam_SYSINC =
meam_SYSLIB =
meam_SYSPATH =
way 2: perform in lammps/src

cd lammps/src

make lib-meam # print help message
make lib-meam args="-m mpi" # build with default Fortran compiler compatible with your MPI library
make lib-meam args="-m serial" # build with compiler compatible with "make serial" (GNU Fortran)
make lib-meam args="-m ifort" # build with Intel Fortran compiler using Makefile.ifort
2c. lib/poems
cd lib/poems
make -f Makefile.icc
2d. lib/atc
 cd lib/atc
 vi Makefile.lammps.installed
        - - - as below- - -

 make -f Makefile.mpic++
[Makefile.lammps.installed modify]

 
user-atc_SYSINC =
user-atc_SYSLIB =
user-atc_SYSPATH =
2e. lib/awpmd

 
$ cd lib/awpmd
$ vi Makefile.lammps.installed
     - - - as below - - -

$ make -f Makefile.mpicc
[Makefile.lammps.installed 수정]

 
user-awpmd_SYSINC =
user-awpmd_SYSLIB =
user-awpmd_SYSPATH =
2f. lib/linalg

 
$ cd lib/linalg
$ cp -p Makefile.gfortran Makefile.ifort
$ vi Makefile.ifort
- - - 수정 사항은 아래 참조 - - -
$ make -f Makefile.ifort
[Makefile.ifort 수정]

 
FC = ifort
FFLAGS = -O3 -fPIC
FFLAGS0 = -O0 -fPIC
2g. lib/latte

 
finding way...

Step 3: Edit makefile

Copy file src/MAKE/Makefile.mpi
to  src/MAKE/MINE/Makefile.f3mpi   and then edit file Makefile.f3mpi

3a # compiler/linker settings:

 
CCFLAGS = -O3 -fno-alias -restrict -ip -unroll0 -shared-intel
LINKFLAGS = -O -L/opt/intel/composer_xe_2013_sp1.3.174/mkl/lib/intel64
LIB = -lstdc++ -lpthread -lmkl_sequential -lmkl_intel_lp64 -lmkl_core

3b MPI Library
MPI_INC =   -I/opt/mpi/intel-14.0/openmpi-1.8.1/include -DMPICH_SKIP_MPICXX -DOMPI_SKIP_MPICXX=1
MPI_PATH =    
MPI_LIB = -L/opt/mpi/intel-14.0/openmpi-1.8.1/lib
3c FFT library
FFT_INC =   -I/opt/fftw/3.3.4/openmpi-1.8.1/intel-14.0/double/include/fftw3 
FFT_PATH = 
FFT_LIB = /opt/fftw/3.3.4/openmpi-1.8.1/intel-14.0/double/lib/libfftw3.a

Step 4: Compile LAMMPS

Check make options. Go to src  (delete previous make --> make clean-all)
# make
# make f3mpi -j8
test:  mpirun -np 2 lmp_f3mpi

Step 5: Create a /usr/local/lammps/11Aug17ti/bin and copy libraries. 

Go to lammps root directory
 cp -Rv  bench   /uhome/p001cao/local/lammps/11Aug18/bench 
 cp -Rv  doc      /uhome/p001cao/local/lammps/11Aug17ti/doc
 cp -Rv  examples    /uhome/p001cao/local/lammps/11Aug17ti/examples
 cp -Rv  potentials   /uhome/p001cao/local/lammps/11Aug17ti/potentials
 cp -Rv tools   /uhome/p001cao/local/lammps/11Aug17ti/tools
 cp -Rv lib     /uhome/p001cao/local/lammps/11Aug17ti/lib
    cd src
 cp -p lmp_f3mpi  /uhome/p001cao/local/lammps/11Aug17ti/bin


cd /uhome/p001cao/local/lammps/11Aug17ti/bin
cp -p  lmp_f3mpi  lmp_mpi
* Test: at /bin
mpirun -np 2 lmp_mpi
LAMMPS (11 Aug 2017)


Step 6. Create a Modulefile

create modulefile:  lammps/11Aug17ti
and put it into folder:  /uhome/p001cao/local/share/lmodfiles
# for Tcl script use only
set     topdir          /uhome/p001cao/local/lammps
set     version         11.Aug.2017fep

module load   mpi/intel-14.0/openmpi-1.8.1
module load   fftw/3.3.4/openmpi-1.8.1/intel-14.0/double/fftw-3.3.4

setenv          LAMMPS                  $topdir/11Aug17ti
prepend-path    PATH                    $topdir/11Aug17ti/bin
prepend-path    LD_LIBRARY_PATH         $topdir/11Aug17ti/lib
load module

. /etc/profile
module use /uhome/p001cao/local/share/lmodfiles
module load lammps/11Aug17ti
mpirun -np $NSLOTS -machinefile $TMPDIR/machines lmp_mpi ...
Step 7. (Optional) make module usable:
# attach the following command to .bashrc file
module use /uhome/p001cao/local/share/lmodfiles
cd ~ls -la                                                        # to see invisible filesvi .bashrc

References:

Notes:
If copy lmp_linux file using WinSCP, then have to use this command to set it as executable file (assume staying .../bin
#   chmod  +x  lmp_linux
Posted by at No comments:

Install OpenMPI-2.1.5 with GCC-4.4 (USC user) 

1. Assum that GCC compiler was installed on USC
check 
$ gcc -v
gcc version 4.4.7 20120313 (Red Hat 4.4.7-23) (GCC)
Note: install Conda2, will have GCC-7.2.0









2. Configuration and install OpenMPI









cd openmpi-2.1.5

./configure --prefix=/uhome/p001cao/local/openmpi-2.1.5-gcc-4.4 CC=gcc CXX=g++ F77=gfortran FC=gfortran --with-devel-headers --enable-binaries 

make -j8 
make all install
















Test the result

# mpicc -v



gcc version 7.3.0 (Ubuntu 7.3.0-27ubuntu1~18.04)














3. Make module file

create file "openmpi-2.1.5"
# for Tcl script use only
set topdir /uhome/p001cao/local/openmpi-2.1.5-gcc-4.4
set version 2.1.5
prepend-path PATH $topdir/bin
prepend-path LD_LIBRARY_PATH $topdir/lib
prepend-path INCLUDE $topdir/include
prepend-path MANPATH $topdir/share/man

save it in: /uhome/p001cao/local/share/lmodfiles/mpi/gcc-4.4
Posted by at No comments:

Tuesday, June 25, 2019

Basic concepts of elastic waves

( under constructed)


Read more »
Posted by at No comments:

Thursday, June 20, 2019

Numerical analysis

What is numerical analysis?
Numerical analysis is solving the mathematical problems (engineering and physical applications) by using numerical methods, i.e., the Algorithms that are used to obtain the numerical solutions.

Why do we need numerical analysis?
In engineering and physical applications, there are a lot of problems which are difficult or impossible to solve algebraically using elementary calculus. They need systemic methods, i.e., numerical methods, for solving them numerically on computers or calculators. Typical numerical methods are iterative in nature and, for a well-chosen problem and a good starting value, will frequently converge to a desired value.
Think about solving such below problem by using your calculus?



Floating-point forms of Numbers
In decimal notation, every real number can be presented by a finite or an infinite sequence  of decimal digits. Most computers have two ways of representing numbers, called fixed point and floating point.


(under constructed)


Ref.:
1. How to Post Math Equations in Blogger Using Latex
Posted by at No comments:
Subscribe to: Posts (Atom)