Tuesday, November 12, 2019

Pure python & envs using Pip

I. Python+pip on Linux:

1. Need OpenSSL to avoid error: (and configure ./configure --with-ssl)

wget https://www.openssl.org/source/openssl-1.1.1g.tar.gz
tar -xf openssl-1.1.1g.tar.gz
cd openssl-1.1.1g

./config --prefix=/home1/p001cao/local/app/tool_dev/openssl \
--openssldir=/home1/p001cao/local/app/tool_dev/openssl
make -j 8 
make install 

2. Need "libffi" to avoid "_ctypes" error
tar -xf libffi-3.3.tar.gz
cd libffi-3.3
./configure --prefix=/home1/p001cao/local/app/tool_dev/libffi-3.3

Download 

export PYTHON_VERSION=3.7.5

curl -O https://www.python.org/ftp/python/${PYTHON_VERSION}/Python-${PYTHON_VERSION}.tgz tar -xvzf Python-${PYTHON_VERSION}.tgz cd Python-${PYTHON_VERSION}

Build and install Python#

Configure, make, and install Python:

./configure \ --prefix=/uhome/p001cao/local/app/python/${PYTHON_VERSION} \ --enable-shared \ --enable-ipv6 \ LDFLAGS=-Wl,-rpath=/uhome/p001cao/local/app/python/${PYTHON_VERSION}/lib,--disable-new-dtags

make -j 8 
make install 



REF; https://docs.rstudio.com/resources/install-python-source/


create module python3
set topdir /home1/p001cao/local/app/tool_dev/python37
module load openSSL
prepend-path    PATH                    $topdir/bin
prepend-path    INCLUDE         $topdir/include
prepend-path    LD_LIBRARY_PATH         $topdir/lib

Test: 
module load tool_dev/python37
python3 -V


python3 -m pip install --upgrade pip
python3 -m pip install --upgrade pip --trusted-host pypi.org --trusted-host files.pythonhosted.org

2. Setting Up a Virtual Environment

Setting up a programming environment provides us with greater control over our Python projects and over how different versions of packages are handled. 
venv (for Python 3) and virtualenv (for Python 2) allow you to manage separate package installations for different projects. They essentially allow you to create a “virtual” isolated Python installation and install packages into that virtual installation. When you switch projects, you can simply create a new virtual environment and not have to worry about breaking the packages installed in the other environments.
Choose which directory you would like to put your Python programming environments in, or create a new directory in installing dir of python:

cd  /uhome/p001cao/local/python/Python37
mkdir enviroments
cd  enviroments
module load python/python37
python3 -m venv    py37impi
source py37impi/bin/activate

create module python/python37
set               topdir            /uhome/p001cao/local/python/Python37/enviroments/py37impi
module load openSSL
prepend-path    PATH                    $topdir/bin
prepend-path    LD_LIBRARY_PATH         $topdir/lib
prepend-path    INCLUDE                 $topdir/include

if { [ module-info mode load ] } {
    puts stdout "source $topdir/bin/activate;"
} elseif { [ module-info mode remove ] } {
    puts stdout "deactivate"
}

2. Install packages:

load corresponding envs:
module load python/py37impi

python3 -m pip install --upgrade pip
pip3 install tess ovito numpy scipy matplotlib mpi4py impi




III. Some problems:
SSL:
https://joshspicer.com/python37-ssl-issue

wget https://www.openssl.org/source/openssl-1.0.2q.tar.gz
tar xvf openssl-1.0.2q.tar.gz
cd   openssl-1.0.2q
./config  --prefix=/uhome/p001cao/local/openssl-1.0.2
make
make install
create module openSSL
set               topdir           /uhome/p001cao/local/openssl-1.0.2
prepend-path    PATH                    $topdir/bin
prepend-path    LD_LIBRARY_PATH         $topdir/lib
prepend-path    INCLUDE                 $topdir/include


IV. Install package from source

cd sourceFolder
pip install -e .


Sunday, November 10, 2019

Window's subLinux (WSL)

https://www.tegakari.net/en/2020/08/windows-subsystem-for-linux-2wsl2_vol1/

I. Install WSL on Windows

https://docs.microsoft.com/en-us/windows/wsl/install-win10
There are 2 versions: Comparing WSL 2 and WSL 1. Should use WSL 2 for better performance.

a. Prepare
# Enable both the Windows Subsystem for Linux and the Virtual Machine Platform optional components.
Open PowerShell as an admin and run the following script: (After the script is complete, you need to reboot your machine, since this enables new Windows features.)
dism.exe /online /enable-feature /featurename:Microsoft-Windows-Subsystem-Linux /all /norestart
dism.exe /online /enable-feature /featurename:VirtualMachinePlatform /all /norestart

# Download & Install the Linux kernel update package
# Set WSL 2 as your default version: in Powershell
wsl --set-default-version 2

b. Installing WSL: 
# using Microsoft Store
go to: https://aka.ms/wslstore
and intsall a Linux shell
# not use Microsoft Store 
https://docs.microsoft.com/en-us/windows/wsl/install-manual
download: Ubuntu_2004.2020.424.0_x64.appx
in powerShell: Add-AppxPackage .\Ubuntu_2004.2020.424.0_x64.appx


c. Create a Unix Account
Open up a Command Prompt and run the following command:  > bash 

d. Location
on Window 10, "~home/user" directory locates at : 
## for WSL1
%localappdata%\lxss\home\{username}
C:\Users\{user}\AppData\Local\lxss\{username}
## for WSL2
\\wsl$\Ubuntu-20.04\home\{username}

## Hard drives: cd  /mnt/d/work/



II. Work on WSL

# Login root as default
open cmd:   ubuntu config --default-user root

# Update Ubuntu: sudo apt update && apt upgrade

# delete folder/file
sudo -r -f \folder

# install GCC
sudo apt-get install gcc-10 g++-10 gfortran-10

# install python
# check:   python --version
## set isntall python3 as default (similar for python2, )
sudo apt install python-is-python3 python-dev python3-dev

1. Insall Modules
sudo apt-get install tcl environment-modules

2. install cuda
2a. Install NVIDIA driver for WSL2
https://docs.nvidia.com/cuda/wsl-user-guide/index.html
460.20_gameready_win10-dch_64bit_international.exe

Note: Do not install any Linux display driver in WSL. The Windows Display Driver will install both the regular driver components for native Windows and for WSL support.

2b. Install cuda-toolkit for WSL2
https://docs.nvidia.com/cuda/wsl-user-guide/index.html
# Launch Ubuntu terminal

# First, set up the CUDA network repository. 

sudo apt-key adv --fetch-keys http://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/7fa2af80.pub
sudo sh -c 'echo "deb http://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64 /" > /etc/apt/sources.list.d/cuda.list'
sudo apt-get update
# Now install CUDA. Note that for WSL 2, you should use the cuda-toolkit-<version>
#(Do not choose the cuda, cuda-11-0, or cuda-drivers meta-packages under WSL 2)
sudo apt-get install -y cuda-toolkit-11-0
#test  (path: \\wsl$\Ubuntu-20.04\usr\local\cuda-11.1)
nvcc --version # cuda version

NOTE: gpu works with Window Insider Prebuild 20236 -->
        


3. OpenMPI + gcc +gpu
cd wSourceCode
tar xvf openmpi-4.1.0rc3.tar.gz
cd openmpi-4.1.0rc3
mkdir buildGCC-cuda && cd buildGCC-cuda
##
export myCUDA=/usr/local/cuda-11.0
../configure CC=gcc CXX=g++ FC=gfortran F77=gfortran \
--with-sge --without-verbs --without-ucx --with-cuda=${myCUDA} \
--prefix=/opt/app/openmpi/4.1.0-gcc9.3-cuda11
make
sudo make install

## use,
export PATH=/opt/app/openmpi/4.1.0-gcc9.3-cuda11/bin:$PATH
export LD_LIBRARY_PATH=/opt/app/openmpi/4.1.0-gcc9.3-cuda11/lib:$LD_LIBRARY_PATH

## make Module files (\\wsl$\Ubuntu-20.04\home\tha\1moduleFiles)
# module use /home/tha/1moduleFiles
set topdir /opt/app/openmpi/4.1.0-gcc9.3-cuda11 prepend-path PATH $topdir/bin prepend-path LD_LIBRARY_PATH $topdir/lib prepend-path INCLUDE $topdir/include prepend-path PKG_CONFIG_PATH $topdir/lib/pkgconfig # this is required

4. Compile lammps
### we have GTX 1060 with Pascal architecture
sudo apt install zlib1g-dev ocl-icd-opencl-dev  pkg-config
sudo apt-get install libblas-dev liblapack-dev libgsl-dev   #for plumed
##
git clone --branch master https://github.com/lammps/lammps.git lammps_master
cd lammps_master
git pull origin master
mkdir build-gpu && cd build-gpu

##
module load cmake-3.18.3
module load ompi/4.1.0-gcc9.3-cuda11
module load cuda-11.2
#--
export PATH=$PATH:/opt/app/openmpi/4.1.0-gcc9.3-cuda11/bin
export CC=mpicc
export CXX=mpic++
export FORTRAN=mpifort
cmake ../cmake -C ../cmake/presets/all_on.cmake \
-DLAMMPS_EXCEPTIONS=yes -DBUILD_MPI=yes -DBUILD_OMP=yes -DLAMMPS_MACHINE=mpi \
-DPKG_USER-OMP=yes -DPKG_USER-INTEL=no -DPKG_KOKKOS=yes \
 -DPKG_GPU=yes -DGPU_API=cuda -D GPU_ARCH=sm_61 \
-DPKG_USER-SMD=yes -DDOWNLOAD_EIGEN3=yes -DDOWNLOAD_VORO=yes \
-DPKG_KIM=no -DDOWNLOAD_KIM=no -DPKG_LATTE=no -DPKG_MSCG=no -DPKG_USER-ATC=no -DPKG_USER-MESONT=no  \
-DPKG_USER-ADIOS=no -DPKG_USER-NETCDF=no -DPKG_USER-QUIP=no -DPKG_USER-SCAFACOS=no \
-DPKG_USER-VTK=no -DPKG_USER-H5MD=no \
-DPKG_USER-PLUMED=yes -DDOWNLOAD_PLUMED=yes \
-DCMAKE_C_COMPILER=mpicc -DCMAKE_CXX_COMPILER=mpic++ -DCMAKE_Fortran_COMPILER=mpifort \
-DCMAKE_INSTALL_PREFIX=/opt/app/lammps/master-gpu
make -j 8
sudo make install
####################
-DGPU_API=cuda -D GPU_ARCH=sm_61



III. Install Anaconda on WSL:

    wget https://repo.continuum.io/archive/Anaconda3-2019.07-Linux-x86_64.sh
  • Run installation :  bash Anaconda3-2019.07-Linux-x86_64.sh
  • Update conda (optional):   conda update conda

Manually add the Anaconda bin folder to your PATH:

1. open file  "~/.bashrc"  (note: use Vim in Linux, donot open using Windows explorer)
(C:\Users\thang\AppData\Local\Packages\CanonicalGroupLimited.UbuntuonWindows_79rhkp1fndgsc\LocalState\rootfs\home\tha)

2. added :      export PATH=/home/tha/anaconda3/bin:$PATH

Test the installed python:


  • open Sub_Linux_Prompt, and type: which python

/home/tha/anaconda3/bin/python

II. Install Xeus-cling Kernel for Jupyter notebook on Window's subLinux (WSL)

3. Install Xeus-cling

- Assume Conda is installed
- open Sub_Linux_Prompt, and type:         conda install xeus-cling notebook -c QuantStack -c conda-forge
- Update xeus-cling (optional):    conda upate xeus-cling

4. set the BROWSER for WSL:

add the following command to the bottom of my ~/.bashrc file.

vim .bashrc 

export BROWSER='/mnt/c/Program Files (x86)/Google/Chrome/Application/chrome.exe'


NOTE: open .bashrc file using Windows may cause "permission error" on next launch. To solve it, type:
sudo chmod -R 777 /home/thang/.bashrc

5. Using:

  • open "Sub_Linux_Prompt" from an "arbitrary folder": Shift + Right Click and selecting the Open Linux shell here 
  • type: jupyter notebook
  • copy the last URL in Sub_Linux_Prompt to IE

Ref.:
https://github.com/QuantStack/xeus-cling/blob/master/README.md
https://libinruan.github.io/2018/11/06/Install-Jupyter-s-C-kernel-on-Windows-Subsystem-for-Linux/
https://gist.github.com/kauffmanes/5e74916617f9993bc3479f401dfec7da
https://www.howtoforge.com/vim-basics

Friday, November 8, 2019

OVITO

I. Ovito Interpreter

Packages are provided for the following platforms:
Operating systemArchitectureCPython versions
Windows64-bit (x86_64)3.7 / 3.8
Linux64-bit (x86_64)3.5 / 3.6 / 3.7 / 3.8

1. install Ovito with pure python: (require python3.7)

Note: Ovito interface encounters error when use with conda-jupyter. So use pip-jupyter on pure python3.7
make anaconda available in cmd windows, and can double click to open jypyter file
add 2 new environment variable path: Environment Variables --> System Variables --> path --> edit
C:\DevProgram\python37
C:\DevProgram\python37\Scripts

python -m pip install --upgrade pip
pip install tess ovito jupyter numpy scipy













Installing dependencies

sudo apt-get install build-essential git cmake-curses-gui qt5-default libboost-dev libqt5scintilla2-dev \
                     libavcodec-dev libavdevice-dev libavfilter-dev libavformat-dev libavresample-dev \
                     libavutil-dev libswscale-dev libnetcdf-dev libhdf5-dev libhdf5-serial-dev \
                     libbotan1.10-dev libmuparser-dev libqwt-qt5-dev python3-dev \
                     python3-sphinx python3-numpy python3-matplotlib python3-pyqt5 \
                     libfftw3-dev xsltproc docbook-xml



Getting the source code

Then download OVITO's source code into a new subdirectory named ovito/:
git clone https://gitlab.com/stuko/ovito.git ovito


Compiling OVITO

Within that directory, create a build directory and let CMake generate the Makefile:
cd ovito
mkdir build
cd build
cmake -DOVITO_BUILD_DOCUMENTATION=ON \
      -DCMAKE_BUILD_TYPE=Release \
      -DPYTHON_EXECUTABLE=/usr/bin/python3 \
      -DPYTHON_INCLUDE_DIR=/usr/include/python3.5m \
      -DPYTHON_LIBRARY=/usr/lib/x86_64-linux-gnu/libpython3.5m.so \
      ..
If this step fails, or if you want to disable certain components of OVITO, you can now run "ccmake ." to open the CMake configuration program. Changing the PYTHON_INCLUDE_DIR and PYTHON_LIBRARY settings may be needed to build OVITO against a different Python version.
Finally, compile OVITO:
make -j4
After a successful build, the executable can be found in the ovito/build/bin directory.

Thursday, November 7, 2019

some packages to update on Centos 6

Some packages need to update

1. install glibc-2.16 in CentOS 6.5

Some app need higher glibc. For ex. openmpi compiled on centos7 will cannot be run on centos6, but require GLIBC_2.16 to be installed on centos6

ERROR:    /lib64/libc.so.6: version `GLIBC_2.16' not found 

check the current glibc:
ldd --version
---------
ldd (GNU libc) 2.12

install: (Note: glibc-2.17 cause error and cannot run on centos6)
wget http://ftp.gnu.org/gnu/glibc/glibc-2.16.0.tar.gz
tar zxvf glibc-2.16.0.tar.gz 
cd glibc-2.16.0
mkdir build 
cd build

../configure --prefix=/uhome/p001cao/local/updateCentos6/glibc-2.16

make -j20
make install 

create module updateCentos6/glibc-2.16

Tuesday, September 10, 2019

Install PLUMED

compilers: PLUMED 2.4 requires a compiler that supports C++11: (need one among of the follows)
gcc 4.8.1      (need to install GCC)
clang 3.3
intel 15        (need to install Intel)

NOTE: Intel alone does not fully support C++11 if environment without gcc 4.8 or newer. Installing an external GCC is an solution for this, but this may cause some error when running Plumed due to cross-compiling.

tar xvzf plumed2-2.5.2.tar.gz

# download branch v2.5
git clone https://github.com/plumed/plumed2.git --branch=v2.5   plumed2-2.5.x
cd plumed2-2.5.x
git pull origin v2.5                           # or    git checkout v2.5

# download branch with PYCV
git clone --branch v2.6-pycv-devel  https://github.com/giorginolab/plumed2-pycv.git   plumed2-2.6pycv
cd    plumed2-2.6pycv

# download branch hack-the-tree      
git clone   --branch hack-the-tree    https://github.com/plumed/plumed2.git    plumed2-2.7htt
cd plumed2-2.7htt
git pull origin hack-the-tree

git pull origin master  

# or Clone a specific tag name using git clone: https://git-scm.com/docs/git-clone
git clone <url> --branch=<tag_name>

I. OMPI + Intel

1. USC1

module load mpi/openmpi4.0.2-Intel2019xe
module load intel/mkl-2019xe

check:  mpiicpc  -v                      (intel C++)
            mpicxx --version             ( gcc C++)

Notes: openMPI must be compile with gcc 4.8 or newer (load gcc/gcc-7.4.0 when compile openMPI)

# Install PLUMED

(to compile with mpi-enable, need to use compiler: CXX=mpic++   CC=mpicc)
chose modules to install: https://www.plumed.org/doc-v2.5/user-doc/html/mymodules.html
enable/disable modules:
./configure --enable-modules=+crystallization-colvar
./configure --enable-modules=all:-colvar-multicolvar 
BLAS and LAPACK Libs
a. separate compile Blas & Lapack
b. use Blas & Lapack from intel_mkl
LIBS="-mkl"
c. or use internal link: (blas & lapack is automatically built, need FORTRAN compiler)
--disable-external-blas --disable-external-lapack \

VMD trajectory plugins
https://www.plumed.org/doc-master/user-doc/html/_installation.html

Making lepton library faster
--enable-asmjit

#Configuring PLUMED
./configure --prefix=/uhome/p001cao/local/app/plumed2/2.6htt \
CXX=mpic++ LIBS="-mkl" \
--enable-openmp --enable-modules=all --enable-asmjit

# or
./configure --prefix=/uhome/p001cao/local/app/plumed2/2.6 \
CXX=mpic++ --disable-external-blas --disable-external-lapack \
--enable-openmp --enable-modules=all --enable-asmjit

# create Module file + test
prepend-path PATH $topdir/bin
prepend-path   PATH                $topdir/bin
prepend-path   LD_LIBRARY_PATH     $topdir/lib
prepend-path   INCLUDE             $topdir/include
prepend-path   PKG_CONFIG_PATH     $topdir/lib/pkgconfig          # this is required in order to Lammps can found Plumed

# test: 
module load plumed2/2.6.0
plumed help

# USC2:

module load mpi/ompi4.0.3-intel19u5
module load intel/compiler-xe19u5
module load intel/mkl-xe19u5
Configure

./configure CXX=mpic++ CC=mpicc  LIBS="-mkl" \
--enable-openmp --enable-modules=all --enable-asmjit \
--prefix=/home1/p001cao/local/app/plumed2/2.7htt 

make -j 8
make install

II. Install PLUMED using lMPI-2019xe

# 1. USC 1:
(use this, bc compilers are available for all clusters)
NOTE: intelMPI on eagle does not work, due to wrong path
    1. Module load:
    module load intel/compiler-xe19u5
    module load mpi/impi-xe19u5
    module load intel/mkl-xe19u5
    module load compiler/gcc/9.1.0
    module load conda/py37

    Configure
    ./configure CXX=mpiicpc CC=mpiicc LIBS="-mkl" \
    --enable-openmp --enable-modules=all --enable-asmjit \
    --prefix=/uhome/p001cao/local/app/plumed2/2.6httIMPI \

    # 2. USC 2:
    # impi2016 (not support openMP)
    module load mpi/intel-xe2016/impi-xe2016u4
    module load compiler/intel/xe2016u4
    check: $ which mpiicpc

    # or impi2019
    (source /home1/p001cao/local/app/intel/xe19u5/compilers_and_libraries_2019.5.281/linux/mpi/intel64/bin/mpivars.sh)
    module load mpi/impi-xe19u5
    module load intel/mkl-xe19u5
    module load compiler/gcc-9.2.0 
    source mpivars.sh
    Configure
    ./configure CXX=mpiicpc CC=mpiicc LIBS="-mkl" \
    --enable-openmp --enable-modules=all --enable-asmjit \
    --prefix=/home1/p001cao/local/app/plumed2/2.7htt-impi

    III. OMPI+ GCC

    don't use external "mkl" --> will cannot compile
    # USC 1
    module load mpi/ompi4.0.3-gcc9.2.0
    Configure
    ./configure CXX=mpic++ CC=mpicc \
    --enable-openmp --enable-modules=all --enable-asmjit \
    --prefix=/uhome/p001cao/local/app/plumed2/2.7htt-gcc
    ###############################


    # USC 2

    module load mpi/ompi4.0.3-gcc9.2.0
    module load tool_dev/binutils-2.32              # gold linker -->  must use the same linker with MPI
    Configure 
    ./configure CXX=mpic++ CC=mpicc LDFLAGS="-fuse-ld=gold -lrt" \
    --enable-openmp --enable-modules=all --enable-asmjit \
    --prefix=/home1/p001cao/local/app/plumed2/2.7htt-gcc

    ################
    module load compiler/gcc/7.4.0
    module load mpi/gcc-7.4.0/ompi/3.1.4
    Configure 
    ./configure CXX=mpic++ CC=mpicc \
    --enable-openmp --enable-modules=all --enable-asmjit \
    --prefix=/home1/p001cao/local/app/plumed2/2.7htt-gcc3

    IV. Install PLUMED using openmpi-4.0.1 + GCC-7.4.0 (CAN)

    1. Module load:

    module load mpi/openmpi4.0.1-gcc7.4.0
    module load gcc/gcc-7.4.0

    check:  mpic++ --version             ( gcc C++)

    2. Install PLUMED

    unzip plumed2-hack-the-tree.zip
    cd plumed2-hack-the-tree

    Configuring PLUMED: 
    ./configure --prefix=/home/thang/local/app/plumed2/2.6.0-gcc \
    CXX=mpic++ --disable-external-blas --disable-external-lapack \
    --enable-openmp --enable-modules=all

    V. Stuff commands

    1. See the the test input files of an action
    go to "regtest" folder of plumed source code, type command:

    grep <ACTION_NAME> */*/plumed.dat

    Ex: grep DISTANCE */*/plumed.dat

    don't use external "mkl" --> will cannot compile

    module load mpi/ompi4.0.3-clang10
    Configure
    ./configure CXX=mpic++ CC=mpicc \
    --enable-openmp --enable-modules=all --enable-asmjit \
    --prefix=/home1/p001cao/local/app/plumed2/2.7htt-clang

    III. Use MVAPICH2-GCC
    module load mpi/mvapich2-2.3.2-gcc9.2.0
    module load conda/py37mvapichSupp
    Configure 
    ./configure CXX=mpic++ CC=mpicc LDFLAGS="-fuse-ld=gold -lrt" \
    --enable-openmp --enable-modules=all --enable-asmjit \
    --prefix=/home1/p001cao/local/app/plumed2/2.7htt-mva

    Monday, July 29, 2019

    install UCX

    I. Install UCX

    ##################################
    UCX need to compile OpenMPI to use InfiniBand

    To use UCX you need to
    1. Get the recent release from https://github.com/openucx/ucx/releases
    2. Build and make it avaliable to your machines
    3. Configure OMPI --with-ucx="path-to-ucx" and rebuild/reinstall it
    Afterwards when you launch you set UCX pml:
    $ mpirun -mca btl self -mca pml ucx ....
    To control which device and what transport are being used you can add following env variables:
    $ mpirun -mca btl self -mca pml ucx -x UCX_NET_DEVICES=mlx5_0:1 -x UCX_TLS=rc,shm ....
    Try to experiment with different TLS's see here for more info.
    ##################################

    ##NOTE:
    * OpenMPI 4.0,3 support ucx 1.7 or older
    * OpenMPI 4.0,4 support newer ucx

    #### install from Source (work now, but should not be use to avoid runtime errors)
    ## Requirements: autoconf-2.69b, libtool-2.4.6, automake-1.14
    # git clone --branch master https://github.com/openucx/ucx  ucx-master
    cd ucx-master
    module load tool_dev/autoconf-2.69b
    module load tool_dev/automake-1.14
    module load tool_dev/libtool-2.4.6 
    export ACLOCAL_PATH=/home1/p001cao/local/app/tool_dev/libtool-2.4.6/share/aclocal
    ./autogen.sh
    mkdir build   &&  cd build

    ### (install from Release --> no need ./autogen.h)
    tar xvf ucx-1.10.1.tar.gz
    cd ucx-1.10.1
    mkdir build   &&  cd build
    ## USC2
    module load tool_dev/binutils-2.35              # gold
    module load compiler/gcc-10.3
    ##
    export PATH=$PATH:/home1/p001cao/local/app/compiler/gcc-10.3/bin
    export CC=gcc export CXX=g++ export FORTRAN=gfortran
    myKNEM=/home1/p001cao/local/app/tool_dev/knem-1.1.4
    myNUMA=/home1/p001cao/local/app/tool_dev/numactl-2.0.13
    ##--
    ../contrib/configure-release  --enable-mt --with-knem=$myKNEM \
    LDFLAGS="-fuse-ld=gold -lrt  -L$myNUMA/lib -Wl,-rpath,$myNUMA/lib" \
    CFLAGS="-I$myNUMA/include" \
    --prefix=/home1/p001cao/local/app/tool_dev/ucx-1.10


    ## USC1 (eagle)
    module load tool_dev/binutils-2.35              # gold 
    module load compiler/gcc-10.3
    ##
    export PATH=$PATH:/uhome/p001cao/local/app/compiler/gcc-10.3/bin
    export CC=gcc export CXX=g++ export FORTRAN=gfortran
    myKNEM=/uhome/p001cao/local/app/tool_dev/knem-1.1.4
    myNUMA=/uhome/p001cao/local/app/tool_dev/numactl-2.0.13
    ##--
    ../contrib/configure-release --enable-mt --with-knem=$myKNEM \
    LDFLAGS="-fuse-ld=gold -lrt  -L$myNUMA/lib -Wl,-rpath,$myNUMA/lib" \
    CFLAGS="-I$myNUMA/include" \
    --prefix=/uhome/p001cao/local/app/tool_dev/ucx-1.9


    #########################
    --disable-numa
    --with-rc --with-ud --with-dc --with-ib-hw-tm --with-dm --with-cm \
    ## consider options
    --with-verbs(=DIR)      Build OpenFabrics support, adding DIR/include,
                              DIR/lib, and DIR/lib64 to the search path for
                              headers and libraries
      --with-rc               Compile with IB Reliable Connection support
      --with-ud               Compile with IB Unreliable Datagram support
      --with-dc               Compile with IB Dynamic Connection support
      --with-mlx5-dv          Compile with mlx5 Direct Verbs support. Direct Verbs
                              (DV) support provides additional acceleration
                              capabilities that are not available in a regular
                              mode.
      --with-ib-hw-tm         Compile with IB Tag Matching support
      --with-dm               Compile with Device Memory support

    --with-cm               Compile with IB Connection Manager support

    ##-- Consider
    myNUMA=/home1/p001cao/local/app/tool_dev/numactl-2.0.13
    LDFLAGS="-fuse-ld=gold -lrt  -L$myNUMA/lib -Wl,-rpath,$myNUMA/lib" \
    CFLAGS="-I$myNUMA/include" \
    ##--
    # export myKNEM=/home1/p001cao/local/app/tool_dev/knem1.1.3    
    # export myOFI=/home1/p001cao/local/app/tool_dev/libfabric-1.10.1 
    --with-verbs=${myOFI} --with-knem=${myKNEM} \
    https://developer.arm.com/tools-and-software/server-and-hpc/help/porting-and-tuning/building-open-mpi-with-openucx/running-openmpi-with-openucx


    ####2. Intel
    module load intel/compiler-xe19u5
    export PATH=/home1/p001cao/local/app/intel/xe19u5/compilers_and_libraries_2019.5.281/linux/bin/intel64:$PATH
    export CC=icc  export CXX=icpc  export FORTRAN=ifort
    export LD_LIBRARY_PATH=/home1/p001cao/local/app/intel/xe19u5/compilers_and_libraries_2019.5.281/linux/compiler/lib/intel64_lin:$LD_LIBRARY_PATH

    export LD_LIBRARY_PATH=/home1/p001cao/local/app/tool_dev/glibc-2.18/lib:$LD_LIBRARY_PATH
    export LD_LIBRARY_PATH=/usr/local/lib

    export myKNEM=/home1/p001cao/local/app/tool_dev/knem1.1.3    
    export myOFI=/home1/p001cao/local/app/tool_dev/libfabric-1.10.1
    ##--
    ../contrib/configure-release --disable-numa --enable-mt LDFLAGS="-fuse-ld=lld -lrt" \
    --with-verbs=${myOFI} --with-knem=${myKNEM} \
    --prefix=/home1/p001cao/local/app/tool_dev/ucx-1.8-intel


    List of main transports and aliases

    alluse all the available transports.
    small shared memory transports.
    shmsame as "sm".
    ugniugni_rdma and ugni_udt.
    rcRC (=reliable connection), and UD (=unreliable datagram) for connection bootstrap.
    "accelerated" transports are used if possible.
    udUD transport, "accelerated" is used if possible.
    dcDC - Mellanox scalable offloaded dynamic connection transport
    rc_xSame as "rc", but using accelerated transports only
    rc_vSame as "rc", but using Verbs-based transports only
    ud_xSame as "ud", but using accelerated transports only
    ud_vSame as "ud", but using Verbs-based transports only
    tcpTCP over SOCK_STREAM sockets
    rdmacmUse RDMACM connection management for client-server API
    sockcmUse sockets-based connection management for client-server API
    cuda_copyUse cu\*Memcpy for hostcuda device self transfers but also to detect cuda memory
    gdr_copyUse GDRcopy library for hostcuda device self transfers
    cuda_ipcUse CUDA-IPC for cuda devicedevice transfers over PCIe/NVLINK
    rocm_copyUse for host-rocm device transfers
    rocm_ipcUse IPC for rocm device-device transfers
    selfLoopback transport to communicate within the same process

    II. UCX optional Libs

    1. rdma-core (fail)
    UCX detects the exiting libraries on the build machine and enables/disables support for various features accordingly. If some of the modules UCX was built with are not found during runtime, they will be silently disabled.
    • Basic shared memory and TCP support - always enabled
    • Optimized shared memory - requires knem or xpmem drivers. On modern kernels also CMA (cross-memory-attach) mechanism will be used.
    • RDMA support - requires rdma-core or libibverbs library.
    • NVIDIA GPU support - requires Cuda drives
    • AMD GPU support - requires ROCm drivers
    git clone https://github.com/linux-rdma/rdma-core  rdma-core
    cd rdma-core
    tar xvf rdma-core-30.0.tar.gz
    cd rdma-core-30.0
    ####
    module load compiler/gcc-10.1.0
    module load tool_dev/cmake-3.17.2 
    module load tool_dev/libnl-3.0
    module load tool_dev/libtool-2.4.6

    export LD_LIBRARY_PATH=/home1/p001cao/local/app/tool_dev/libnl-3.0/lib:$LD_LIBRARY_PATH ./build.sh



    2. libnuma-devel
    tar xzf numactl-2.0.13.tar.gz
    cd numactl-2.0.13
    ##--
    module load tool_dev/autoconf-2.69b
    ./autogen.sh
    #--
    mkdir build && cd build 
    ../configure --prefix=/home1/p001cao/local/app/tool_dev/numactl-2.0.13


    3. openMPI/UCX: libfabric ()
    wget https://github.com/ofiwg/libfabric/releases/tag/v1.11.1/libfabric-1.11.1.tar.bz2 
    If building directly from the libfabric git tree, run './autogen.sh' before the configure step.
    #--
    module load tool_dev/autoconf-2.69b
    ./autogen.sh
    ##
    tar -xvf libfabric-1.11.1.tar.bz2
    cd libfabric-1.11.1
    module load compiler/gcc-10.2  

    ## IB cluster
    ./configure --prefix=/uhome/p001cao/local/app/tool_dev/libfabric-1.11.1-IB

    ## noIB cluster
    ./configure --prefix=/uhome/p001cao/local/app/tool_dev/libfabric-1.11.1-noIB

    ## module 
    prepend-path PKG_CONFIG_PATH $topdir/lib/pkgconfig


    4. openMPI/UCX: KNEM
    tar zxvf knem-1.1.4.tar.gz 
    cd knem-1.1.4
    ./configure --prefix=/uhome/p001cao/local/app/tool_dev/knem-1.1.4

    5. openMPI/UCX: XPMEM
    --> cannot install: require linux kernel 4.x
    check: uname -a
    ##--
    tar zxvf xpmem-2.6.3.tar.gz
    cd xpmem-2.6.3
    ##--

    ./configure --prefix=/home1/p001cao/local/app/tool_dev/xpmem-2.6.2