Installing mpi4py & Voro++

I. Install mpi4py

1. on Window

Note: 

- install Visual Studio Build Tools at this link: http://go.microsoft.com/fwlink/?LinkId=691126&fixForIE=.exe

- download Microsoft MPI v10.1, install both msmpisdk.msi and msmpisetup.exe

https://docs.microsoft.com/en-us/message-passing-interface/microsoft-mpi

#check: mpiexec -help

1.1a. install mpi4py with conda: 

# for intel MPI 

pip install mpi4py impi

1.1b.  install mpi4py on Python without Conda: (being used)
install another python3.7 independent to Conda
https://www.python.org/downloads/windows/
pip install mpi4py numpy scipy 

pip search mpi 
pip install impi

# register mpiexec
mpiexec -register
windows user & password

test:       mpiexec -np 2 python -m mpi4py.bench helloworld                       

1.2 Run:

- with anaconda: open Anaconda_Prompt
 $ mpiexec -np 8 python script.py

- without anaconda: open cmd as administrator
 $ mpiexec -np 8 python script.py

2. on Linux

Install Conda

make sure mpi4py link to right MPI lib (OpenMPI or MPICH,  or Impi), then use right command to run: 

mpirun -np 5 python
# or
mpiexec -np 5 python

2.1. Install with installing conda-mpi:  

to chose right version mpi4py, have to attach option [-c CHENNAL] in conda install,
    conda-forge
    intel
    bioconda
    anaconda

Note: conda-forge contains both version of mpi4py for openmpi and mpich
conda search -c conda-forge mpi4py     # find package
conda search -c intel       mpi4py   
conda search -c anaconda    mpi4py   

conda install [-c chennal] <package_name>=<version>=<build_string>

conda create --name new_name --clone old_name
conda remove --name old_name --all

# for mpi4py with OpenMPI: (being used)
module load conda/conda3

conda create  -n  py37ompi python=3.7 scipy numpy scikit-learn

source activate   py37ompi
pip install tess ovito                  # voro++ 

conda install -c conda-forge mpi4py=3.0.3=py37hd0bea5a_0

# for mpich 

conda create --name py37mpich --clone py37ompi

source activate    py37mpich
conda uninstall mpi4py mpi

conda install -c conda-forge mpi4py=3.0.3=py37hcf07815_0  

# for intel MPI (just support python 3.6)
conda create  -n  py36impi python=3.6 scipy numpy scikit-learn
source activate    py36impi 
conda install -c intel mpi4py=3.0.0=py36_intel_0 
pip install tess ovito 

TEST:
mpirun -np 5 python -m mpi4py.bench helloworld
## ------
Hello, World! I am process 0 of 5 on leopard.
Hello, World! I am process 1 of 5 on leopard.
Hello, World! I am process 2 of 5 on leopard.
Hello, World! I am process 3 of 5 on leopard.

Hello, World! I am process 4 of 5 on leopard.

NOTE: there are 3 env for python with include conda-mpi: py36mpi, py37mpi, py27mpi
 - but load conda-mpi maybe cause unexpected conflict with other MPI. So consider to install mpi4py only, without include conda-mpi --> use pip install

2.2. Install without installing conda-mpi:  (being used)

Note: using conda to install mpi4py, it also installsconda-mpi which cannot control and may conflict with other MPI. To use openMPI we want, must use pip install: (but this normally fail to link MPI compiler with python36, work with python37)
module load conda/conda3

conda create  -n  py37 python=3.7 scipy numpy 

source activate   py37

pip install mpi4py tess

TEST: (this word on Centos7)
module load mpi/openmpi4.0.2-Intel2019xeU4
module load conda/py37

mpirun -np 5 python -m mpi4py.bench helloworld
## ------

TEST: ( on Centos6 --> glibc error)
module load mpi/openmpi4.0.2-Intel2019xe     
module load conda/py37

mpirun -np 5 python -m mpi4py.bench helloworld

## ------

II. Install Voro++

module load conda3

source activate py37 

pip install     tess                                           #  voro++ library

Ref : https://github.com/abria/TeraStitcher/wiki/Multi-CPU-parallelization-using-MPI-and-Python-scripts
https://oncomputingwell.princeton.edu/2018/11/installing-and-running-mpi4py-on-the-cluster/

Pure python & envs using Pip

I. Python+pip on Linux:

https://www.digitalocean.com/community/tutorials/how-to-install-python-3-and-set-up-a-local-programming-environment-on-centos-7

1. Need OpenSSL to avoid error: (and configure ./configure --with-ssl)

wget https://www.openssl.org/source/openssl-1.1.1g.tar.gz
tar -xf openssl-1.1.1g.tar.gz
cd openssl-1.1.1g

./config --prefix=/home1/p001cao/local/app/tool_dev/openssl \
--openssldir=/home1/p001cao/local/app/tool_dev/openssl

make -j 8 

make install 

2. Need "libffi" to avoid "_ctypes" error

wget https://github.com/libffi/libffi/releases/download/v3.3/libffi-3.3.tar.gz

tar -xf libffi-3.3.tar.gz
cd libffi-3.3

./configure --prefix=/home1/p001cao/local/app/tool_dev/libffi-3.3

Download 

export PYTHON_VERSION=3.7.5

curl -O https://www.python.org/ftp/python/${PYTHON_VERSION}/Python-${PYTHON_VERSION}.tgz tar -xvzf Python-${PYTHON_VERSION}.tgz cd Python-${PYTHON_VERSION}

Build and install Python#

Configure, make, and install Python:

./configure \ --prefix=/uhome/p001cao/local/app/python/${PYTHON_VERSION} \ --enable-shared \ --enable-ipv6 \ LDFLAGS=-Wl,-rpath=/uhome/p001cao/local/app/python/${PYTHON_VERSION}/lib,--disable-new-dtags

make -j 8 

make install 

REF; https://docs.rstudio.com/resources/install-python-source/

create module python3

set topdir /home1/p001cao/local/app/tool_dev/python37
module load openSSL

prepend-path    PATH                    $topdir/bin

prepend-path    INCLUDE         $topdir/include

prepend-path    LD_LIBRARY_PATH         $topdir/lib

Test: 
module load tool_dev/python37
python3 -V

python3 -m pip install --upgrade pip
python3 -m pip install --upgrade pip --trusted-host pypi.org --trusted-host files.pythonhosted.org

2. Setting Up a Virtual Environment

https://packaging.python.org/guides/installing-using-pip-and-virtual-environments/

Setting up a programming environment provides us with greater control over our Python projects and over how different versions of packages are handled. 

venv (for Python 3) and virtualenv (for Python 2) allow you to manage separate package installations for different projects. They essentially allow you to create a “virtual” isolated Python installation and install packages into that virtual installation. When you switch projects, you can simply create a new virtual environment and not have to worry about breaking the packages installed in the other environments.

Choose which directory you would like to put your Python programming environments in, or create a new directory in installing dir of python:

cd  /uhome/p001cao/local/python/Python37

mkdir enviroments

cd  enviroments

module load python/python37

python3 -m venv    py37impi

source py37impi/bin/activate

create module python/python37

set               topdir            /uhome/p001cao/local/python/Python37/enviroments/py37impi
module load openSSL
prepend-path    PATH                    $topdir/bin
prepend-path    LD_LIBRARY_PATH         $topdir/lib
prepend-path    INCLUDE                 $topdir/include

if { [ module-info mode load ] } {
    puts stdout "source $topdir/bin/activate;"
} elseif { [ module-info mode remove ] } {
    puts stdout "deactivate"
}

2. Install packages:

load corresponding envs:
module load python/py37impi

python3 -m pip install --upgrade pip

pip3 install tess ovito numpy scipy matplotlib mpi4py impi




III. Some problems:
SSL:
https://joshspicer.com/python37-ssl-issue

wget https://www.openssl.org/source/openssl-1.0.2q.tar.gz
tar xvf openssl-1.0.2q.tar.gz
cd   openssl-1.0.2q
./config  --prefix=/uhome/p001cao/local/openssl-1.0.2
make
make install

create module openSSL

set               topdir           /uhome/p001cao/local/openssl-1.0.2

prepend-path    PATH                    $topdir/bin

prepend-path    LD_LIBRARY_PATH         $topdir/lib

prepend-path    INCLUDE                 $topdir/include

IV. Install package from source

cd sourceFolder

pip install -e .

Numpy.savetxt

1   Saving numpy array to txt file row wise