Installing Gkeyll

“Mostly Painless” – Petr Cagas

Installation instructions for both the gkyl simulation framework and the postgkyl postprocessing tool are provided below.

gkyl install

There are two options for installing gkyl. One can install directly from the source code, or via the Conda package. Installing directly from the source is the preferred option, as this method gives users more control over the installation process. For many users who will wish to run gkyl on a cluster, which will have cluster-built versions of the Message Passing Interface (MPI) for parallel simulations, and potentially other gkyl depedencies, the source build will allow users to set the appropriate paths to the cluster installations of these dependencies.

Installing from source (preferred)

To install gkyl from source, first clone the GitHub repository using:

git clone https://github.com/ammarhakim/gkyl

Navigate into the gkyl directory to begin.

In many cases, an installation of gkyl will involve building most of gkyl’s dependencies only require a modern C/C++ compiler and Python 3. The full list of dependencies is:

  • C/C++ compiler with C++17 support (But NOT Clang >= 12.0 provided by Xcode 12)
  • Python 3 >=3.6
  • MPI compiler with MPI3 support (>=openmpi 3.0 or >=mpich 3.0)
  • LuaJIT 2.1.0
  • ADIOS 1.13.1 (But NOT >=ADIOS 2.0)
  • Eigen 3.3.7
  • CUDA Toolkit >=10.2 (if building with GPU support, NOT FULLY SUPPORTED)

The following instructions assume that at minimum the user has both a C/C++ compiler with C++17 support and Python 3.

Installing from source manually

If machine files are not available, the dependencies, configuration, and build can be done manually.

The first step is to build the dependencies. Depending on your system, building dependencies can be complicated. On a Mac or Linux machine you can simply run the mkdeps.sh script in the install-deps directory. To build dependencies cd to:

cd gkyl/install-deps

First, please check details by running:

./mkdeps.sh -h

On most supercomputers you will likely need to use the system recommended compilers and MPI libraries. In this case, you should pass the appropriate compilers to mkdeps.sh as follows:

./mkdeps.sh CC=cc CXX=cxx MPICC=mpicc MPICXX=mpicxx ....

You should only build libraries not provided by the system. In practice, this likely means LuaJIT, ADIOS and, perhaps Eigen. (Many supercomputer centers at DOE already offer ADIOS builds and should be preferred instead of your own builds). A typical command will be:

./mkdeps.sh --build-adios=yes --build-luajit=yes --build-eigen=yes

(in addition, you may need to specify compilers also).

By default, the mkdeps.sh script will install dependencies in $HOME/gkylsoft directory. If you install it elsewhere, you will need to modify the instructions below accordingly.

Once you have all dependencies installed, you can build gkyl itself by cd-ing to the top-directory in the source. gkyl uses the Waf build system. You do NOT need to install waf as it is included with the distribution. However, waf depends on Python (included on most systems). Waf takes a number of options. To get a list do

./waf --help

There are two build scenarios: first, all dependencies are installed in $HOME/gkylsoft directory, and second, you are using some libraries supplied by your system.

If you have installed all dependencies in the gkylsoft directory you can simply run:

./waf configure CC=mpicc CXX=mpicxx

where CC and CXX are names of the MPI compilers to use. Note that in some cases the full path to the compiler may need to be specified. If the compilers are already in your path, then you can omit all flags.

If you need to use system supplied builds, you need to specify more complex set of paths. Although you can do this by passing options to the waf build script, it is easiest to follow these steps:

  • Copy the configure-par.sh-in script to configure-par.sh
  • Modify this script to reflect the locations of various libraries on your machine. In particular, if you are using pre-built libraries you will likely need to change the information about MPI and ADIOS.
  • Run the configure-par.sh script

Once the configuration is complete, run the following command to build and install (note: if you are working on a cluster and using environment modules, you may need to load them at this point):

./waf build install

The builds are created in the ‘build’ directory and the executable is installed in $HOME/gkylsoft/gkyl/bin, unless you specified a different install prefix. The executable can only be run from the install directory [1].

If you need to clean up a build do:

./waf clean

If you need to uninstall do:

./waf uninstall

LuaJIT builds easily on most machines with standard GCC compiler. Often, you may run into problems on older gcc as they do not include the log2 and exp2 functions unless c99 standard is enabled. To get around this, modify the src/Makefile in LuaJIT. To do this, change the line:

CC= $(DEFAULT_CC)

to:

CC= $(DEFAULT_CC) -std=gnu99

To build on Mac OS X Mojave and beyond you must set the following env flag:

export MACOSX_DEPLOYMENT_TARGET=10.YY

where YY is the version number of the OSX operating system. For example, to build on OS X Mojave the env flag is:

export MACOSX_DEPLOYMENT_TARGET=10.14

and to build on OS X Catalina the env flag is:

export MACOSX_DEPLOYMENT_TARGET=10.15

Installing with Conda

The gkyl package is also available to be installed via Conda, although this gives less flexibility for keeping the code up-to-date as gkyl development continues. Once Conda is installed and added to the PATH, gkyl can be obtained with:

conda install -c gkyl gkeyll

Note, that this will also install all dependencies into the Conda install directory. Often this may lead to some conflicts, particularly for the MPI installation, specially if there is another version of MPI already located in the system. gkyl should be run using the MPI provided by Conda.

In general, having Conda and source-built gkyl on the same machine can cause confusion. In that case please use explicit paths to the mpiexec and gkyl executable you wish to use when running simulations.

Troubleshooting

Having trouble building? We will try to compile a list of suggestions and common error messages in this troubleshooting site.

Footnotes

[1]The reason for this is that gkyl is in reality a LuaJIT compiler extended with MPI. Hence, for the compiler to find Lua modules (i.e. gkyl specific code) certain paths need to be set which is done relative to the install location.

Postgkyl install

There are two options for getting Postgkyl. The first option is to get the Conda package and the second is to clone the source code repository. Installing via Conda is preferred for the majority of users as it requires literally a single Conda command. What is more, Conda is already the suggested way of installing all the dependencies. On the other hand, the later option has an advantage of always having the most up-to-date version and is generally required for users that want to contribute to the code.

Installing with Conda (preferred for non-developers)

Postgkyl can be installed with Conda with literally a single command:

conda install -c gkyl postgkyl

Note that the flag for the Gkeyll channel, -c gkyl, is required even for updating. However, it can be permanently added.

conda config --add channels gkyl
conda install postgkyl

Updates can be downloaded with:

conda update -c gkyl postgkyl

Note

To install a new package, users need the write permission for the Anaconda directory. If this is not the case (e.g. on a computing cluster), one can either create a Conda environment (see tip below) or install Conda into the $HOME directory.

Tip

To create a Conda environment for postgkyl, use

conda create -n pgkylenv python=3

Then activate the environment with

conda activate pgkylenv

and install postgkyl using the commands above (or the ones below to install from source).

After install, one must always have the pgkylenv environment activated in order to use postgkyl.

Installing from source (preferred for developers)

Postgkyl source code is hosted in a GitHub repository. To get Postgkyl running, one first needs to clone the repository and install dependencies.

First, clone the repository using:

git clone https://github.com/ammarhakim/postgkyl

Postgkyl has these dependencies, which are readily available thru Conda:

All these dependencies can be easily obtained from the Gkeyll Conda channel, via

conda install -c gkyl postgkyl --only-deps

Once the dependencies are installed, postgkyl can be installed by navigating into the postgkyl repository and running

python setup.py install
python setup.py develop

Note that these commands only ever need to be run once (even if one is modifying source code). Changes to the source code will be automatically included because we have installed in development mode.

Building adiospy from source

Adios can also be built manually from the source code. Note that for the manual build, Adios needs to be already installed and its bin directory added to the PATH (the default Gkeyll location is ~/gkylsoft/adios/bin/). The standard location for the wrapper in the Gkeyll installation is gkyl/install-deps/adios-x.x.x/wrappers/numpy/. After navigating to that directory, build and install adiospy via

make python
python setup.py install

This currently does not work out of the box with the clang compiler because of a deprecated library. This can be overcome removing the -lrt flag from the line 33 of the Makefile. The edited lines 32 and 33 should look like this:

adios.so:
        python setup.py build_ext

This will allow to complete the adiospy build successfully and it has no know consequences for Postgkyl.

Switching from Conda version to repository

While the Conda build of Postgkyl is the suggested version for the majority of users, the source code repository is required for any code contributions. We should stress that when switching between the different version, it is strongly advised to remove the other version. Having both may lead to an unforeseen behavior based on the relative order of components in the PATH and PYTHONPATH.

The Conda version can be uninstalled with:

conda uninstall postgkyl

Note on the Windows Linux Subsystem (WSL)

Historically, Gkeyll was supported only for Unix-like operating systems. However, since the Windows 10 Anniversary Update in August 2016, it became possible to run Gkeyll even on Windows although with reduced performance. This changed with the introduction of WSL 2 in Spring/Summer 2020. With WSL 2, the performance on Unix and Windows is indistinguishable.

Requirements:

  • For x64 systems: Version 1903 or higher, with Build 18362 or higher
  • For ARM64 systems: Version 2004 or higher, with Build 19041 or higher
  • For WSL 2 (recommended): Build 18362 and a CPU supporting Hyper-V virtualization

Enabling and installing WSL

First, WSL needs to be enabled. This can be done either using GUI or PowerShell. For the former, go to Turn Windows features on or off in the Control Panel and check Windows Subsystem for Linux at the very bottom.

_images/wf.png

Alternatively, this can be done in a terminal (running the PowerShell as an Administrator):

dism.exe /online /enable-feature /featurename:Microsoft-Windows-Subsystem-Linux /all /norestart

The Linux distribution itself is available thru Microsoft Store. Currently, the following flavors are available:

It is strongly recommended to then enable WSL 2 for much better performance. First, enable virtualization in Administrator PowerShell

dism.exe /online /enable-feature /featurename:VirtualMachinePlatform /all /norestart

Download and install the kernel update package. And finally, set WSL 2 as the default (again using PowerShell as Administrator).

wsl --set-version <distribution name> 2
wsl --set-default-version 2

Interaction between Windows and Linux; GUI

As of September 2020, WSL2 does not directly support GUI; however, this is currently supposed to be in the works. Until the official release, this can be overcome with a 3rd party X-server like VcXsrv (this is our recommended option as the other option does not seem to work on some configurations) or Xming. Note that when using VcXsrv, the Disable access control checkbox needs to be marked when setting XLaunch. Otherwise, the X11 forwarding would not work properly.

_images/xlaunch.png

Finally, the $DISPLAY environmental variable needs to be set up on the Linux side. In WSL 1 this was as simple as

export DISPLAY=:0

WSL 2, however, uses virtualization and the forwarding address changes which each launch of the system. The address is stored in /etc/resolv.conf. The $DISPLAY can be set up with:

export DISPLAY=$(cat /etc/resolv.conf | grep nameserver | awk '{print $2}'):0

Another issue might be caused by tools that want to open a browser, for example, the commonly used jupyter-lab. This can be easily overcome by launching the server without a browser using jupyter-lab --no-browser and then copying the address including the token to a Windows browser.

Finally, it is often useful to access the Linux files from Windows. Using WLS 2, the Linux root, /, is located at \wsl$\<DISTRIBUTION_NAME>.

Known issues

There is currently a known issue where Windows and Linux clocks might get desynchronized when the computer sleeps. This might cause issues with Git and update installation using sudo apt update. There is a workaround that works until this issue gets patched and that is manually calling sudo hwclock -s to manually synchronize the time.