# Compilation

## CEMAC (recommended)

### Pre-built executables

WRFChem has been built with all compiler and MPI combinations on ARC4 here:
```bash
/nobackup/cemac/software/build/WRFChem/
```
The pre-built executables are found in these locations:
```bash
/nobackup/cemac/software/apps/WRFChem/
```

### Custom executables

To build your own WRFChem executables:

- First create the empty build directories.
- Then copy the `build.sh` and the WRFChem .tar.gz file for the version of choice (e.g. 4.2 below).
- Then run the build script as below.
- This copies over the code, builds everything, puts the executables in `software/apps/WRFChem/`, and hardlinks in the correct NetCDF libraries to avoid accidentally pointing to the wrong NetCDF libraries (e.g. from conda) through `/nobackup/WRFChem/build_scripts/linknchdf5.sh`.
- When finished, update `WRFotron/config.bash` to direct to this new build, remove the WRFChem module from being loaded, and copy over the manual blueprints as `pre.bash`, `main.bash`, and  `main_restart.bash` (reasons why given below in Manual compilation).  
- Optional: Can then add any custom edits and [manually recompile](compilation.html#compile-wps-wrfmeteo-and-wrfchem).
```bash
cd /nobackup/${USER} # replace ${USER} with your username
mkdir -p software/build/WRFChem/4.2/1
mkdir -p software/build/WRFChem/4.2/src
cd software/build/WRFChem/4.2/1
cp /nobackup/WRFChem/build_scripts/4.2/build.sh .
cp /nobackup/cemac/software/build/WRFChem/4.2/src/WRFChem4.2.tar.gz ../src/
./build.sh 

# go to your WRFotron and update config.bash to point to this new build, e.g.:
# cd /nobackup/${USER}/WRFotron
# gedit config.bash # or any other text editor
# WRFdir=/nobackup/${USER}/software/build/WRFChem/4.2/1/intel-19.0.4-openmpi-3.1.4/WRFChem4.2/WRFChem4.2 # example here for WRFChem, the exact path will be specific to your build(s)

# in config.bash remove WRFchem/4.2 from the module load line, e.g.:
# module load intel/19.0.4 openmpi/3.1.4 ncl/6.5.0 nco/4.6.0 wrfchemconda/3.7 sge

# from the namelists folder, copy over the manual build pre.bash and main.bash, e.g.:
# cp namelists/pre.bash.blueprint_manual pre.bash
# cp namelists/main.bash.blueprint_manual main.bash
# cp namelists/main_restart.bash.blueprint_manual main_restart.bash
```
To build and use a custom preprocessor:

- First copy over the default preprocessor code from `/nobackup/WRFChem` (e.g. anthro_emis).
- Then copy over the makefile modifier to the same folder.
- Then add your custom edits to the preprocessor.
- Then create the custom preprocessor.
- When finished, update `WRFotron/config.bash` to direct to this new custom preprocessor.
```bash
cd /nobackup/${USER} # replace ${USER} with your username
cp -r /nobackup/WRFChem/anthro_emis .
cd anthro_emis
cp /nobackup/WRFChem/build_scripts/fix_makefiles.sh .
. fix_makefiles.sh
# make your custom edits
make_anthro
# update WRFotron/config.bash to point to this new processor
```

### Misc

To always be able to view and use all the software CEMAC has built when you run `module avail`, add the following lines to `.bashrc`:   
```bash
if [ -r /nobackup/cemac/cemac.sh ] ; then
  . /nobackup/cemac/cemac.sh
fi
```
The recommended compiler and MPI combination is:
```bash
compiler: intel
mpi: openmpi
```
```{note}
IntelMPI on ARC4 is not optimized and contains a bug. Run the following command to run smoothly with IntelMPI: `export I_MPI_HYDRA_TOPOLIB=ipl`
```

## Manual (alternative)

### Setup

Download WRFotron, WRFChem, make a copy for WRFMeteo without the chemistry folder, download WPS, download [WPS Geography files](https://www2.mmm.ucar.edu/wrf/users/download/get_sources_wps_geog.html):
```bash
cd /nobackup/${USER}
git clone https://github.com/wrfchem-leeds/WRFotron.git
git clone https://github.com/wrf-model/WRF.git WRFChem4.2
git clone https://github.com/wrf-model/WPS.git WPS4.2

cp -r WRFChem4.2 WRFMeteo4.2
cd WRFMeteo4.2
rm -rf chem
```
Or copy these folders over from `/nobackup/WRFChem`:
```bash
cd /nobackup/${USER}
cp -r /nobackup/WRFChem/{WRFotron,WRFChem4.2,WRFMeteo4.2,WPS4.2,WPSGeog4} .
```
You will need to remove, or at minimum, change the module load line at the top of `config.bash`. The modules intel, openmpi, and WRFchem are for the CEMAC installation, and keeping these (and potentially others) can interfere with executables. These need to be removed. NCL, NCO, and conda can be used from CEMAC for manual runs, or you could have your own personal conda environments with NCL and NCO (see below). You can see the manual blueprint in the repository: [config.bash.blueprint_manual](https://github.com/wrfchem-leeds/WRFotron/blob/master/namelists/config.bash.blueprint_manual).
```bash
# cemac compilation uses
module load intel/19.0.4 openmpi/3.1.4 WRFchem/4.2 ncl/6.5.0 nco/4.6.0 wrfchemconda/3.7 sge

# for manual compilation remove (at least) intel, openmpi, and WRFchem
module load ncl/6.5.0 nco/4.6.0 wrfchemconda/3.7 sge
```
The executables within `pre.bash` need to be copied over directly, rather than just linked which is adequate for the CEMAC method. To do this make both of the following replacements. You can see the manual blueprint in the repository: [pre.bash.blueprint_manual](https://github.com/wrfchem-leeds/WRFotron/blob/master/namelists/pre.bash.blueprint_manual).
```bash
# on line 21, replace:
for aFile in util geogrid ungrib link_grib.csh metgrid
# with:
for aFile in util geogrid geogrid.exe ungrib ungrib.exe link_grib.csh metgrid metgrid.exe

# and then on line 80, replace:
cp -r ${WRFdir}/run/* .
# with:
cp -r ${WRFdir}/run/* .
rm *.exe
cp -r ${WRFdir}/main/*.exe .
cp -r ${WRFmeteodir}/main/wrf.exe wrfmeteo.exe
```
All executables and preprocessors will need to have `./` before them to execute. This includes `ungrib.exe`, `geogrid.exe`, `metgrid.exe`, `real.exe`, `megan_bio_emiss`, `wesely`, `exo_coldens`, `anthro_emiss`, `fire_emis`, and `mozbc` in `pre.bash`. Also, `wrfmeteo.exe` and `wrf.exe` in `main.bash`. Also, `wrf.exe` in `main_restart.bash`. You can see the manual blueprints in the repository: [pre.bash.blueprint_manual](https://github.com/wrfchem-leeds/WRFotron/blob/master/namelists/pre.bash.blueprint_manual), [main.bash.blueprint_manual](https://github.com/wrfchem-leeds/WRFotron/blob/master/namelists/main.bash.blueprint_manual), and [main_restart.bash.blueprint_manual](https://github.com/wrfchem-leeds/WRFotron/blob/master/namelists/main_restart.bash.blueprint_manual).
Add links to the preprocessor executables `anthro_emis`, `fire_emis`, and `mozbc` by adding the following code. You can see the manual blueprints in the repository: [pre.bash.blueprint_manual](https://github.com/wrfchem-leeds/WRFotron/blob/master/namelists/pre.bash.blueprint_manual).
```bash
ln -s ${WRFanthrodir}/anthro_emis . # section 4.a, line 148
ln -s ${WRFfiredir}/fire_emis .     # section 4.b, line 164
ln -s ${WRFMOZARTdir}/mozbc .         # section 6,   line 186
```
Download flex (tool for generating scanners: programs which recognize lexical patterns in text).  
[Download](https://www2.acom.ucar.edu/wrf-chem/wrf-chem-tools-community) and compile (in serial) preprocessors:  
- anthro_emis (anthropogenic emissions preprocessor).  
- fire_emiss (fire emissions preprocessor).  
- megan (biogenic emissions preprocessor).  
- mozbc (preprocessor for lateral boundary and initial conditions).  
- wes-coldens (exocoldens and season_wesely, O<sub>2</sub> and O<sub>3</sub> column densities and dry deposition).  
- Check preprocessors have the correct modules and libraries linked via: `ldd preprocessor`.  
```bash
conda deactivate # maybe multiple times
module purge
module load intel netcdf openmpi
export NETCDF=$(nc-config --prefix)
export NETCDF_DIR=$NETCDF
export FC=ifort

./make_anthro

./make_fire_emis

./make_util megan_bio_emiss

./make_mozbc

./make_util wesely

./make_util exo_coldens
```

### Conda
Download the latest [miniconda](https://docs.conda.io/en/latest/miniconda.html):
```bash
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
```
Run bash script, read terms, and set path:
```bash
bash Miniconda3-latest-Linux-x86_64.sh
```
Create conda environment with Python 3 (with some libraries for analysis), NCL, and NCO:  
```bash
conda create -n python3_ncl_nco -c conda-forge xarray salem xesmf numpy scipy pandas matplotlib rasterio affine ncl nco wrf-python dask geopandas descartes
```
To activate/deactivate conda environment:  
```bash
conda activate python3_ncl_nco
conda deactivate
```
For more information on conda, [see here](https://docs.conda.io/projects/conda/en/latest/user-guide/index.html).  

Create separate environments for downloading ECMWF data (requires Python 2) and ncview, which you can then load temporarily to execute these functions:  
```bash
conda create -n python2_ecmwf -c conda-forge ecmwf-api-client 
conda create -n ncview -c eumetsat -c conda-forge ncview libpng
```

### Compile WPS, WRFMeteo, and WRFChem
Setup:
```bash
# compiler setup
COMPILER_VER='intel:19.0.4'
MPI_VER='openmpi:3.1.4' # could use intelmpi instead of openmpi, and then also need: export I_MPI_HYDRA_TOPOLIB=ipl
CMP=${COMPILER_VER%:*}
CMP_VER=${COMPILER_VER#*:}
MP=${MPI_VER%:*}
MP_VER=${MPI_VER#*:}
FLAVOUR="${CMP}-${CMP_VER}-${MP}-${MP_VER}"

# modules
conda deactivate # maybe multiple times if many environments activated
module purge
module load licenses sge ${CMP}/${CMP_VER} ${MP}/${MP_VER} netcdf hdf5 patchelf

# environment variables - shell
NETCDF=$(nc-config --prefix)
NETCDF_DIR=$NETCDF
YACC='/usr/bin/yacc -d'
FLEX_LIB_DIR='/nobackup/WRFChem/flex/lib'
LD_LIBRARY_PATH=$FLEX_LIB_DIR:$LD_LIBRARY_PATH
JASPERLIB='/usr/lib64'
JASPERINC='/usr/include'

# environment variables - WRFChem
WRF_EM_CORE=1 # selects the ARW core
WRF_NMM_CORE=0 # ensures that the NMM core is deselected
WRF_CHEM=1 # selects the WRF-Chem module
WRF_KPP=1 # turns on Kinetic Pre-Processing (KPP)
WRFIO_NCD_LARGE_FILE_SUPPORT=1 # supports large wrfout files

# export variables
export FC CC NETCDF NETCDF_DIR YACC FLEX_LIB_DIR LD_LIBRARY_PATH JASPERLIB JASPERINC
export WRFIO_NCD_LARGE_FILE_SUPPORT WRF_KPP WRF_CHEM WRF_NMM_CORE WRF_EM_CORE
```
WRFChem compilation:
- HPC option will be specific to your HPC architecture.
- ARC4 = 15 = INTEL (ifort/icc) (dmpar) e.g. Distributed-Memory Parallelism MPI.
- Compile for basic nesting: option 1.
- Compile real (as oppose to ideal simulations).
- Thousands of messages will appear. Compilation takes about 20-30 minutes.
- How do you know your compilation was successful? If you have all four `main/*.exe`.
- Check the executables have all relevant linked libraries: `ldd main/wrf.exe`.
```bash
cd /nobackup/${USER}/WRFChem
./clean -a
./configure # 15 for intel (ifort/icc) (dmpar) hpc architecture, 1 for basic nesting

./compile em_real >& log.compile 
```
WPS compilation (requires a successfully compiled WRF):
- HPC option will be specific to your HPC architecture.
- ARC4 = 17 = INTEL (ifort/icc) (serial).
- Sometimes `configure.wps` can assign the incorrect path to WRFChem, check and edit if required.
- How do you know your compilation was successful? If you have `geogrid.exe`, `metgrid.exe`, and `ungrib.exe`.
- Check the executables have all relevant linked libraries: `ldd geogrid.exe`.
```bash
cd /nobackup/${USER}/WPS
./clean -a
./configure # 17 for intel (ifort/icc) (serial) hpc architecture

gedit configure.wps
WRF_DIR="/nobackup/${USER}/WRFChem"

./compile >& log.compile
```
WRFMeteo compilation:
- Deselect the WRFChem module.
- HPC option will be specific to your HPC architecture.
- ARC4 = 15 = INTEL (ifort/icc) (dmpar).  
- Compile for basic nesting: option 1.
- Compile real (as oppose to ideal simulations).
- Thousands of messages will appear. Compilation takes about 20-30 minutes.
- How do you know your compilation was successful? If you have all four `main/*.exe`.
- Check the executables have all relevant linked libraries: `ldd main/wrf.exe`.
```bash
export WRF_CHEM=0

cd /nobackup/${USER}/WRFMeteo
./clean -a
./configure # 15 for intel (ifort/icc) (dmpar) hpc architecture, 1 for basic nesting

./compile em_real >& log.compile
```
Preprocessors:
- If make any changes to preprocessors then they require recompilation.
- Raw preprocessors downloaded from [here](http://www.acom.ucar.edu/wrf-chem/download.shtml).
- Ensure the setup is the same as above for manual compilation of WPS/WRFChem/WRFMeteo.
  - May need to check if preprocessor requires a different module version that currently compiled with.
  - If Makefile cannot locate the correct NetCDF path, may need to add `-lnetcdff`.
  - Note for wes_coldens: FC hardcoded in `make_util`.

If need JASPER:
```bash
wget http://www2.mmm.ucar.edu/wrf/OnLineTutorial/compile_tutorial/tar_files/jasper-1.900.1.tar.gz
tar xvfz jasper-1.900.1.tar.gz
./configure
make
make install
export JASPERLIB=/usr/lib64 # not installed need own jasper
export JASPERINC=/usr/include
```
If need FLEX:
```bash
cd /nobackup/${USER}/flex/lib
./configure --prefix=$(pwd)/../flex
export FLEX_LIB_DIR='/nobackup/${USER}/flex/lib'
```
