Julia on the HPC Clusters

Julia is a flexible dynamic language, appropriate for scientific and numerical computing, with performance comparable to traditional statically-typed languages. One can write code in Julia that is nearly as fast as C. Julia features optional typing, multiple dispatch, and good performance, achieved using type inference and just-in-time (JIT) compilation, implemented using LLVM. It is multi-paradigm, combining features of imperative, functional, and object-oriented programming.

 

Modules

To use Julia you need to load an environment module. For instance, on Adroit:

$ module avail julia
------------------ /usr/licensed/Modules/modulefiles ------------------
julia/0.4.7 julia/0.5.1 julia/0.7.0 julia/1.0.3 julia/1.2.0 julia/1.4.1
julia/0.5.0 julia/0.6.0 julia/1.0.1 julia/1.1.0 julia/1.3.0 julia/1.5.0

$ module load julia/1.5.0
$ julia

 

Serial Jobs

Here is a simple Julia script (myscript.jl):

println("Hello, world.")

Below is the Slurm script:

#!/bin/bash
#SBATCH --job-name=serial_jl     # create a short name for your job
#SBATCH --nodes=1                # node count
#SBATCH --ntasks=1               # total number of tasks across all nodes
#SBATCH --cpus-per-task=1        # cpu-cores per task (>1 if multi-threaded tasks)
#SBATCH --mem-per-cpu=4G         # memory per cpu-core (4G is default)
#SBATCH --time=00:01:00          # total run time limit (HH:MM:SS)
#SBATCH --mail-type=begin        # send email when job begins
#SBATCH --mail-type=end          # send email when job ends
#SBATCH --mail-user=<YourNetID>@princeton.edu

module purge
module load julia/1.5.0

julia hello_world.jl

To run the Julia script, simply submit the job to the cluster:

$ sbatch job.slurm

After the job completes, view the output with cat slurm-*:

Hello, world.

Use squeue -u $USER to monitor queued jobs.

To run the example above on Della, for example, carry out these commands:

$ ssh <YourNetID>@della.princeton.edu
$ cd /scratch/gpfs/<YourNetID>
$ git clone https://github.com/PrincetonUniversity/hpc_beginning_workshop
$ cd hpc_beginning_workshop/RC_example_jobs/julia
# edit email address in job.slurm using a text editor
$ sbatch job.slurm

 

Running Parallel Julia Scripts using the Distributed Package

Julia comes with built-in parallel programming support. While many of the parallel packages are still under development, they can be used to achieve a significant speed-up. This example presents a simple use case of the Distributed package. The Julia script below illustrates the basics of using spawnat and fetch:

using Distributed

# launch worker processes
num_cores = parse(Int, ENV["SLURM_CPUS_PER_TASK"])
addprocs(num_cores)

println("Number of cores: ", nprocs())
println("Number of workers: ", nworkers())

# each worker gets its id, process id and hostname
for i in workers()
    id, pid, host = fetch(@spawnat i (myid(), getpid(), gethostname()))
    println(id, " " , pid, " ", host)
end

# remove the workers
for i in workers()
    rmprocs(i)
end

Here is the Slurm script:

#!/bin/bash
#SBATCH --nodes=1                # node count
#SBATCH --ntasks=1               # total number of tasks across all nodes
#SBATCH --cpus-per-task=7        # cpu-cores per task (>1 if multi-threaded tasks)
#SBATCH --mem-per-cpu=4G         # memory per cpu-core (4G is default)
#SBATCH --time=00:01:00          # total run time limit (HH:MM:SS)
#SBATCH --mail-type=begin        # send email when job begins
#SBATCH --mail-type=end          # send email when job ends
#SBATCH --mail-user=<YourNetID>@princeton.edu

module purge
module load julia/1.5.0

srun julia hello_world_distributed.jl

The output should be something like:

Number of cores: 8
Number of workers: 7
2 19945 tiger-i25c1n11
3 19947 tiger-i25c1n11
4 19948 tiger-i25c1n11
5 19949 tiger-i25c1n11
6 19950 tiger-i25c1n11
7 19951 tiger-i25c1n11
8 19952 tiger-i25c1n11

There is much more that can be done with the Distributed package. You may also consider looking at distributed arrays and multithreading on the Julia website.

 

Julia Environments and GPU Packages

If you are working on multiple Julia projects where each project requires a different set of packages then you should use environments to isolate the different software required for each project.

$ ssh tigergpu
$ module load julia/1.5.0 cudatoolkit/10.2 cudnn/cuda-10.2/7.6.5
$ julia
$ ]
$ activate flux-env
$ add CuArrays Flux DifferentialEquations DiffEqFlux
$ # press the backspace or delete key
$ exit()

Note that this will place your Julia environment in your home directory. You will need to point to this when activating it in scripts. For instance, if your Julia script is in /home/NetID/myjob then do: activate "../flux-env". Below is a sample Julia script (myscript.jl):

using Pkg
Pkg.activate("../flux-env")
Pkg.instantiate()

using Flux, CuArrays

z = CuArrays.cu([1, 2, 3])
println(2 * z)

m = Dense(10,5) |> gpu
x = rand(10) |> gpu
println(m(x))

Below is an appropriate Slurm script (job.slurm):

#!/bin/bash
#SBATCH --job-name=myjob         # create a short name for your job
#SBATCH --nodes=1                # node count
#SBATCH --ntasks=1               # total number of tasks across all nodes
#SBATCH --cpus-per-task=1        # cpu-cores per task (>1 if multi-threaded tasks)
#SBATCH --mem-per-cpu=4G         # memory per cpu-core (4G per cpu-core is default)
#SBATCH --gres=gpu:1             # number of gpus per node
#SBATCH --time=00:01:00          # total run time limit (HH:MM:SS)

module purge
module load julia/1.5.0 cudatoolkit/10.2 cudnn/cuda-10.2/7.6.5

julia myscript.jl

Submit the job with: sbatch job.slurm Here's the output on Adroit:

┌ Warning: Some registries failed to update:
│     — /home/jdh4/.julia/registries/General — failed to fetch from repo
└ @ Pkg.Types /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.2/Pkg/src/Types.jl:1171
[ Info: Building the CUDAnative run-time library for your sm_35 device, this might take a while...
Activating environment at `~/flux-env/Project.toml`
  Updating registry at `~/.julia/registries/General`
  Updating git-repo `https://github.com/JuliaRegistries/General.git`
Float32[2.0, 4.0, 6.0]
Float32[-0.835879, 0.3685953, 1.0108142, -0.29181987, 0.31272212]

This warning is arising because GitHub is inaccessible from the compute nodes. You can do a manual update periodically on the head node like this:

$ julia
julia> using Pkg
julia> Pkg.activate("../flux-env")
Activating environment at `~/flux-env/Project.toml`

julia> Pkg.update()
  Updating registry at `~/.julia/registries/General`
  Updating git-repo `https://github.com/JuliaRegistries/General.git`
 Resolving package versions...
 Installed DiffEqFlux ─ v1.3.2
  Updating `~/flux-env/Project.toml`
  [aae7a2af] ↑ DiffEqFlux v1.3.1 ⇒ v1.3.2
  Updating `~/flux-env/Manifest.toml`
  [aae7a2af] ↑ DiffEqFlux v1.3.1 ⇒ v1.3.2

julia> exit()

Note that there are no GPUs on the head node of TigerGPU and no internet connection on the compute nodes. To run MNIST for example you will need to download the data first.

 

More on Environments and Storing Packages

If you want to store your Julia packages on /scratch/gpfs to free space on /home then set this environment variable in your ~/.bashrc file:

export JULIA_DEPOT_PATH=/scratch/gpfs/$USER/myjulia

To store environments on /scratch/gpfs follow these steps:

$ module load julia/1.5.0
$ julia
> using Pkg
> Pkg.activate("/scratch/gpfs/<YourNetID>/myenv")
> Pkg.add("DifferentialEquations")
> exit()

The following Slurm script can then be used:

#!/bin/bash
#SBATCH --job-name=julia         # create a short name for your job
#SBATCH --nodes=1                # node count
#SBATCH --ntasks=1               # total number of tasks across all nodes
#SBATCH --cpus-per-task=1        # cpu-cores per task (>1 if multi-threaded tasks)
#SBATCH --mem-per-cpu=4G         # memory per cpu-core (4G per cpu-core is default)
#SBATCH --time=00:10:00          # total run time limit (HH:MM:SS)

module purge
module load julia/1.5.0

julia myscript.jl

The first four lines of myscript.jl are:

using Pkg
Pkg.activate("/scratch/gpfs/<YourNetID>/myenv")
Pkg.instantiate()
using DifferentialEquations

 

Debugging

For GPU programming, to see which libraries are being used, start Julia in this way:

$ JULIA_DEBUG=CUDAapi julia

 

Getting Help

If you encounter any difficulties while running Julia on the HPC clusters then please send an email to cses@princeton.edu or attend a help session.