- How to Access the Stellar Cluster
- How to Use the Stellar Cluster
- Important Guidelines
- Login Nodes
- Hardware Configuration
- Job Scheduling (QOS Parameters)
- Compiler Flags and Math Libraries
- Environment Modules
- Visualization Nodes
- Jupyter OnDemand
- Maintenance Window
Stellar, a heterogeneous cluster with AMD and Intel processors, built to support large-scale parallel jobs predominantly for use by researchers in astrophysical sciences, plasma physics, physics, chemical & biological engineering and atmospheric & oceanic sciences.
To use the Stellar cluster you have to request an account on Stellar and then log in through SSH.
- Requesting Access to Stellar
Access to the large clusters like Stellar is granted on the basis of brief faculty-sponsored proposals. See section titled For large clusters: Submit a proposal or contribute for details. Perseus and Eddy users will not get an account automatically. Have your PI send a request to Research Computing.
If, however, you are part of a research group with a faculty member who has contributed to or has an approved project on Stellar, that faculty member can sponsor additional users by sending a request to firstname.lastname@example.org. Any non-Princeton user must be sponsored by a Princeton faculty or staff member for a Research Computer User (RCU) account.
- Logging into Stellar
Once you have been granted access to Stellar, you can connect via SSH.
For PU and PPPL, connect to the Intel login node (VPN required from off-campus):
$ ssh <YourNetID>@stellar.princeton.edu
For GFDL, connect to the AMD login node (VPN required from off-campus):
$ ssh <YourNetID>@stellar-amd.princeton.edu
- For more on how to SSH, see the Knowledge Base article Secure Shell (SSH): Frequently Asked Questions (FAQ). If you have trouble connecting then see our SSH page.
Since Stellar is a Linux system, knowing some basic Linux commands is highly recommended. For an introduction to navigating a Linux system, view the material associated with our Intro to Linux Command Line workshop.
Using Stellar also requires some knowledge on how to properly use the file system, module system, and how to use the scheduler that handles each user's jobs. For an introduction to navigating Princeton's High Performance Computing systems, view the material associated with our Getting Started with the Research Computing Clusters workshop. Additional information specific to Stellar's file system, priority for job scheduling, etc. can be found below.
All users are required to read and abide by the Stellar usage guidelines:
- PU guidelines
- PPPL guidelines
- GFDL guidelines
The login nodes, stellar-intel and stellar-amd, should be used for interactive work only such as compiling programs and submitting jobs as described below. Please remember that these are shared resources for all users. No jobs should be run on the login nodes with the exception of brief tests that last no more than a few minutes and use only a few CPU-cores. Where practical, we ask that you entirely fill the nodes so that CPU core fragmentation is minimized. For this cluster, stellar, that means multiples of 96 cores.
Use the "snodes" command to see the number of available nodes. Nodes have quad sockets with 24 cores/socket per node and 8GB/core memory. The back end network is 100Gb Infiniband, HDR100.
The /tigress and /projects directories are mounted on the login node (stellar) as well as the compute nodes over NFS. This is for access to data and software built for projects.
|Processor||Nodes||Cores per Node||Memory per Node||Max Instruction Set|
|2.9 GHz Intel Cascade Lake||296||96||768 GB||AVX-512|
|2.6 GHz AMD EPYC Rome||187||128||512 GB||AVX2|
2.6 GHz AMD EPYC Rome
*There are 2 GPUs per node.
Nodes on the Intel side will consist of PU-only and PPPL-only nodes. Think of a Venn diagram here with PU and PPPL circles. There will be an intersection of some number of nodes so that either side can expand. Those nodes will be weighted differently so they are the very last to be assigned.
Each GPU is an NVIDIA A100 and has 40 GB of memory. The nodes of stellar are connected with HDR 100 Infiniband. Run the "shownodes" command for additional information about the nodes. There is one large-memory node (4 TB) available to all users that is not mentioned in the table above. These may only be used for jobs that utilize more than 460 GB of memory. Please write to email@example.com for more information. For more technical details about the Stellar cluster, see the full version of the hardware systems table.
The default memory allocation on Stellar is 8 GB per core. Relying on the default value will work nicely for the Intel nodes but it can cause problems for the AMD nodes since they offer 512/128=4 GB per core, which translate to --mem-per-cpu=4000M (which is not equal to --mem-per-cpu=4G since 1 MB is 1024 KB in Slurm memory directives). Be sure to explicitly set the memory in Slurm scripts for the AMD nodes. Failure to do this may result in the following error message:
sbatch: error: Batch job submission failed: Requested node configuration is not available
Scheduling for PPPL and GFDL is done based on project. Users in these groups should add the following Slurm directive to all scripts:
#SBATCH -A <account-name>
|QOS||Time Limit||Jobs per User||Cores per User||Cores Available|
Use the "qos" command to see the latest values for the table above.
Any job submitted to the PU or PPPL partition requesting 47 or fewer CPU-cores will be assigned to the serial queue. Jobs in this queue will have the lowest priority of all jobs since the cluster is intended for multinode jobs. If you need to run a large number of serial jobs (47 cores or less) then you should consider moving that work to another cluster such as Della.
The Intel nodes feature Cascade Lake processors with AVX-512 as the highest instruction set. As a starting point, consider using these optimization flags when compiling a C++ code, for instance:
$ ssh <YourNetID>@stellar-intel.princeton.edu $ module load intel/2021.1.2 $ icpc -Ofast -xCORE-AVX512 -o mycode mycode.cpp
The Intel Math Kernel Library (MKL) is automatically loaded as a module when an Intel compiler module is loaded.
$ module load gcc-toolset/10 $ g++ -Ofast -march=cascadelake -o mycode mycode.cpp
The AMD nodes feature the EPYC processor with AVX2 as the highest instruction set. See the Quick Reference Guide by AMD for compiler flags for different compilers (AOCC, GCC, Intel) and the AOCC user guide. As a starting point, consider using these optimization flags when compiling a C++ code, for instance:
$ ssh <YourNetID>@stellar-amd.princeton.edu $ module load aocc/3.0.0 aocl/aocc/3.0_6 $ clang++ -Ofast -march=native -o mycode mycode.cpp
For a parallel Fortran code:
$ ssh <YourNetID>@stellar-amd.princeton.edu $ module load aocc/3.0.0 aocl/aocc/3.0_6 openmpi/aocc-3.0.0/4.1.0 $ mpif90 -Ofast -march=native -o hw hello_world_mpi.f90
Load the aocl module to make available the BLIS and libFLAME linear algebra libraries by AMD as well as FFTW3 and ScaLAPACK. Excellent performance was found for the High-Performance LINPACK benchmark using GCC and these libraries.
If you wish to use the Intel compiler for the AMD nodes then consider these flags:
$ module load intel/2021.1.2 $ icpc -Ofast -march=core-avx2 -o mycode mycode.cpp
Use the -march option above if you encounter the following error message:
Please verify that both the operating system and the processor support Intel(R) X87, CMOV, MMX, FXSAVE, SSE, SSE2, SSE3, SSSE3, SSE4_1, SSE4_2, POPCNT, AVX and F16C instructions.
The environment modules that you load define part of your software environment which plays a role in determining the results of your code. Run the "module avail" command to see the available modules. For numerous reasons including scientific reproducibility, when loading an environment module you must specify the full name of the module. This can be done using module load, for example:
$ module load intel/184.108.40.206
You will encounter an error if you do not specify the full name of the module:
$ module load anaconda3 ERROR: No default version defined for 'anaconda3' $ module load anaconda3/2020.11 $ python --version 3.8.5
Notable Modules and Modules to Avoid
- aocc/<version> makes the AMD compilers available
- aocl/<compiler>/<version> makes the AMD math libraries available
- cmake/3.18.2 provides a newer CMake over the system version (3.11.4)
- gcc/4.85 provides an older GNU Compiler Collection (GCC); it should only be used in rare cases
- gcc/8.3.1 is equivalent to using the system GCC
- gcc-toolset/10 makes GCC 10.2.1 available (use this when the system GCC is insufficient)
- nvhpc/21.1 provides the NVIDIA compilers and libraries (the compilers replace PGI)
- rh/devtoolset/7 makes GCC 7.3.1 available; it should be avoided in favor of the system GCC
If you would rather use short aliases instead of full module names then see the environment modules page.
The software environment on Stellar is very similar to the other Research Computing clusters. See the general documentation for Princeton University Research Computing. If you find that you need software packages that are not installed on Stellar then please send a request via e-mail to firstname.lastname@example.org.
The Anaconda Python distribution should be used when working with Python on Stellar:
$ module avail anaconda3 $ module load anaconda3/2020.11 $ python --version
See our Python page for more information on using the Anaconda Python distribution on the Research Computing clusters. One may also consider installing Miniconda. We do not provide a Python 2 anaconda module on Stellar since that version of the language has been unsupported for over a year.
The system Python is available but it should be avoided in favor of the Anaconda Python distribution which provides optimizations for our hardware. The system Python exists largely for the system administrators to install software. These commands illustrate its use:
$ python -bash: python: command not found $ python3 Python 3.6.8 (default, Nov 15 2020, 11:45:35) [GCC 8.3.1 20191121 (Red Hat 8.3.1-5)] on linux Type "help", "copyright", "credits" or "license" for more information. >>> $ python2 Python 2.7.17 (default, Nov 16 2020, 23:55:19) [GCC 8.3.1 20191121 (Red Hat 8.3.1-5)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>>
Again, for scientific work one should use the Anaconda Python distribution and not the system Python.
Stellar is intended for large multinode jobs so MATLAB is not available on the compute nodes. It is available on the visualization nodes, however. If you use MATLAB on the visualization nodes then please restrict the number of CPU-cores that you use. MATLAB can be used on other clusters such as Della.
To use Globus to transfer data to the /scratch/gpfs filesystem of Stellar, which is shared with Traverse, use this endpoint:
Princeton Traverse/Stellar Scratch DTN
There are two dedicated nodes for visualization and data analysis:
$ ssh <YourNetID>@stellar-vis1.princeton.edu # PU $ ssh <YourNetID>@stellar-vis2.princeton.edu # PPPL
These nodes support TurboVNC and they each offer two NVIDIA V100 GPUs for GPU-enabled software. Please use these nodes for visualization and data analysis instead of the stellar head nodes.
There is a dedicated GFDL node for running Jupyter notebooks at https://mystellar.princeton.edu. You will need to be on the VPN to connect from off-campus. Follow the directions on our Jupyter page for working with custom Conda environments. Enter your account information in the field labeled "Extra slurm options". An example of this would be:
Stellar will be down for routine maintenance on the second Tuesday of every month from approximately 6 AM to 2 PM. This includes the associated filesystems of /scratch/gpfs, /projects and /tigress. Please mark your calendar. Jobs submitted close to downtime will remain in the queue unless they can be scheduled to finish before downtime (see more).