Checking your quota and requesting quota increases

OUTLINE

 

Introduction

Use the checkquota command to check your current storage and your storage limits. Here's an example:

$ checkquota
          Storage/size quota filesystem report for user: ceisgrub
Filesystem            Mount              Used   Limit  MaxLim Comment
Tiger2 home           /home            13.9GB  18.6GB    20GB 
Tiger2 scratch GPFS   /scratch/gpfs     9.2GB   500GB   512GB 
Tigress GPFS          /tigress        397.9MB   500GB   512GB  

          Storage number of files used report for user: ceisgrub
Filesystem            Mount              Used   Limit  MaxLim Comment
Tiger2 home           /home            104.2K       0       0 
Tiger2 scratch GPFS   /scratch/gpfs    172.2K       0       0 
Tigress GPFS          /tigress            229       0       0 

For quota increase requests please use this website:

         https://forms.rc.princeton.edu/quota

The difference between Limit and Used is the available space for a given filesystem. For instance, in the example above the user has 18.6 - 13.9 = 4.7 GB available on /home. Follow the link at the bottom of the output of checkquota to request a quota increase for any of the filesystems: https://forms.rc.princeton.edu/quota.

The first section of the two in the output above shows the amount of storage used. The rightmost three numbers are the block limits, listed in either kilobytes (K), gigabytes (G), or megabytes (M). The first is the number of blocks in use. The second is the soft quota; warnings are issued when this threshold is reached. The third is the "hard" quota (or limit); file creation or extension will fail when this threshold is reached.

The second section shows the number of files. The rightmost three numbers are the current usage and limits. The first is the number of files in existence. The second is the soft quota; warnings are issued when this threshold is reached. The third is the "hard" quota (or limit); file creation will fail when this threshold is reached.

Note: There are 1024 bytes in a KB, 1024 KBs in an MB, 1024 MBs in a GB, and 1024 GBs in a TB.

 

How do I request a quota increase?

Please complete this form. If you need an increase in the number of files then indicate this in the "Notes" field of the form.

 

How do I find and remove unnecessary files?

First identify the directories that are using the most space. To inspect your /home directory use this command after replacing <YourNetID> :

$ du -h --max-depth=1 /home/<YourNetID> | sort -hr

For another filesystem such as /scratch/gpfs/ use this command (it may take a minute):

$ du -h --max-depth=1 /scratch/gpfs/<YourNetID> | sort -hr

You can remove individual files with:

$ rm <file1> <file2> <file3>

Or remove entire directories with:

$ rm -rf <directory1> <directory2> <directory3>

 

Are you using /scratch/gpfs or /scratch/network?

Be aware that in addition to your /home directory, you have a large amount of space on /scratch/gpfs/<YourNetID> or /scratch/network/<YourNetID>. See the Data Storage page to learn about these filesystems. In many cases, users do not need a quota increase but instead need to learn about the different filesystems.

 

How should I deal with large Conda environments?

Conda environments are typically 1-10 GB in size. By default they are stored in /home/<YourNetID>/.conda. Because a typical quota size for a /home directory is 10-50 GB, it is common to run out of space. This section suggests ways of dealing with the storage requirements of Conda environments.

Request More Space

One solution is to request a larger quota for /home by completing this form. There is a limit to how large the quota can be made so consider the options below.

 

Remove Unnecessary Packages and Caches

Packages and other files for each environment are downloaded and saved in a cache directory called pkgs. Over time there may be packages that are no longer needed by any of your environments. Run the following commands to remove these packages and related files:

$ module load anaconda3/2022.5
$ conda clean --all

For more on this command see the Conda docs.

 

Move Your ~/.conda Directory to /scratch/gpfs or /scratch/network

If you already have one or more Conda environments then consider moving them to a location where you have more space using these commands:

Adroit

$ rsync -avu $HOME/.conda /scratch/network/$USER/
$ rm -Rf $HOME/.conda
$ cd $HOME
$ ln -s /scratch/network/$USER/.conda .conda

 

Della, Stellar, Tiger, Traverse

$ rsync -avu $HOME/.conda /scratch/gpfs/$USER/
$ rm -Rf $HOME/.conda
$ cd $HOME
$ ln -s /scratch/gpfs/$USER/.conda .conda

The commands above create a symbolic link in your /home directory that points to the .conda directory in /scratch/network or /scratch/gpfs. While these locations are not backed up, it is generally easy to recreate an environment if there is a failure. You can also periodically export an environment file that would allow you to recreate the environment.

 

Using ~/.condarc to Redirect Installs to /scratch/gpfs or /scratch/network

This approach is good if you do not have any existing Conda environments or you are willing to start over. This approach will install everything in a location where you have more space. Run the command below to create a ~/.condarc file which will do the redirect:

Adroit

$ echo -e "pkgs_dirs:\n - /scratch/network/$USER/CONDA/pkgs\nenvs_dirs:\n - /scratch/network/$USER/CONDA/envs" >> ~/.condarc

 

Della, Stellar, Tiger, Traverse

$ echo -e "pkgs_dirs:\n - /scratch/gpfs/$USER/CONDA/pkgs\nenvs_dirs:\n - /scratch/gpfs/$USER/CONDA/envs" >> ~/.condarc

Then create an environment as usual. The packages and the contents of your environment will be stored in /scratch/network or /scratch/gpfs instead of /home. Consider making an environment file for environments that might be hard to recreate since /scratch/gpfs is not backed-up.

You could also use miniconda to do the same.

 

Using --prefix

Another approach is to install the environment directly on /scratch/gpfs/<YourNetID> from the beginning. While the files on this filesystem are not backed up, most environments are easy to recreate in the event of a failure. This approach will reduce the storage requirements for your ~/.conda directory. Here's an example:

$ module load anaconda3/2022.5
$ conda create --prefix /scratch/gpfs/$USER/ml-env scikit-learn matplotlib ipython

A modification is needed to activate the environment in the Slurm script:

#!/bin/bash
#SBATCH --job-name=myjob         # create a short name for your job
#SBATCH --nodes=1                # node count
#SBATCH --ntasks=1               # total number of tasks across all nodes
#SBATCH --cpus-per-task=1        # cpu-cores per task (>1 if multi-threaded tasks)
#SBATCH --gres=gpu:1             # number of gpus per node
#SBATCH --mem=2G                 # total memory (RAM) per node
#SBATCH --time=00:00:30          # total run time limit (HH:MM:SS)
#SBATCH --mail-type=begin        # send email when job begins
#SBATCH --mail-type=end          # send email when job ends
#SBATCH --mail-user=<YourNetID>@princeton.edu

module purge
module load anaconda3/2022.5
conda activate /scratch/gpfs/$USER/ml-env

python myscript.py