OUTLINE
- Introduction
- How do I request a quota increase?
- How do I find and remove unnecessary files?
- Are you using /scratch/gpfs or /scratch/network?
- How should I deal with large Conda environments?
Move Your ~/.conda Directory to /scratch/gpfs or /scratch/network
Using ~/.condarc to Redirect Installs to /scratch/gpfs or /scratch/network
Using --prefix
Introduction
Use the checkquota command to check your current storage and your storage limits. Here's an example:
$ checkquota Storage/size quota filesystem report for user: ceisgrub Filesystem Mount Used Limit MaxLim Comment Tiger2 home /home 13.9GB 18.6GB 20GB Tiger2 scratch GPFS /scratch/gpfs 9.2GB 500GB 512GB Tigress GPFS /tigress 397.9MB 500GB 512GB Storage number of files used report for user: ceisgrub Filesystem Mount Used Limit MaxLim Comment Tiger2 home /home 104.2K 0 0 Tiger2 scratch GPFS /scratch/gpfs 172.2K 0 0 Tigress GPFS /tigress 229 0 0 For quota increase requests please use this website: https://forms.rc.princeton.edu/quota
The difference between Limit and Used is the available space for a given filesystem. For instance, in the example above the user has 18.6 - 13.9 = 4.7 GB available on /home. Follow the link at the bottom of the output of checkquota to request a quota increase for any of the filesystems: https://forms.rc.princeton.edu/quota.
The first section of the two in the output above shows the amount of storage used. The rightmost three numbers are the block limits, listed in either kilobytes (K), gigabytes (G), or megabytes (M). The first is the number of blocks in use. The second is the soft quota; warnings are issued when this threshold is reached. The third is the "hard" quota (or limit); file creation or extension will fail when this threshold is reached.
The second section shows the number of files. The rightmost three numbers are the current usage and limits. The first is the number of files in existence. The second is the soft quota; warnings are issued when this threshold is reached. The third is the "hard" quota (or limit); file creation will fail when this threshold is reached.
Note: There are 1024 bytes in a KB, 1024 KBs in an MB, 1024 MBs in a GB, and 1024 GBs in a TB.
How do I request a quota increase?
Please complete this form. If you need an increase in the number of files then indicate this in the "Notes" field of the form.
How do I find and remove unnecessary files?
First identify the directories that are using the most space. To inspect your /home directory use this command after replacing <YourNetID> :
$ du -h --max-depth=1 /home/<YourNetID> | sort -hr
For another filesystem such as /scratch/gpfs/ use this command (it may take a minute):
$ du -h --max-depth=1 /scratch/gpfs/<YourNetID> | sort -hr
You can remove individual files with:
$ rm <file1> <file2> <file3>
Or remove entire directories with:
$ rm -rf <directory1> <directory2> <directory3>
Are you using /scratch/gpfs or /scratch/network?
Be aware that in addition to your /home directory, you have a large amount of space on /scratch/gpfs/<YourNetID> or /scratch/network/<YourNetID>. See the Data Storage page to learn about these filesystems. In many cases, users do not need a quota increase but instead need to learn about the different filesystems.
How should I deal with large Conda environments?
Conda environments are typically 1-10 GB in size. By default they are stored in /home/<YourNetID>/.conda. Because a typical quota size for a /home directory is 10-50 GB, it is common to run out of space. This section suggests ways of dealing with the storage requirements of Conda environments.
Request More Space
One solution is to request a larger quota for /home by completing this form. There is a limit to how large the quota can be made so consider the options below.
Remove Unnecessary Packages and Caches
Packages and other files for each environment are downloaded and saved in a cache directory called pkgs. Over time there may be packages that are no longer needed by any of your environments. Run the following commands to remove these packages and related files:
$ module load anaconda3/2022.5 $ conda clean --all
For more on this command see the Conda docs.
Move Your ~/.conda Directory to /scratch/gpfs or /scratch/network
If you already have one or more Conda environments then consider moving them to a location where you have more space using these commands:
Adroit
$ rsync -avu $HOME/.conda /scratch/network/$USER/ $ rm -Rf $HOME/.conda $ cd $HOME $ ln -s /scratch/network/$USER/.conda .conda
Della, Stellar, Tiger, Traverse
$ rsync -avu $HOME/.conda /scratch/gpfs/$USER/ $ rm -Rf $HOME/.conda $ cd $HOME $ ln -s /scratch/gpfs/$USER/.conda .conda
The commands above create a symbolic link in your /home directory that points to the .conda directory in /scratch/network or /scratch/gpfs. While these locations are not backed up, it is generally easy to recreate an environment if there is a failure. You can also periodically export an environment file that would allow you to recreate the environment.
Using ~/.condarc to Redirect Installs to /scratch/gpfs or /scratch/network
This approach is good if you do not have any existing Conda environments or you are willing to start over. This approach will install everything in a location where you have more space. Run the command below to create a ~/.condarc file which will do the redirect:
Adroit
$ echo -e "pkgs_dirs:\n - /scratch/network/$USER/CONDA/pkgs\nenvs_dirs:\n - /scratch/network/$USER/CONDA/envs" >> ~/.condarc
Della, Stellar, Tiger, Traverse
$ echo -e "pkgs_dirs:\n - /scratch/gpfs/$USER/CONDA/pkgs\nenvs_dirs:\n - /scratch/gpfs/$USER/CONDA/envs" >> ~/.condarc
Then create an environment as usual. The packages and the contents of your environment will be stored in /scratch/network or /scratch/gpfs instead of /home. Consider making an environment file for environments that might be hard to recreate since /scratch/gpfs is not backed-up.
You could also use miniconda to do the same.
Using --prefix
Another approach is to install the environment directly on /scratch/gpfs/<YourNetID> from the beginning. While the files on this filesystem are not backed up, most environments are easy to recreate in the event of a failure. This approach will reduce the storage requirements for your ~/.conda directory. Here's an example:
$ module load anaconda3/2022.5 $ conda create --prefix /scratch/gpfs/$USER/ml-env scikit-learn matplotlib ipython
A modification is needed to activate the environment in the Slurm script:
#!/bin/bash #SBATCH --job-name=myjob # create a short name for your job #SBATCH --nodes=1 # node count #SBATCH --ntasks=1 # total number of tasks across all nodes #SBATCH --cpus-per-task=1 # cpu-cores per task (>1 if multi-threaded tasks) #SBATCH --gres=gpu:1 # number of gpus per node #SBATCH --mem=2G # total memory (RAM) per node #SBATCH --time=00:00:30 # total run time limit (HH:MM:SS) #SBATCH --mail-type=begin # send email when job begins #SBATCH --mail-type=end # send email when job ends #SBATCH --mail-user=<YourNetID>@princeton.edu module purge module load anaconda3/2022.5 conda activate /scratch/gpfs/$USER/ml-env python myscript.py