Data and Storage
Storage specific to the VACC¶
/gpfs1
is the main pool of storage used by the VACC. We provide daily backups, copied to a data recovery location, and are generally able to retain the last 7 days, 4 weeks of weekly backups, and 3 months of monthly backups, although there is some fluctuation based on overall usage. If you need files recovered from backups, please email us at vacchelp@uvm.edu with details about what files or directories you need recovered and from what time.
/gpfs2
is considered scratch space. That means there are NO BACKUPS taken of /gpfs2
, which is linked to a scratch
folder in your home directory. Disk quotas are higher in /gpfs2/scratch
, so it’s great for large temporary or intermediate files.
/gpfs3
is available as scratch space and consists of 192TB of NVMe SSD, replicated to provide a 96TB filesystem. This is scratch space and has NO BACKUPS. Old data may be flushed occasionally.
GPFS disk quotas¶
The amount of gpfs storage is your disk quota. Disk quotas are determined by the VACC account owner’s tier — the level of support the principal investigator (PI) is providing to the VACC.
All members of a PI-group share the group’s disk quota. A “PI-group” includes the PI and any sponsored users under that account. If you have no sponsored users, the PI-group is just you. Check the disk usage of your PI-group with the command groupquota
.
GPFS disk storage & file limits¶
In the table below, “block” is the total size of your group’s disk storage. “Files” is the total number of files allowed to be written to disk.
gpfs1
TIER | BLOCK Soft Limit / Hard Limit |
FILES Soft Limit / Hard Limit |
|
---|---|---|---|
1 | 2TB / 4TB | 1M / 1.5M | |
2 | 8TB / 13TB | 3M / 6M | |
3 | 20TB / 25TB | 6M / 12M |
gpfs2
TIER | BLOCK Soft Limit / Hard Limit |
FILES Soft Limit / Hard Limit |
|
---|---|---|---|
1 | 4TB / 8TB | 1.5M / 3M | |
2 | 15TB / 25TB | 4M / 8M | |
3 | 35TB / 45TB | 8M / 16M |
Soft Limit vs. Hard Limit¶
If you look at the table above, you will see two types of limits: soft and hard.
- Soft limits are the amount of resources you are allocated based on your tier and the filesystem you are using.
- Hard limits are the buffer of resources we allow you because we understand that sometimes resources are unintentionally exceeded.
- You have a grace period of 7 days to reduce your usage of resources back to your soft limit.
For example, if you unintentionally exceed the soft limit of resources allocated to your tier/filesystem, your work will not be terminated and deleted for 7 days (the grace period). However, once the grace period of 7 days expires, your work will be terminated and files will be deleted to the soft limit you are allocated based on your tier/filesystem.
Research storage (Netfiles)¶
Do NOT use the VACC for for general file storage. The storage space on the GPFS file servers is for data you are currently using on the cluster. Please do not save/store general data or data that will not be used for active work there. For data storage, see Research Storage.
Research Storage can be requested that is accessible from the VACC cluster. When that is created, all Netfiles 'shares' will show up under the directory /netfiles
. The specific server that is included in the message sent when shares are created is NOT used.
Transferring data to the VACC cluster¶
There are multiple ways to transfer data to and from the VACC cluster.
Using a command line terminal¶
On MacOS, Linux, and on UVM administered Windows machines, there should be a Terminal application. On Windows 10 and 11, the Command Prompt or PowerShell application can be used.
From the terminal prompt, you can copy files
using the scp
command. The scp
command needs to know the name of the
cluster host, the user on that host, the name of the file(s) being copied,
and the location to which they should be copied. In the following examples,
substitute your own NetID for netid
in the text.
Note that the colon (:
) character is used to separate the name of the host from the directory path. Anything after the :
that does not begin with a /
is assumed to be a file or directory in your home directory.
Copying a file or directory to the VACC cluster¶
Copying a file called some_file.txt
from the current directory on your computer to your home directory on the VACC, you would use the first example below. To copy a file called iris.R
to the scripts
directory under your home directory on the VACC, use the second example. Note that the file will have the same name when copied since no alternate filename is given.
# Copy some_file.txt to ~/some_file.txt
scp some_file.txt netid@login.vacc.uvm.edu:
# Copy iris.R to the VACC into the directory ~/scripts; will not create ~scripts
scp iris.R netid@login.vacc.uvm.edu:scripts
# Copy the directory and contents of scripts to your home directory on the VACC
scp -r scripts netid@login.vacc.uvm.edu:
Copying a file or directory from the VACC cluster¶
All of these examples presume you want to copy to the current directory on
your own computer, specified by ./
.
# Copy some_file.txt from the VACC to the current directory
scp netid@login.vacc.uvm.edu:some_file.txt ./
# Copy iris.R to the VACC into the directory ~/scripts; will not create ~scripts
scp netid@login.vacc.uvm.edu:scripts/iris.R ./
# Copy the scripts directory from the VACC to your computer
scp -r netid@login.vacc.uvm.edu:scripts ./
Using a graphical file transfer program¶
There are several graphical file transfer programs available for both MacOS and
Windows. We will use FileZilla as an example. All such programs typically
need the same information, which we will provide here. These will often want
to use sftp
instead of scp
. The results are the same.
Once you have installed and started FileZilla, open a new connection with the following connection settings.
Field label | Value |
---|---|
File protocol: | SFTP |
Host name: | login.vacc.uvm.edu |
User name: | netid |
Password: | your UVM NetID password |
Port: | 22 |
We recommend that you NOT save your password, but save the rest for easy, future use.
Click on Connect to make the connection. FileZilla should present to panels with files and folders, one will be your computer and the other will be the VACC cluster. You should be able to drag and drop files or folders from one to the other.
Using Globus¶
Globus is a web-based data management tool that enables researchers to easily share and receive large data sets with collaborators from other institutions.
Globus has been configured to work with data stored on Netfiles shares, VACC accounts, and Microsoft OneDrive – allowing UVM researchers to easily share and receive enormous data transfers using existing research storage solutions.
Please see the page at the Globus Research Computing knowlegebase for more information about using Globus.