Archive/backup datasets to tape

From Etp
Jump to: navigation, search

DE users can request dataset copy to a tape endpoint at GridKa, named FZK-LCG2_LOCALGROUPTAPE. This is intended for cases such as:

  • critical datasets and critical phase of analysis, you want to make sure a 2nd copy of these datasets is available
  • old but important datasets (reference for paper, thesis, etc). Put copy on tape to free space on disk endpoint

For transfer/access of the datasets on FZK-LCG2_LOCALGROUPTAPE you can use in principle the the standard Rucio tools. For an existing dataset already stored somewhere simply add a rule, e.g.

rucio add-rule user.myname:MyDataSetName 1 FZK-LCG2_LOCALGROUPTAPE

However, there are these additional requirements when you want to replicate your own data files onto tape endpoints.

1. Check whether you have quota to write to FZK-LCG2_LOCALGROUPTAPE

rucio list-account-limits --rse FZK-LCG2_LOCALGROUPTAPE myaccount 

If you don't have quota ask Guenter to set/increase.

2. When creating a Rucio dataset from your own files you must specify explicitly a dataset name, i.e.

# this works
rucio -v upload --rse LRZ-LMU_SCRATCHDISK user.myname:MyDataSetName file1 file2 file3
#
# this does not work
rucio -v upload --rse LRZ-LMU_SCRATCHDISK  file1 file2 file3
# upload to disk endpoint works, but it cannot be replicated to tape endpoint

3. You must first create and upload the dataset to a disk endpoint and then replicate it to FZK-LCG2_LOCALGROUPTAPE.

# upload to disk first
rucio -v upload --rse LRZ-LMU_SCRATCHDISK user.myname:MyDataSetName file1 file2 file3
# replicate to Tape
rucio add-rule user.myname:MyDataSetName 1 FZK-LCG2_LOCALGROUPTAPE
#
# check status (typically takes some 10 mins for a few GB)
rucio list-rules user.myname:MyDataSetName