CVMFS Monthly Catchup

→ Europe/London
Teams

Teams
Adam Parker (Data Solutions Unit), Shaun de Witt
Hide

Agenda and minutes

Previous actions: (new actions and updates in red)

# Action Owner Status Due
01 Enquire about Azure cloud resources for hosting CVMFS stratum-0 data server SdW

Asked John Galley. awaiting response.

Required resources:
2 cores, 4GB RAM, 500GB storage

  
02 Request VM on STFC IRIS for initial benchmarking tests AP

Form emailed to Deniza 9/9/24. 

Awaiting feedback.

19/09/24
03 Demonstrate CVMFS client installation and data access on CUMULUS login nodes (unprivileged user account) SD Done 19/09/24
04 Demonstrate CVMFS client installation and data access on CUMULUS compute nodes (no internet access)  SD, AP Blocked by 05 - requires squid proxy setup first 19/09/24
05 Put in ticket for VM on the cumulus node previously known as login-03. Find out suitable resources requirements for request beforehand.  SD

Ticket raised.

Frederick has raised some concerns with the implementation. Meeting required between SD/AP with SdW and FF.

19/09/24
06 Find out how versioning works on CVMFS SD Done-ish 19/09/24
07 Demonstrate CVMFS client installation and data access on CSD3 compute nodes ( internet access is available)  SD, AP Done 19/09/24
08 Contact ligo people about cvmfs for data distribution
Duncan MacLeod (MacLeodDM@cardiff.ac.uk)
Vivien Raymond (RaymondV@cardiff.ac.uk)
 AP   19/09/24

 

Agenda: (notes from meeting in red)

  • Status of external server requests for initial benchmarking tests (Azure and IRIS)
  • Data access on HPC computer nodes (CUMULUS and CSD3)
  • (MAIN EVENT) feedback from CVMFS workshop
  • progress towards benchmarking. Testing plan
  • AOB 

 

Minutes:

Versioning:

The data published by a CVMFS server exists in versioned (or tagged) and immutable "snapshots". A new snapshot is published with every change. Previous snapshots are preserved and remain accessible. Rolling back a snapshot may delete any intermediate snapshots saved. 

Each file is hashed and the file system is stored as a hash tree (automatic deduplication). When data is modified only the changes are hashed, so only the modifications are tracked between versions (like git). Users can access older versions of the repository by referencing the hash of the snapshot or using named tags. 

  1. List available snapshots (tags): cvmfs_talk -i <repository_name> tag list
  2. Temporarily switch to a tag: export CVMFS_REPOSITORY_TAG=<tag_name>
  3. Temporarily switch to a snapshot using a hash: export CVMFS_REPOSITORY_DATE=<timestamp_or_hash>
  4. Revert back to the latest version: unset CVMFS_REPOSITORY_TAG and unset CVMFS_REPOSITORY_DATE

 

CVMFS workshop report:

  • Useful to meet developers etc.
  • identified some possible collaborations
There are minutes attached to this event. Show them.