Login

CVMFS Discussion with Jump Trading

→ Europe/London
C7.2.01 (C7)

C7.2.01

C7

Culham Campus Abingdon, Oxfordshire, OX14 3DB
Adam Parker (Data Solutions Unit, UKAEA), Matt Harvey (Jump Trading)
Description

Jump Trading is an international trading firm committed to applying cutting-edge research to global financial markets, and operates substantial HPC and data infrastructure to support that mission. 

This session will share some of Jump’s experiences managing a business-critical data archive subject to exponential growth (many 100s PB, doubling time ~ 2 years). In particular, describing the process of re-architecting this live production system to transition from traditional on-premises block (GPFS) storage to cloud object storage, using CERN’s CVMFS filesystem.

Matt Harvey (Jump Trading)

CVMFS 2024
CVMFS - UKAEA Feb 2025.pdf
Matt's Talk
Hide

Present: AP, ...

Agenda + Discussion Points

  • Introductions
  • Background on CVMFS (unaware what it is)

  • Current UKAEA Status + Plans

    • Including summary of our experiments / systems
    • CSD3 + Cumulus

  • Jump Trading approach

    • How is CVMFS being used for data
    • Current infrastructure
      • Amount of data
      • Number of systems
      • Caches/Squids/Varnish
    • Special considerations for data
    • How much support are you getting from CERN
    • Suitability for AI/ML workflows

  • General Data Handling

    • Previously discussed Globus
    • Data transfer/distribution
    • OSDF/Pelican
    • Adios?

  • Technical Talk at 3pm (C7 Harwell)

There are minutes attached to this event. Show them.
    • 11:00 → 11:15
      Introductions 15m
      Speaker: Adam Parker (Data Solutions Unit, UKAEA)
    • 11:15 → 12:00
      Roundtable Discussion 45m
      Speakers: Adam Parker (Data Solutions Unit, UKAEA), Alejandra Gonzalez-Beltran, Andrew Lahiff (Advanced Computing), Jonathan Hollocombe (Data Solutions Unit, UKAEA), Matt Harvey (Jump Trading), Nathan Cummings (High Performance Data Analytics), Samuel (Computing) Jackson (UKAEA), Shaun de Witt, Stephen Dixon (The UDA man)
    • 12:00 → 13:00
      Lunch 1h
    • 13:00 → 13:45
      Roundtable Discussion 45m
      Speakers: Adam Parker (Data Solutions Unit, UKAEA), Alejandra Gonzalez-Beltran, Andrew Lahiff (Advanced Computing), Jonathan Hollocombe (Data Solutions Unit, UKAEA), Matt Harvey (Jump Trading), Nathan Cummings (High Performance Data Analytics), Samuel (Computing) Jackson (UKAEA), Shaun de Witt, Stephen Dixon (The UDA man)
    • 13:45 → 14:00
      Coffee 15m
    • 14:00 → 14:30
      Closing Remarks and AOB 30m
      Speakers: Adam Parker (Data Solutions Unit, UKAEA), Alejandra Gonzalez-Beltran, Andrew Lahiff (Advanced Computing), Jonathan Hollocombe (Data Solutions Unit, UKAEA), Matt Harvey (Jump Trading), Nathan Cummings (High Performance Data Analytics), Samuel (Computing) Jackson (UKAEA), Shaun de Witt, Stephen Dixon (The UDA man)
Powered by Indico v3.3.5