Notice: This Documentation is in the process of being updated, some of the information may be out of date, or incorrect.

Introduction#

The FAIR Data Pipeline is intended to enable tracking of provenance of FAIR (findable, accessible, interoperable and reusable) data used in epidemiological modelling. Pipeline APIs written in C, C++, FORTRAN, Java, Julia, Python and R can be called by modelling software for data ingestion. These interact with a local relational database storing metadata and the local filesystem, and are configured using a yaml file associated with the model run. Local files and metadata can be synchronised with a remote registry via a command line interface (fair).

The key benefits of using the FAIR Data Pipeline are:

• Opensource, all code is available on the FAIRDataPipeline GitHub
• Data recorded in a FAIR fashion (metadata on all data and code open and available for inspection)
• Provenance tracing allows model outputs to be traced to inputs and modelling code
• Multiple language support
• Designed to run on a broad range of platforms (including HPC, inside Safe Havens)
• Designed to be set up and completed online (to down-/up-load data) and run offline (Safe Havens will require this)
• Open metadata provides knowledge of or access to shared central data for specific domains (e.g. COVID-19 epidemiological modelling)

Running Models#

To use the FAIR Data Pipeline with a piece of modelling software, you must add a language specific Pipeline API as a dependency and interact with data registered in the pipeline via the methods it presents. Each model run must be configured using a config.yml file which specifies inputs and outputs by metadata.

graph LR;
    subgraph CLI
        fair
    end
    subgraph Local API
        API[Pipeline API]
        CY[config.yml]
    end
    subgraph localhost
        LR[Registry]
        FS[File Store]
    end
    subgraph Model
        MC[Model code]
    end
 
    fair --> CY
    CY --> API

    fair --> MC

    MC --> |read/write/link_*| API
    API --> |read/write/link_*| LR

    LR --> |read/link_*| API
    API --> |read/link_*| MC

    API --> |write_*| FS
    FS  --> |read_*| API

    MC --> |"(from link_write)"| FS
    FS --> |"(from link_read)"| MC

Getting data#

The command line interface fair is used to download and upload data and metadata required for and produced by model runs.

graph LR;
    subgraph Remote
        RR
        OS
        URI
    end
    subgraph Local
        LR
        FS
    end

    RR[Remote Registry]-->|fair pull| LR[Local Registry]
    LR-->|fair push| RR

    RR-->OS(Managed Object Store)
    RR-->URI(Arbitrary URI)
    LR-->FS(Local Filesystem)

    OS-->|fair pull| FS
    URI-->|fair pull| FS
    FS-->|fair push| OS
    FS-->|fair push| URI