Notice: This Documentation is in the process of being updated, some of the information may be out of date, or incorrect.
Introduction #
The FAIR Data Pipeline is intended to enable tracking of provenance of
FAIR (findable, accessible, interoperable and reusable) data used in epidemiological modelling. Pipeline APIs written in
C,
C++,
FORTRAN,
Java,
Julia,
Python and
R can be called by modelling software for data ingestion. These interact with a local relational database storing metadata and the local filesystem, and are configured using a yaml file associated with the model run. Local files and metadata can be synchronised with a remote registry via a command line interface (
fair).
The key benefits of using the FAIR Data Pipeline are:
- Opensource, all code is available on the FAIRDataPipeline GitHub
- Data recorded in a FAIR fashion (metadata on all data and code open and available for inspection)
- Provenance tracing allows model outputs to be traced to inputs and modelling code
- Multiple language support
- Designed to run on a broad range of platforms (including HPC, inside Safe Havens)
- Designed to be set up and completed online (to down-/up-load data) and run offline (Safe Havens will require this)
- Open metadata provides knowledge of or access to shared central data for specific domains (e.g. COVID-19 epidemiological modelling)
Running Models #
To use the FAIR Data Pipeline with a piece of modelling software, you must add a language specific Pipeline API as a dependency and interact with data registered in the pipeline via the methods it presents. Each model run must be configured using a
config.yml file which specifies inputs and outputs by metadata.
graph LR; subgraph CLI fair end subgraph Local API API[Pipeline API] CY[config.yml] end subgraph localhost LR[Registry] FS[File Store] end subgraph Model MC[Model code] end fair --> CY CY --> API fair --> MC MC --> |read/write/link_*| API API --> |read/write/link_*| LR LR --> |read/link_*| API API --> |read/link_*| MC API --> |write_*| FS FS --> |read_*| API MC --> |"(from link_write)"| FS FS --> |"(from link_read)"| MC
Getting data #
The command line interface
fair is used to download and upload data and metadata required for and produced by model runs.
graph LR; subgraph Remote RR OS URI end subgraph Local LR FS end RR[Remote Registry]-->|fair pull| LR[Local Registry] LR-->|fair push| RR RR-->OS(Managed Object Store) RR-->URI(Arbitrary URI) LR-->FS(Local Filesystem) OS-->|fair pull| FS URI-->|fair pull| FS FS-->|fair push| OS FS-->|fair push| URI