Data organization in HEPnOS
HEPnOS is a hierarchical, distributed, write-once-read-many, typed key/value store. This page goes over these terms one by one to explain how they apply to HEPnOS.
HEPnOS stores a hierarchy of datasets identified by a name within their parent dataset. Datasets can store other datasets, but also runs, which, contrary to datasets, are identified by a run number within their parent datasets. Runs store subruns, which are identified by a subrun number within their parent run. Finally subruns store events, identified by an event number within their parent subruns.
In a sense, this hierarchy looks very much like that of a file system, except for the fact that runs, subruns, and events are identified by a number rather than a name. But contrary to a file systems, datasets, runs, subruns, and events do not store files. They store key/value pairs.
HEPnOS runs on a set of nodes. Data is distributed across these nodes such that the metadata corresponding to all datasets within the same parent dataset are stored on the same node (which is determined by consistent hashing the name of the parent dataset). Similarly, the metadata corresponding to all runs within a common dataset is located on the same node (determined by consistent hashing the name of the parent dataset). The same goes for subruns and for event. This organization was chosen to ease the process of iterating over all entries within a common parent entry (e.g. listing all runs within a dataset, or all subruns within a run).
Key/value entries (whether associated with datasets, runs, subruns, or events) that share a common parent are however not all stored on the same node. They are distributed based on consistent hashing of the key as well as the type of value. This is because we expect client programs to access key/value pairs in a more random manner and mostly in parallel. Hence this distribution prevents bottlenecks.
Key/value pairs in HEPnOS can only be written once and in an atomic manner. There is no delete operation, nor any way to overwrite the value associated with a key. This is in line with the expected use-cases of producer-consumer workflows where new data products are generated from previous events. This also greatly simplifies concurrent accesses and consistency.
The values stored in HEPnOS are instances of serializable C++ objects. When storing a key/value pair, a string version of the type of C++ object is appended to the key. Hence it is possible to store multiple C++ objects of distinct types under the same key.