Data Management

Simply put, a source of information.

This may sound so obvious it’s not worth saying, but it’s worth mentioning so we don’t start making assumptions like “all data sets have a file”, etc.

In building the model, it’s worth considering some common hypothetical situations, such as when the data set is:

• a file on the local disk
• a file in an s3 bucket
• a file on the web
• a connection to an actively-changing DB
• a simple value

Data sources may operate under a number of different schemes:

single release

There is only one version of the data — e.g. the Iris data set.

multiple (infrequent) release

The data set has (frequently versioned) updates — e.g. ClinVar, which has monthly releases.

multiple (frequent) release

The data set changes rapidly, with releases frequently un-versioned — e.g. an API which proves a weather forecast for the next few days.

in flux

The data is near-constantly changing — e.g. a connection to a database which is being updated live.

To support reproducibility well, we should be able to provide guarantees regarding versions and data integrity. A version parameter would be useful for recording which release of the data is being used, and data integrity can be accounted for by hashing the data and recording the hash in the data specification too. In the absence of version information, recording the hash also allows for a particular instance of a data set to be unambiguously referred to.

While hashing the data has a number of clear benefits, hashing large datasets can be rather time-consuming. Looking at the xxHash homepage, we see that MD5 and SHA1 both have a bandwidth a bit under a gigabyte per second. If we were to add xxHash.jl as a dependency, it looks like we could get up to ~30 GB/s (more realistically, at this point storage retrieval will be the bottleneck: SATA3 maxes out around 0.5 GB/s, and NVMe over PCIe 4.0 x2 has a maximum of 4 GB/s). So, while always verifying the content of small (<10GB) datasets seems reasonable, if we allow for people to be hypothetically accessing multi-terabyte datasets over AWS for instance, this suddenly starts seeming like a much worse idea. Furthermore, with frequently updating (possibly un-versioned) data sets, constantly updating the hash seems like a needless complication.

Not always hashing the data isn’t too bad though, as long as we provide alternative methods of keeping track of which instance of a data set is being used. Supporting a version parameter is a good step, but for frequently updating (possibly un-versioned) data sets the date at which it is acquired could be helpful.

Let us now consider where we put data that is saved. It would be good to support multiple versions of the same data set, put each instance of a data set under a folder based on the UUID (renaming the data set doesn’t change the data after all). The file name can be selected progressively, using the best version information we have available. We may as well keep the extension so any person (or program) looking at the data has a hint to its nature, but it can be ignored for our purposes.

• $(UUID)/hash-$(hash).$(ext)
• $(UUID)/version-$(version).$(ext)
• $(UUID)/datetime-$(datetime).$(ext)
• $(UUID)/data.$(ext)

Data must be stored in a certain location. So, we must provide a way of specifying both:

• what the location of the data is — the data path
• how that location should be accessed — the storage driver

It is also possible that one may want to specify multiple storage locations for the data, for instance:

• A publically available (slow to access) copy of the data (slow, always available)
• A speedy copy available on the lab’s local storage server (fast, not always available)

In such a situation it would seem sensible to be able to provide both locations as options, but give priority to the closer data store. This would also allow mirrors for data sets to be given, improving robustness.

As mentioned earlier, “acquiring the data” can take many forms. It could be downloading a file, or opening a connection to a remote filesystem and passing a connection to a file. A single storage driver could reasonably provide multiple modes of access. We would want to provide the ability to request a particular mode of access to allow for intelligent restrictions. For instance it would make sense to just download a small file instead of opening a remote file handle, and similarly one may want to only open a remote file handle to a massive file, and never try to download it.

Acquiring the data is an excellent first step, however to actually do anything with it, we need a Julia representation we can work with. We want a solution that is a simple and convenient as possible, while allowing for the wide array of data formats, and Julia types that could reasonably represent the data.

Note that I say “types” not “type”, as there may well be multiple Julia objects the data could be represented as (e.g. IO, Array, or DataFrame). We can embrace this passibility by supporting multiple “loaders”, and allowing the user to request a particular form of the data. Each loader could provide a list of types it can provide (qualified by module, if outside of Base. and Core.), which is then matched against the type the user is asking for when they ask for the data. Note that we will also want to match the loader method(s) that provide that type against the modes of access that the storage driver(s) can provide.

There is a further complication in the various different ways a file can be parsed. For example when using CSV.read one can specify the types each column should be read as. This necessitates the ability to provide parameters to the loader too.

Often, in the process of analysing/processing some data, more data is created.

We can represent the relation between data sets as a DAG, for instance here’s one from my work:

A derived data set could be constructed via a bespoke loader, with the other data sets as additional inputs and provided as keyword arguments (like modules).

Ideally, we could ask for any data set and it would be lazily constructed as needed from its dependencies. There already exists a Julia library (Dagger.jl ) which lazily executes computations as a DAG, and parallelises them when possible. If this can be applied nicely here, it could allow us to add a lot of functionality with comparatively little effort.

It’s also worth explicitly noting that with a git annex storage driver, thanks to the ability to prioritise storage locations, results that have been computed on another machine could be fetched automatically — isn’t that neat!

Producing intermediate data will require support for writing. I think we can consider a writer to be a mirror to a loader.

I’m quite attracted to this interpretation, as it means that writer can be structured near-identically to loaders. It gives both us and the user less to think about.

write(writer::Val{:csv}, destination::IO, from::DataFrame; parameters::Dict{String, <:Any}, CSV::Module)
# ...
register_writer!(:csv, [(:CSV, "336ed68f-0bac-5ca0-87d4-7b16caf5d00b")])

We also get support for bespoke/anonymous writers for free, which is rather handy when constructing data dependencies.

It’s entirely realistic that a bespoke data loader could go to great pains to reformulate the data source into something much easier to work with², or that an esoteric format may naturally takes a long time to be processed into a Julia object.

In many cases, I expect loading a dump of the Julia object to be faster than reparsing the data source.

These situations make a compelling case for caching support. Thanks to the exitance of packages like JLD2, this should be fairly easy to implement. Thanks to the ability to prioritise storage locations, there’s a good chance this can be implemented as a top-priority filesystem store, with a writer that writes to that location. As such, caching wouldn’t be directly supported by the specification, but possible using the existing components of it. I find this to be an attractive option.

We may have to have this as an explicit feature though. The hard part (as always) is cache invalidation. Particularly in the case of bespoke loaders, a change to the loading function may well change the resulting Julia object. This can be resolved by including the hash of the file defining the bespoke loader, in addition to information on the data dependencies (if any).

Another approach we could take is defining a cache loader/writer. If we supply the entire data object it could compute hashes etc. automatically, without needing funky arguments. I feel that this may well be the best option.

Only certain results are cacheable though. There should either be a list of non-cacheable types (e.g. IO, Function), or loaders should supply cachability information with the result.

Conceptually, taking a .csv file and running CSV.read on it is rather simple.

What about taking an encrypted, compressed, .csv file and then:

1. Unencrypting it
2. Decompressing it
3. Parsing it with CSV.File

I’d argue that we’ve just created three new derived data sets from the original data:

• The data, unencrypted
• The data, unencrypted and decompressed
• A DataFrame representation of the unencrypted, decompressed data

There are two approaches we could take here:

1. Allow each data set to have multiple loaders, which feed into each other
   • This would require expanding the loader syntax
2. Treat each transformation of the data as a new data set

I am inclined towards the second approach for a number of reasons, both conceptual and practical.

• “It’s all the same data” feels like sweeping the problem under the rug
• Treating each transformation as a new data set can already be cleanly expressed with data dependencies
• It’s conceivable that some of these layers may be slow, and having each layer be a data set allows for easier caching

The glaring downside of the second approach is that the data specification will be a tad more complicated, however I think this can be near-entirely mitigated by tooling.

It’s tempting to say that all data referenced should reside in a single specification file (encapsulation is good), however there are some eminently reasonable situations in which this would be needed. For example:

• The team you’re working with has a project data collection, and you want to try a spin-off where you take that data collection and add a few data sets of your own.
• A useful collection of data sets should be able to be published as a package and imported, making their data sets availible without having to resort to glorified copy-paste.

One solution to this would be to support a stack of data collections. Furthermore, by adding a name parameter to a particular collection (given in the specification file or set manually when loading) references to a data set by the same name in multiple collections can be disambiguated. I’m inclined to say that any reference to a data set should have the name parameter supplied, with the exception of the “base” layer of the data stack.

This should easily allow for data packages like RDatasets to be reproduced under this framework.

It would be nice to have a convenient way to refer to a dataset. I suggest the following syntax (with optional components enclosed in square brackets):

[LAYER:] DATASETNAME or UUID [@VERSION] [#HASH] [::TYPE]

The @VERSION tag should also support a special @latest form to refer to the most recent version available.

By explicitly constructing this syntax, we can unify the manner in which we refer to a data set across this framework, in the specification file and in code.

One interface to this could be via say a data"" macro, e.g.

• data"demo#59e6fa4962523389"
• data"demo@v0.1"
• data"RDatasets:datasets/iris"

A data set may well be reused in multiple projects. Having n copies has the potential to waste a lot of space if the data sets are large. Having a central data store allows for a single copy to be shared across each project.

This introduces an additional complication though, as obsolete data files can accumulate, taking up more and more space. To avoid this issue, some form of data store garbage collection must be implemented. In order for this to be viable, the store needs to possess a certain amount of metadata on the items stored within it.

At its simplest, we could have a list of data specification files contributing to the store. Any files in the store which have no connections to any of the data specification can be presumed to be garbage and removed.

We could go for something a bit richer, and give the data store it’s own data manifest with information on each file, enumerating the data specifications which refer to the data. This allows for garbage collection in the same manner as the “simple” option, but also provides a few more possibilities — such as listing which data specifications are responsible for the largest files in the store. I am inclined to go with this approach.

So far, I have outlined a lot of the functionality I think a good data management system should have, more consideration is warranted for how such functionality should be supported.

Ultimately, we want a system that not just supports the situations I have in mind, but situations I haven’t even conceived of. Of course, the extent to which we can consider currently inconceivable usecases is rather limited — but we can try to make a system which is a flexible as reasonably possible. To encourage this, I will endeavour to isolate the essential core of the overall system discussed and then design the rest of the features as extensions/plugins. It is my hope that doing this will result in two major benefits:

• A clearer vision and implementation of the foundational functionality
• A more capable plugin system, with better potential to be applicable to situations I have not thought of

The kernel of the ideas presented here is the relation between information and data. Stored somewhere we have data, bits strung together. We want to work with information, the semantics of the data. Here, we have devised a way of flowing between these forms.

This is the haecceity³ of this framework. Let us thus try to build the prototype system as a collection of such relations, and a plugin system powerful enough to add everything else on top of that.

For starters, we don’t want to accidentally have to deal with format incompatibilities, should we realise we made a mistake in future. A config version is thus required.

data_config_version=1

To help enable data layers, we should add a name parameter, and may as well add a uuid to provide an escape from name collisions.

name="example"
uuid="83d40074-8c69-46f0-b65d-e8f12dd83ead"

This feels like the bare essentials to me. Further information can be organised under a data key (preventing users from creating a data set called data, a small loss in my mind).

It is already evident that each data set declaration will possess a number of parameters. To avoid needless repetition, we can define a defaults property.

[data.defaults]
store="global"
recency=-1

Really, a data store is just a labelled storage location. As such, we should be able to get away with two parameters:

• the name (label) of the data store
• the storage driver used

Additional properties can then be passed as parameters to the storage driver.

[[data.store]]
name="global"
driver="filesystem"
path="@__GLOBALSTORE__"

In this section we will use the Iris data set as an example, constructing a specification which loads it from a cached copy if it exists, and from a URL if not.

This should be formalised further if the ideas here are taken further, but for now this is largely “specification by example”.

• DataCollection
• DataSet
• DataStore
• AbstractDataTransformer
• DataStorage{:driver}
• DataLoader{:driver}
• DataWriter{:driver}

TBD

The REPL should exist to make common operations as convenient as possible, but should not provide any functionality inaccessible via the API.

It’s too early to work everything out, but we can consider some common situations and what a good experience could look like.