Self-describing Dataset (SDD)¶
In a mature Enterprise Knowledge Graph (EKG), the information provided (or consumed) by every connected data source (or target) can and should be described as a logical dataset that we call a self-describing dataset (SDD).1 Depending on maturity and architecture these SDDs are also known as data products or data services.
The general idea is that everything that be said about a given dataset be described---as close as possible and ideally in---that dataset itself.
Many topic areas can require such description, most of which would be described as "policies" in the form of models that are enforceable by EKG/Platform Services at run-time.
Core Data-catalog information¶
Core information about the dataset.
Linkage to concepts, ontologies, taxonomies and vocabularies.
Identifier registration office, creation policies, identifier related security policies and the like.
Personas, their privileges and jurisdiction-specific entitlement policies.
Ownership / Intellectual Property¶
- This also unveils what "authoritative data" the dataset contains, is it the "golden copy" and "authoritative source" for any given datapoint or not?
Policies that describe in detail how any given datapoint can be constructed from other datapoints whether they reside in the EKG or elsewhere.
- provide the detail that's necessary to build up the proper end-to-end lineage information.
- enable smart generic EKG/Platform Services or EKGOps Pipelines to transform data in a model-driven fashion
Quality Metrics and Assessments¶
Policies that describe how the overall "hygiene" of the dataset can be checked.
- enable smart generic EKG/Platform Services or EKGOps Pipelines to do a quick "smoke test" of the data and/or a thorough check on all kinds of (technical) details such as data formats etc (relates to data profiling as well)
Data Profiling Policies¶
In order to be able to calculate the cost of data it is useful to know---per dataset---what was paid to obtain or create the data.
What are the pricing options for users of the data.
Usage Agreement Policies / Service Level Agreements¶
Dataset-specific service level agreements or policies to create such agreements.
Usage Metric Collection Policies¶
How is the data used and how are usage metrics collected and is that mandatory or optional, depending on the Usage Agreement Policy.
Revenue can be determined in multiple ways, ideally all usage of all data covered by the SDD is priced and charged for (see pricing policies and usage metrics) which would make the revenue computation very easy but usually data is not priced (because there is no infrastructure and priority for it) so the calculation of the value of the data becomes much more complicated and can only be derived from the position of the dataset in the context of the larger data supply chain.
E.g. cost of creation, maintenance, "cost of data".
E.g. in which processes, use cases, LOBs is the data used and deemed to be critical.
- determine the value of data
- determine what the criticality is of the data (for regulatory, risk and business continuation purposes)
- determine whether alternative sources may be available
- determine whether business processes could be improved
E.g. security classification, PII data, confidentiality, criticality, per context/use case.
Data Life Cycle¶
How is temporality being dealt with, which parts of the dataset are just reflecting the "current state" and which parts are historic (or future / "what if") data? And which design patterns are used to maintain temporality? Smart model-driven EKG/Platform services can then a) maintain temporality and b) use it properly.
Data Retention & Archiving Policies¶
Physical Dataset Policies¶
Given the logical dataset---the SDD---which physical manifestations does it have, what types, which ones are allowable etc.
For instance, one SDD produced overnight by a batch-job, stored as an RDF file in an S3 bucket, may be loaded into different databases of different types at different locations.
In the context of a more mature EKG platform, smart services may wish to control the proliferation of data in all its various manifestations across the enterprise and enforce policies.
E.g. which data may or may not appear in logs or only in obfuscated form.
Obfuscation / Masking Policies¶
How does the actual obfuscation or masking take place, what are the technical mechanisms used for which particular datapoints? (encryption, hashing, which fields are randomized etc).
- ensure that no confidential data appears in any logging or monitoring systems and the like.
- ensure that proper obfuscation or masking policies are enforced when generating test datasets.
- ensure that, even in production, only entitled personas (see entitlement policies) can see the non-obfuscated versions of certain datapoints.
- make it possible that obfuscated versions of datapoints may be part of larger objects seen by certain personas in certain use cases without having to leave those datapoints out of the picture altogether.
TODO: Add a side-box about "computations over encrypted data" (see also Encryption Policies)
Test Data Policies¶
Policies to create test-datasets derived from the production version of the given SDD.
This is not about the tests themselves, which are usually done in the context of a use case, but more about how to create realistic test-data in the first place.
- Even for public datasets, especially if they are very large, it may be useful to have a policy (model) for the creation of a subset of the data that can realistically be used for testing purposes that contains all the relevant edge cases.
- For confidential data, masking policies or obfuscation policies may come into play.
- Goal is to have smart EKG/Platform Services that generate test data on demand for DEV/TEST/UAT environments.
Query & Search Policies¶
E.g. which methods are supported for advanced queries or search requests.
E.g. what are the recommended/mandatory instructions for inference engines.
- determine whether it is mandatory to host the SDD in an environment where a given type of inference engine is available.
- datasets with datapoints that are tied to OWL ontologies that are heavily relying on reasoning
- requires OWL-DL or OWL-RL etc
- is designed to run with reasoning with multiple schemas (see for instance Stardog's Reasoning with Multiple Schemas)
- requires to be loaded in a triplestore product that supports forward chaining (i.e. Ontotext GraphDb) or backward chaining (i.e. Stardog)
E.g. how long can any given datapoint reside in a cache such as an HTTP proxy etc.
- create architectures that heavily rely on caching to improve performance
- inform SPARQL query developers about their options to cache the results of any given SPARQL statement (since SPARQL is an HTTP protocol its quite natural to leverage the standard HTTP caching facilities)
- enable smart caching services to automatically flush stale entries from their caches intelligently
- ensure that data retention policies are also enforced by making sure that certain datapoints do not stick around in long-term caches and content delivery networks etc.
Issue Management Policies¶
Change Management Policies¶
E.g. which datapoints can only be retrieved in encrypted form or could be used in encrypted form by certain computations.
Note that this list of topic areas does not include "use cases" since use cases are assumed to be defined elsewhere (in fact, in their own datasets) where the use case links to the dataset but not vice versa because each self-describing dataset is assumed to be designed and developed independently from any given use case. Self-describing datasets are "use case agnostic" and as much as possible "unbiased", supporting existing use cases and any number of unknown future use cases.