How to bring your own personal-data dataset

You hold or licence a dataset about people: a loyalty-scheme export, a fraud-marker list, a customer or account file, the kind of linked records a credit bureau or a fraud database keeps. This page shows how to map it onto your investigation graph so a lookup on an email address, a phone number or a card number reaches the person, the addresses, the accounts and the markers tied to it, without the raw identifiers sitting in a file anyone could read and without the records leaving your machine.

Viokit is the tool here, never the data provider. It does not fetch, buy or hold this data. You bring a file you already hold lawfully, and you map it.

Before you start

You are responsible for holding and using the data lawfully. Viokit cannot tell where a file came from or whether you may hold it. The manifest makes you write down the lawful basis, the source and the licence, because a later reviewer will ask for them. Writing them down records your claim; it does not make the claim true. If the dataset is personal data, take advice before you rely on it. See the operate page and Offline datasets for the law that applies and for what Viokit does to keep the data in.

Viokit never obtains leaked data. A file whose data comes from a breach, a leak, a stealer log or an unlawful scrape stays out, whatever it would add.

Make a starter

The viokit-dataset tool writes a starter manifest and a fabricated sample you can edit:

viokit-dataset init ./my-dataset --id loyalty-markers

That writes loyalty-markers.dataset.json and a fake loyalty-markers.csv. Open the manifest and describe your real file.

Describe your file

The manifest says four things.

  1. Where it comes from and on what basis. Fill in source (how and from where you obtained it), license (the licence you hold it under) and lawfulBasis (the lawful basis you rely on, in your own words). The tool refuses a manifest that leaves any of these empty.
  2. What each row is. entity names the kind each row is (a person), the column that holds its stable id, and the columns that are its identifiers (a name, a date of birth, a marker reason). links relate the row to other things it names: a home address, a loyalty or credit account, a device, a credit application. Each link's column holds the whole key of the thing it points at, so your dataset and Viokit's other sources meet on one node.
  3. What to look it up by. keys lists the identifiers a lookup can start from: an email address, a phone number, a card number. Each key says how it is hashed (see below) and how the row's person relates to it (the person uses-email the address, holds-account the card).
  4. What to reveal only after a match. reveal marks the fields that stay hidden until the case records a basis for showing them: a date of birth, a full home address, a fraud-marker reason. The match still runs; those fields wait for the basis.

Hashing keeps the identifiers out of the index

When you build the dataset, Viokit hashes the lookup identifiers. The index can tell that an email address is present, but it does not hold a list of the email addresses in the file. So a copy of the index is not a copy of everyone's contact details.

  • Email, phone, username and address keys use a salted hash, with a random salt made for this dataset. A lookup hashes the value you type and matches it. The node it builds is the value you already hold, so the clear value never has to be stored.
  • A card number uses a keyed hash, so a full card number reaches one card node while the graph shows only the first six and last four digits, the scheme and the expiry. It needs a deployment card key (VIOKIT_CARD_KEY) set as a secret, kept out of the datasets folder. With no card key, card lookups are off.

A hash of an email, a phone or a card number is not encryption. It defeats a generic lookup table and stops the index being read at a glance. It does not make a stolen index safe. The real protection stays the same as for every offline dataset: no network path, a local index, every lookup audited, and the data withheld from exports.

Build it

viokit-dataset validate ./my-dataset/loyalty-markers.dataset.json
viokit-dataset build    ./my-dataset/loyalty-markers.dataset.json --out ./my-dataset

build reads your file, hashes the identifiers, writes a prepared file that holds no identifier in clear, compiles a Viokit dataset manifest, writes a sidecar, and builds the index. inspect shows what a built dataset holds without running a lookup:

viokit-dataset inspect ./my-dataset --id loyalty-markers

Looking people up

A dataset whose rows are people is held by the purpose gate: a lookup is refused until the case records a purpose, the same rule every person search follows. A field you marked revealed-after-match stays hidden until the case records the basis you named for it. Every match is recorded in the dataset audit with the lawful basis, so the log answers not only who looked up whom but under what basis they revealed it.

A worked example, built from obviously fake data, is in the @viokit/dataset-sdk package under examples/. See also the format and hashing in detail.

Docs

↑↓ to moveEnter to openEsc to close