How to bring your own personal-data dataset
You hold or licence a dataset about people: a loyalty-scheme export, a fraud-marker list, a customer or account file, the kind of linked records a credit bureau or a fraud database keeps. This page shows how to map it onto your investigation graph so a lookup on an email address, a phone number or a card number reaches the person, the addresses, the accounts and the markers tied to it, without the raw identifiers sitting in a file anyone could read and without the records leaving your machine.
Viokit is the tool here, never the data provider. It does not fetch, buy or hold this data. You bring a file you already hold lawfully, and you map it.
Before you start
You are responsible for holding and using the data lawfully. Viokit cannot tell where a file came from or whether you may hold it. The manifest makes you write down the lawful basis, the source and the licence, because a later reviewer will ask for them. Writing them down records your claim; it does not make the claim true. If the dataset is personal data, take advice before you rely on it. See the operate page and Offline datasets for the law that applies and for what Viokit does to keep the data in.
Viokit never obtains leaked data. A file whose data comes from a breach, a leak, a stealer log or an unlawful scrape stays out, whatever it would add.
Make a starter
The viokit-dataset tool writes a starter manifest and a fabricated sample you can edit:
viokit-dataset init ./my-dataset --id loyalty-markers
That writes loyalty-markers.dataset.json and a fake loyalty-markers.csv. Open the manifest and
describe your real file.
Describe your file
The manifest says four things.
- Where it comes from and on what basis. Fill in
source(how and from where you obtained it),license(the licence you hold it under) andlawfulBasis(the lawful basis you rely on, in your own words). The tool refuses a manifest that leaves any of these empty. - What each row is.
entitynames the kind each row is (aperson), the column that holds its stable id, and the columns that are its identifiers (a name, a date of birth, a marker reason).linksrelate the row to other things it names: a home address, a loyalty or credit account, a device, a credit application. Each link's column holds the whole key of the thing it points at, so your dataset and Viokit's other sources meet on one node. - What to look it up by.
keyslists the identifiers a lookup can start from: an email address, a phone number, a card number. Each key says how it is hashed (see below) and how the row's person relates to it (the personuses-emailthe address,holds-accountthe card). - What to reveal only after a match.
revealmarks the fields that stay hidden until the case records a basis for showing them: a date of birth, a full home address, a fraud-marker reason. The match still runs; those fields wait for the basis.
Hashing keeps the identifiers out of the index
When you build the dataset, Viokit hashes the lookup identifiers. The index can tell that an email address is present, but it does not hold a list of the email addresses in the file. So a copy of the index is not a copy of everyone's contact details.
- Email, phone, username and address keys use a salted hash, with a random salt made for this dataset. A lookup hashes the value you type and matches it. The node it builds is the value you already hold, so the clear value never has to be stored.
- A card number uses a keyed hash, so a full card number reaches one card node while the graph shows
only the first six and last four digits, the scheme and the expiry. It needs a deployment card key
(
VIOKIT_CARD_KEY) set as a secret, kept out of the datasets folder. With no card key, card lookups are off.
A hash of an email, a phone or a card number is not encryption. It defeats a generic lookup table and stops the index being read at a glance. It does not make a stolen index safe. The real protection stays the same as for every offline dataset: no network path, a local index, every lookup audited, and the data withheld from exports.
Build it
viokit-dataset validate ./my-dataset/loyalty-markers.dataset.json
viokit-dataset build ./my-dataset/loyalty-markers.dataset.json --out ./my-dataset
build reads your file, hashes the identifiers, writes a prepared file that holds no identifier in
clear, compiles a Viokit dataset manifest, writes a sidecar, and builds the index. inspect shows what
a built dataset holds without running a lookup:
viokit-dataset inspect ./my-dataset --id loyalty-markers
Looking people up
A dataset whose rows are people is held by the purpose gate: a lookup is refused until the case records a purpose, the same rule every person search follows. A field you marked revealed-after-match stays hidden until the case records the basis you named for it. Every match is recorded in the dataset audit with the lawful basis, so the log answers not only who looked up whom but under what basis they revealed it.
A worked example, built from obviously fake data, is in the @viokit/dataset-sdk package under
examples/. See also the format and hashing in detail.