Masking guide

Masking replaces sensitive values with stand-ins and keeps the dataset usable: the same input value always gets the same stand-in, in every table, so keys still join. dogm mask does it on your own machine, with a key you hold, so the raw rows never reach DOGM. It's free on every plan and needs no account.

What you need#

• The dogm CLI (build it from source until the first release).
• Your export as one CSV or JSONL file per table, in one directory: customers.csv, orders.csv.
• A mask spec that describes those tables.

1. Make a key#

dogm keygen --out key.bin

wrote a 32-byte mask key to key.bin
keep it secret: anyone with it can rebuild the mapping for candidate values

The key is 32 random bytes from your operating system. keygen never overwrites a file, so you can't lose a key by running it twice. Keep it out of version control. Store it the way you'd store a database password, because the same key and key version always give the same mapping, and that's what lets you mask next month's export consistently with this month's.

2. Describe the export#

A mask spec uses the spec format with source: external tables. Columns declare a type and, where they hold personal data, a pii type; primary_key and ref tell the engine which columns join. This one is the engine's own test fixture:

spec_version: 1
tables:
  customers:
    source: external
    primary_key: { column: customer_id }
    columns:
      customer_id: { type: string }
      full_name:   { type: string, pii: person.name }
      email:       { type: string, unique: true, pii: person.email }
      tin:         { type: string, pii: ph.tin }
      city:        { type: string, pii: ph.city }
      birthdate:   { type: date, pii: person.birthdate }
      notes:       { type: string, nullable: 1, pii: person.name }
  orders:
    source: external
    primary_key: { column: order_id }
    columns:
      order_id:    { type: string }
      customer_id: { ref: customers.customer_id }
      order_date:  { type: date }
      total_php:   { type: money }
      status:      { type: string }
mask:
  key_ref: local_key
  columns:
    customers.customer_id: { mode: fpe_ff1, domain: customer }
    orders.customer_id:    { mode: fpe_ff1, domain: customer }
    customers.full_name:   { mode: consistent }
    customers.email:       { mode: consistent, from: customers.full_name, on_collision: suffix }
    customers.tin:         { mode: fpe_ff1 }
    customers.city:        { mode: consistent }
    customers.birthdate:   { mode: shift_date, domain: customer_dates, entity: customers.customer_id, max_days: 60 }
    orders.order_date:     { mode: shift_date, domain: customer_dates, entity: orders.customer_id, max_days: 60 }
    customers.notes:       { mode: redact }
    orders.order_id:       { mode: hash, domain: order, prefix: "ORD-", length: 16 }

Every column with a pii type needs a mask entry, or the compiler stops with E_MASK_PII_UNMASKED. That's on purpose: a personal-data column can't slip through because you forgot it. dogm compile mask.yaml checks the spec without touching any data.

3. Mask#

dogm mask mask.yaml --in ./export --out ./masked --key-file key.bin

masked 2 tables, 30 rows, to ./masked (key version 1)
  customers.csv 12 rows sha256 afb5f01a…
  orders.csv 18 rows sha256 c42c2e95…
checks 18 passed, 0 failed
note: Masking is pseudonymization, not anonymization: …

./masked gets one file per table plus the report (report.json, report.junit.xml, report.sarif). The checks cover types and nulls, uniqueness of masked unique columns, and that every masked foreign key still exists among the masked parent keys. Rules aren't evaluated on real data.

Here's one customer before and after. Your key gives different values, which is the point:

customer_id,full_name,email,tin,city,birthdate,notes
C-1000002,Quintessa Zabaleta,qzabaleta@mailbox.ph,234-567-890-000,Cebu City,1991-07-02,"Called twice, wants a refund"
C-3997606,Alfonso Cordero,alfonso.cordero.1gw57zsv2peas@example.net,344-196-757-359,Angeles,1991-07-25,

The ID and TIN keep their format, the name and city come from fictional pools, the email is rebuilt from the masked name on an example domain, the birthdate moved by a keyed offset, and the free-text note is gone. In orders.csv, that customer's orders carry the same masked ID, so the join still works.

Modes#

Keys, domains and joins#

• Domains. A column's domain defaults to its pii type; a masked column with neither pii nor domain is E_MASK_NO_DOMAIN. Columns with the same domain and key mask identically by value, so a name in customers and the same name in replies get the same stand-in.
• Foreign keys. When a key column is masked, every column that references it must use the same mode and domain, or joins would break. The compiler checks this (E_MASK_FK_MISMATCH). The FF1 tweak comes from the domain label, never the column, which is why customers.customer_id and orders.customer_id above share domain: customer.
• Unique columns. consistent maps into a finite pool and can collide, so a unique column needs fpe_ff1, hash, or consistent with on_collision: suffix.
• Emails move to an RFC 2606 example domain. keep_domain: true keeps the original domain, which can reveal the employer.
• Free text. This version can't find personal data inside free text, so a text column with a pii type must be redacted (E_MASK_FREE_TEXT).
• Key versions. --key-version (default 1) is mixed into the key, so a new version is a new mapping. The version is recorded in the report.

Masking generated data#

A scenario spec can carry a mask block too, and dogm generate then masks each chunk after validation. It needs --key-file, or --no-mask to write unmasked data on purpose; it never silently skips the mask. In the GitHub Action, the key comes from a secret.