Skip to content

Orchestration

One run.Every step checked.

A DOGM run compiles the spec, generates the tables, checks every row and writes the files, a checks report and one output hash you can reproduce. Hosted runs do all of it in the app today, masking included. The dogm CLI runs the same engine on your machine; until its first release, you build it from source.

Try the demo workflow See the CI gate
One run, every step checkedA workflow track with four steps: generate, validate, mask and deliver. A packet travels through each step, each step gets a check mark, and the run ends in a report with a reproducible hash.generatevalidatemaskdeliverreportchecks4 / 4rows1,284sha2569f2c41e0

A run, start to finish

Five steps.One record.

The engine plans the run before it generates anything. Work is cut into fixed chunks, so ten workers or one produce the same output.

  1. 1

    Compile

    Check the approved spec and turn it into a plan: row counts, child offsets and chunk boundaries.

  2. 2

    Generate

    Each chunk computes its rows independently. All children of one parent stay in the same chunk.

  3. 3

    Validate

    Every chunk checks keys, references, types and rules on the values it just wrote.

  4. 4

    Mask

    Columns named in the mask block get keyed stand-ins, after validation, so masked keys still join.

  5. 5

    Deliver

    Parts, manifest.json and the reports go to disk, or to storage for hosted runs.

In the demo The demo runs a four-step version of this workflow, with real checks, in your browser.

The checks report

Proof the dataholds together.

A failed check is a result, not a crash. The run still finishes, and the report names the rows that broke each rule.

Checks run on every generated dataset
CheckHow
Primary keysUnique by construction. Each value is verified by reversing the Feistel shuffle.
ReferencesEvery foreign key is checked against its parent's recomputed number.
Row countsRoot tables exact; child tables match the plan's offsets.
RulesEvery rule on every row, before masking. The report keeps the count and the first 20 failing row keys.
Types and nullsEach value matches its column type; nulls appear only in nullable columns.
UniquenessOther unique columns are merged across chunks by value hash to find duplicates exactly.

The report is JSON, with JUnit XML and SARIF versions for CI.

report.json (excerpt, illustrative)
  • primary_keys passed
  • references passed
  • row_counts passed
  • types_and_nulls passed
  • uniqueness passed
  • rule: payments.paid_on >= parent.order_date
  • failed on 2 rows: PAY-0004417, PAY-0000932

Reproducibility

Rebuild any run,and prove it matches.

Every run writes a manifest.json, and its SHA-256 is the run's output_sha256. Change one byte of output and that value changes.

What the manifest records

  • Engine version
  • Spec hash
  • Seed
  • Row overrides
  • Every part: table, row range, row count and SHA-256, in order

output_sha256 = SHA-256(manifest.json)

The Reproduce command

terminal
$ dogm generate spec.yaml --seed 4242 --rows customers=48 --out ./data

Run it with the engine version recorded on the run and you get the same output_sha256. Every released CLI version will stay installable, so old runs stay reproducible.

CLI, from source Every hosted run shows its own version of this command under Reproduce. The CLI that runs it is built from source until its first release.

Webhooks

Live in the beta

Tell your systems when a run ends.

On the Pro and Team plans.

Events

  • run.succeeded
  • run.failed
  • spec.needs_review
  • Each body is signed with HMAC-SHA256 using a per-endpoint secret, and carries a timestamp to block replays.
  • Failed deliveries retry with backoff for about 24 hours.
  • Endpoints must be HTTPS; private address ranges are blocked.
  • An endpoint whose every delivery has failed for 24 hours is disabled until you turn it back on.

CI gate

Fail the buildwhen the data is wrong.

The DOGM GitHub Action generates a spec's data in your workflow, validates it and fails the step if any check fails.

Live in the beta
  1. 1Installs the dogm CLI on the runner and checks it against the release's SHA256SUMS.
  2. 2Runs dogm generate with the seed from the workflow file, then dogm validate.
  3. 3Uploads report.junit.xml and fails the step if any check failed.

Hosted mode works now: it runs an approved spec version with a project API key, on Pro and Team, and every beta workspace has Pro limits. Local mode, the default, makes no API call and is free on every plan, but it installs the CLI from its first public release, which isn't out yet.

Read the GitHub Action guide

.github/workflows/data.ymlLocal mode
# Local mode. Works once the CLI's first release is public.name: test dataon: [pull_request]permissions:  contents: readjobs:  data:    runs-on: ubuntu-latest    steps:      - uses: actions/checkout@v4      - uses: newmachinesph/dogm/dogm-action@<commit-sha>        with:          spec: specs/ecommerce.yaml          seed: 4242          version: 0.1.0

Early-access beta

Try the demo now.Or start free.

The in-browser demo generates, masks and checks a connected sample, and nothing leaves your browser. Sign up and your workspace gets Pro limits at no cost while the beta lasts.

Try the demo Start free