Skip to content
← Back to projects
Systems Architect · 2022 - now

Private Data & Event Pipeline Architecture

Event-driven ingestion and processing backbones for automation and bookkeeping workflows, with schema validation, auditability and ML enrichment, run as durable workflows that survive restarts and can be replayed.

The problem with most automation is not the happy path. It is the mailbox that goes offline halfway through a sync, the dunning run that must not send the same reminder twice, the model call that times out after the document was already filed. A pipeline that cannot resume from exactly where it stopped ends up with a person re-checking every step by hand, which is the work it was meant to remove.

The backbone is a set of durable workflows on Temporal, written in Go. Every step is idempotent, so a retry after a crash produces the same result rather than a duplicate. Signals between workflows are versioned, so a running process keeps its contract while the code around it changes. Recorded workflow histories are replayed as tests, which turns an incident into a regression test the moment it has been understood. The same backbone runs mailbox synchronisation, the dunning process of an ERP, and the control plane of a coding-agent service on Kubernetes.

Documents and transactions enter through typed intakes with schema validation and land as events with their source attached: the original file, the bank transaction, the model output with its confidence score. Enrichment on private inference, classification and extraction included, adds structure without the data leaving the network. Where a result changes the books or leaves the system, the owner approves it first, and the approval is part of the record.