Associate Data Engineer

The system

Shell’s EV and e-mobility data platform: charge-session records, station telemetry and billing feeds arriving nightly from a couple of dozen charging networks, feeding reporting the business actually used.

I was one of five engineers operating it. Around 40,000 charge-session records a night, and enough runs failed that recovery was a standing item on the board rather than an exception.

What was hard

Failures that all looked identical. A failed run reported that it had failed. It did not say which source, which field, or which assumption had broken. Most of the work was diagnostic before it was corrective — moving validation to ingestion so that the error names the problem.

Schema drift from partners we did not control. Charging networks change their payloads without telling anyone, and there is no version negotiation to lean on. We asserted shape at the boundary and failed one source loudly, instead of letting a null travel three transforms downstream and surface as something unrecognisable.

Being the junior on a tired team. Standing practice for a failed job was to re-run it, because that usually worked. Arguing for root-cause work meant showing it was cheaper, which meant fixing two pipelines on my own time before anyone would spend theirs.

Between validation and root-cause fixes the error rate fell by roughly 75%, and automating the manual ingestion steps removed about 50% of the manual effort.

What I would do differently

I fixed pipelines one at a time. Twenty pipelines had perhaps four underlying causes between them, and I would now spend the first fortnight categorising rather than repairing.