System Note 07 · Enterprise Systems

MongoDB → BigQuery Migration

Bell Canada

GCPBigQueryMongoDBTerraformCloud RunGitLab CI/CDPython
Private build
What it is

The pipelines and cloud setup that copy a 500-million-row MongoDB collection into BigQuery and prove, row by row, that the copy is correct. Built as code — schemas, extraction jobs, and infrastructure all deploy through CI/CD with no clicking around a cloud console.

500M-row table · 5 ingestion pipelines · Terraform + Cloud Run + GitLab CI

Why I built it

The team had never moved a NoSQL source into the warehouse before, and the asset was too big and too important to eyeball.

What went wrong
  1. 01
    The silent NULLs
    Broke
    Some columns loaded as all-NULL at multi-million-row scale, with no error to warn anyone.
    Why
    The loader figured out which source fields to read from the destination column names. Any naming mismatch just read nothing — and quietly wrote NULL instead of failing.
    Fix
    Made the projection explicit and made a mismatch an error, not a shrug. Also chose a hybrid schema (real partitioned/clustered columns plus one JSON payload column) so queries stay fast at 500M rows.
  2. 02
    The delta that wasn’t
    Broke
    The largest collections kept timing out on what was supposed to be a small incremental load.
    Why
    The “delta” job had an empty filter, so every run was secretly a full scan of the entire collection.
    Fix
    Fixed the filter so it only pulls changes, and tuned batch size and parallelism after hitting BigQuery’s write-rate limit. Then built a three-tier reconciler — sampled diff, full ID set-diff, and a resumable every-row compare — to prove the two systems agree.

Evidence before confidence.

← Back to Build Notes