MongoDB → BigQuery Migration
Bell Canada
The pipelines and cloud setup that copy a 500-million-row MongoDB collection into BigQuery and prove, row by row, that the copy is correct. Built as code — schemas, extraction jobs, and infrastructure all deploy through CI/CD with no clicking around a cloud console.
500M-row table · 5 ingestion pipelines · Terraform + Cloud Run + GitLab CI
The team had never moved a NoSQL source into the warehouse before, and the asset was too big and too important to eyeball.
- 01The silent NULLs
- Broke
- Some columns loaded as all-NULL at multi-million-row scale, with no error to warn anyone.
- Why
- The loader figured out which source fields to read from the destination column names. Any naming mismatch just read nothing — and quietly wrote NULL instead of failing.
- Fix
- Made the projection explicit and made a mismatch an error, not a shrug. Also chose a hybrid schema (real partitioned/clustered columns plus one JSON payload column) so queries stay fast at 500M rows.
- 02The delta that wasn’t
- Broke
- The largest collections kept timing out on what was supposed to be a small incremental load.
- Why
- The “delta” job had an empty filter, so every run was secretly a full scan of the entire collection.
- Fix
- Fixed the filter so it only pulls changes, and tuned batch size and parallelism after hitting BigQuery’s write-rate limit. Then built a three-tier reconciler — sampled diff, full ID set-diff, and a resumable every-row compare — to prove the two systems agree.
Evidence before confidence.