concept · concepts/Data Gravity.md

Data Gravity

status: active · created 2026-06-26 · 2 sources · #data-engineering #platform #architecture #performance #craft

Large data is expensive to move, so it exerts a "gravity" that should pull compute toward it: move the small thing (the query, the spec) to the data — not the data to the app.

Grounded 2026-06-26 by McCrory — Data Gravity in the Clouds (2010), the post that coined the term — fetched under the auto-ground convention. The source confirms the heuristic and adds the originating analogy (data as a mass) and the mechanism (latency + throughput).

Explanation

The source (Working Heuristics — Data, Matching, Product, Org, Tooling) frames it as a placement rule: put compute where the data has gravity. Between the two things you could move — the data or the logic that operates on it — move whichever is smaller. The data is almost always the heavy object: pulling a large dataset into an application to process it there incurs transfer cost, staleness, duplication, and a second place for it to drift. Pushing the query or the transformation spec down to where the data already lives avoids all of that.

Dave McCrory's coinage (McCrory — Data Gravity in the Clouds, 2010) supplies the underlying physics: treat data as a mass that, as it accumulates, attracts applications and services. The "accelerators" of that pull are latency and throughput — the closer compute sits to the data, the faster it runs and the more reliant it becomes. And the decisive asymmetry: "data if large enough can be virtually impossible to move," so the compute gravitates to the data, not the reverse.

Why it matters / where it shows up

  • The cost asymmetry is large and one-directional. A query is bytes; a dataset is gigabytes-to-petabytes. Moving the small thing is cheaper by orders of magnitude and stays correct (no copy to keep in sync).
  • It's the placement sibling of Data Contract. Both say do the work at the right place: the contract derives once at the boundary; data gravity computes where the data lives. Re-locating or re-deriving is the shared waste.
  • It generalizes to platform choice. Where the data has gravity often is the hardest, most-recurring part of the workflow — which is why it connects to Choose the Platform by the Hardest Recurring Part.

Open questions

  • When the "small thing" (a model, a heavy UDF) stops being small, where does the gravity argument flip — and is moving a data subset out ever the right call?

Connections

Sources

The wiki is the source. This door reads it at build time and never writes to it.