Anatomy of a metric that lied

The number on slide 4 was net revenue for last quarter. It was wrong by about a hundred thousand dollars, and not one of the six people who touched it on the way there had made a mistake they could see. Each hop did something reasonable. The meaning of the number changed anyway.

Net revenue, last quarter, at each hop

Illustrative values, made up to show the mechanism. Nothing here is measured.

$4,256,112

at stg, +$73,612 vs truth

hop 1/6Currency converted twicein stg_orders

$4.18M to $4.26M (+$73,612)

- amount_usd

+ amount_usd * fx.rate as amount_usd

The payments API already lands amount_usd. Staging multiplies it by the day's EUR rate again, so every EUR order grows by 8 percent.

Ask: What unit is this column in, and who converted it?

slide says $4.30M, should be $4.18M

Hop 1: staging converts EUR twice

The raw orders table already carries a USD amount from the payments API. Staging multiplies it by the day's rate anyway. Every EUR order is now 8 percent bigger, and every test passes.

Hop 2: the rollup forgets refunds

There are two intermediate models. One subtracts refunds. The daily rollup the mart reads is the other one. "Net" in the column name, gross in the column.

Hop 3: the quarter ends in UTC

Finance closes the quarter at midnight Pacific. The mart truncates in UTC. The last evening of the quarter quietly becomes the first evening of the next one.

Hop 4: someone loosened a filter

The explore used to exclude test accounts by type. A partner account needed to show, so the filter became an email pattern. The QA team does not put "test" in their emails.

Hop 5: the tile rounds

The dashboard shows $0.0M. Nothing wrong with that on a tile. The problem is that the rounded value is the one that leaves the tool, and nobody downstream knows.

Hop 6: the screenshot is three weeks old

The tile was captured the morning after close. A marketplace sync gap was backfilled two days later. The slide was pasted once and never refreshed. Errors partly cancel, which is worse: the total looks plausible.

Inspect it

Click any hop or node to read its diff. Toggle each fault on or off and watch the number on the slide move. Every value is illustrative.

What the graph is showing

The walk above is a composite. Every value is made up, and the faults are the ones I keep finding, in the order I usually find them. Two of them I have shipped myself.

$4.18M

what the quarter's net revenue should be

illustrative model

$4.3M

what the slide said

illustrative model

~$485k

total movement across six hops

sum of absolute changes, illustrative

The part that matters is not the size of the error. It is that the six errors do not point the same way. A double conversion and a missing refund join push the number up. A timezone cut and a stale screenshot pull it down. The final figure lands close enough to plausible that nobody looks twice. A number that is wildly wrong gets fixed the same afternoon. A number that is quietly wrong runs for three quarters.

Lineage shows the graph, not the meaning

I like lineage tools. Column-level lineage is a real improvement, and I would not run a warehouse without it. But look at what the tool actually tells you at each edge: this column feeds that column. It does not tell you that the unit changed from EUR to USD, or that “net” stopped meaning net, or that the period boundary moved by seven hours, or that the value was rounded before it left the building.

Those are semantic changes. Lineage tracks structural ones. The graph is correct at every hop in the story above, and every hop still lies.

The last two hops are the ones lineage cannot see at all. A dashboard tile’s number format is not in the DAG. A screenshot pasted into a deck is not in anything. The metric’s most consequential hop, the one where it reaches a person who makes a decision, is the one with the least tooling.

Keep a definition changelog

Tests catch values that violate a rule. They do not catch a rule that changed. The fault at hop 4 passes every test because the test says “revenue is not null and not negative” and the loosened filter satisfies both.

What catches semantic drift is a written record of what the number means, kept next to the model, with a dated entry every time the meaning changes. I call it a definition changelog. It is a markdown file, or a meta block in the dbt yml, and it looks like this:

net_revenue
  2026-04-02  net of refunds and chargebacks; excludes account_type = 'test'
  2026-07-18  refunds now joined from int_order_economics, not stg_refunds
  2026-09-11  test filter changed to email pattern (partner account request)

Three lines. The third line is the one that would have caught hop 4 in review, because a reviewer reading “test filter changed to email pattern” asks the obvious question. A diff of a where clause in a BI tool’s YAML asks nothing.

The changelog does two other things. It gives the slide a footnote: “net revenue as defined on 2026-09-11”. And it gives you a place to write down the unit and the timezone, which is where hops 1 and 3 die.

What lineage cannot fix

Hop 6 is a process problem, not a data problem. Someone pasted a picture of a number into a deck. The fix is boring: the slide links to the tile, or the slide is generated from the tile, or the deck has a “data as of” stamp that someone is responsible for. Pick one. The screenshot is the most common way a correct pipeline delivers a wrong number, and no amount of dbt tests will touch it.

What to do this week

Pick your one metric that reaches an executive. Draw its hops by hand, including the deck. The tool will not show you the last two.
At each hop write one line: unit, gross or net, period boundary, exclusions, capture time. If you cannot fill a line, that is the finding.
Start the definition changelog for that metric. Backfill it from git blame if you have to.
Put a “data as of” stamp on the slide. Make one person own it.
Toggle the faults above one at a time and notice how small each one looks on its own. That is the point.

The number on the slide was not a bug. It was six small definitions drifting apart while everything stayed green. The graph shows you where the number went. Only the changelog tells you what it meant when it got there.