Benford on my own data · Edit Distance

Benford’s law says the first digit of a naturally occurring number is a 1 about 30% of the time and a 9 under 5% of the time. It is the closest thing fraud auditors have to a party trick. I wanted to know how it behaves on data I already had lying around, so I ran it on every numeric file behind this blog.

What Benford predicts for a first digit

Share of numbers starting with each digit, log10(1 + 1/d). Bars are the law; data comes next.

Bars are observed shares, hollow dots are Benford, whiskers are 2 standard errors at this n. Bands are Nigrini's first-digit MAD bands. Data: this blog's dataset-size, salary, and tool-exit files.

The law

In many real collections of numbers, about 30% start with a 1 and under 5% start with a 9. The share for digit d is log10(1 + 1/d). This is the whole law.

Dataset sizes, 2,981 of them

Bytes on disk for the datasets from the small-data post, spanning 12 orders of magnitude. MAD 0.0140, which Nigrini calls marginal. Zenodo alone gets 0.0117, acceptable. The two Hub samples drag it off the curve: fewer 1s than expected, more 6s.

Salaries: 3,104 of them

Annual pay sits between $15k and $431k, so 62% of values start with a 1 and almost none with a 3 or 4. MAD 0.0714. Nothing is fabricated here. The range is too narrow for the law to apply.

Nine deal prices

Only 9 of the 78 tool exits have a public price. Look at the whiskers: at this n almost any bar fits. MAD 0.0946 means nothing.

Make some up

Type 30 numbers you invent, or press Random 30 for uniform draws between 1 and 999. Uniform numbers give every first digit an equal chance, and the shape goes flat.

Explore

Pick any set and flip to the second digit. Second digits are nearly flat by design, which is why they are a tougher test than the first.

What the numbers say

0.0140

MAD, 2,981 dataset sizes

marginal conformity, 12 orders of magnitude

0.0714

MAD, 3,104 salaries

nonconformity, 1.5 orders of magnitude

0.0946

MAD, 9 deal prices

too few values to grade

Take the first significant digit of every value, count each digit’s share, and compare to log10(1 + 1/d). I score the gap with mean absolute deviation across the nine digits. The bands I quote are Nigrini’s, from his book on the law: under 0.006 is close conformity, 0.006 to 0.012 acceptable, 0.012 to 0.015 marginal, and anything over 0.015 is nonconformity. All the figures below come from scripts/benford-stats.mjs, and the piece above recomputes them in the browser from the same files.

The dataset sizes almost behave

The sizes come from the small-data post: bytes on disk for 2,981 public datasets across three catalogues, from 246 bytes to about 307 TB. That is 12 orders of magnitude, which is the kind of range Benford likes. Combined, they score a MAD of 0.0140, marginal. Zenodo’s research datasets on their own score 0.0117, acceptable, with 30.5% of sizes starting with a 1 against the predicted 30.1%.

The two Hugging Face samples are worse: 0.0200 for the recently updated set and 0.0256 for the most downloaded. Both have too few 1s and a bump at 6. I do not know why. My guess, flagged as a guess, is that Hub repositories often store the same data in several formats, which moves sizes around for reasons unrelated to how the data was generated.

The point stands. A set spanning twelve powers of ten lands near the curve without anyone trying, and the misses are small enough that I would look at the data before calling them a problem.

The salaries do not, and that is fine

The salary file has 3,104 full-time responses from a public survey, all between $15,000 and $430,967. That is a span of 1.46 orders of magnitude. 62.2% of the values start with a 1, because most tech salaries sit between $100k and $199k. Only 2.7% start with a 3 and 1.7% with a 4, because almost nobody in this survey earns $30k or $40k, and few earn $300k or more. MAD 0.0714, deep nonconformity.

Nobody made these salaries up. The set fails because it breaks the first precondition: the numbers have to span several orders of magnitude. A distribution squeezed into one decade always has a favourite first digit, and that digit tells you about the range, not about honesty.

Nine deal prices say nothing

The tool-graveyard file lists 78 exits, and only 9 of them have a public price. The first-digit MAD is 0.0946, which would be alarming with 3,000 values. With 9 values it is noise. The whiskers in the fourth scene are two binomial standard errors at that n, and they swallow the bars. I refuse to grade anything under 100 values. That cutoff is mine, not Nigrini’s.

Made-up numbers go flat

The fifth scene lets you type your own set. Uniform random draws between 1 and 999 give every first digit an equal shot, so the shape flattens toward 11% per digit. The 30 draws the script makes with seed 42 score 0.0868, and 3,000 draws score 0.0602 with every digit between 10% and 12%. Invented numbers tend the same way, but none of my files has a fabricated column, so I am only reporting the uniform case.

The second-digit toggle is the harder test. Benford’s second-digit shares run from 12.0% for a 0 down to 8.5% for a 9, so there is less room to look wrong. The salaries still fail it at 0.0206. The combined dataset sizes pass at 0.0071, acceptable.

What it is actually good for

Benford is a screen, not a proof. A nonconforming set is a reason to look, never a finding. The real audit use is on ledgers: an accounts-payable file with tens of thousands of payments from cents to millions, where invoices bunched just under an approval limit show up as a surplus in the first two digits. Nigrini’s own cases are mostly of this kind. The first-two-digit test catches round-number and just-under-threshold habits that a first-digit test smooths over.

The mirror image matters too. A conforming set proves nothing. Anyone who has read this post can fabricate a conforming ledger in ten lines of code.

What to do

Before running the test, check the three preconditions. If the numbers live in one decade, stop.
Use MAD with Nigrini’s bands, and quote n next to it. A MAD without n is a number without a meaning.
Refuse to grade small sets. Draw the standard-error whiskers and let them make the argument.
Treat nonconformity as a queue of records to inspect, and inspect them by hand.
Run the second-digit and first-two-digit tests on anything that passes, since that is where round-number habits hide.