Small data is still most data

Most of the data people actually work with is small. Most of the tooling people reach for is sized for the rare dataset that is not. I wanted a real distribution instead of a hunch, so I pulled the on-disk size of about three thousand public datasets from three catalogues and put them on one log axis.

Hugging Face, 1000 most recently updated, size on disk

1,000 datasets with a reported size, fetched 2026-09-05. Log scale.

Threshold 1 GBDrag on the chart or use the slider. Hover a bar for its count.

Median size

224 MB

Fits in 16 GB of RAM

78%

1 TB or more

A thousand datasets on a log axis

The 1,000 datasets most recently updated on the Hugging Face Hub, by bytes on disk. Each bar is a factor of ten wider than the one before it, or about three on desktop. The pile sits in the middle, not the right.

Draw a line at 1 GB

Everything left of the line fits in a browser tab, a pandas frame, or a single Parquet file you can email. That is most of the sample, and this sample is ML flavoured.

Now 100 MB

Still a big share. A 100 MB CSV opens in a spreadsheet, badly, and in DuckDB in under a second. Nobody needs a cluster for it.

The laptop line and the cluster line

Green is 16 GB, what a mid-range laptop holds in memory. Orange is 1 TB, where a single machine stops being the comfortable answer. The interesting part is how little sits past orange.

Your turn

Drag the threshold, hover a bar, and switch samples. The most-downloaded Hub datasets skew large. Zenodo research data skews small. Neither is your data, but yours is probably closer to the small one.

What the histogram says

224 MB

median size, recent Hub datasets

1,000 most recently updated, Hugging Face

61%

under 1 GB

same sample, 61.2% exactly

at 1 TB or more

32 of 1,000 datasets

The default sample is the 1,000 datasets most recently updated on the Hugging Face Hub, which is about as large-data-friendly a source as exists. It is a machine learning catalogue. People upload training corpora, audio, video, and image sets there. Even so, the median is 224 MB, 45.8% are under 100 MB, and 77.8% fit inside 16 GB of RAM. Only 3.2% reach 1 TB.

Switch to the 981 most-downloaded Hub datasets with a reported size and the picture flips: median 57.6 GB, only 22.5% under 1 GB, 19.2% at 1 TB or more. That is the sample everyone pictures when they say “data”. It is also a popularity contest won by web crawls and multimodal corpora, not a picture of ordinary work.

Switch to Zenodo, where researchers deposit the data behind papers, and the median drops to 2.66 MB. 85.2% are under 1 GB. 96.5% fit in 16 GB. Nothing in the 1,000 most recent records reaches 1 TB.

The point is the shape, not the number

I do not care whether the true median for your team is 20 MB or 2 GB. I care that on two of the three samples the mass sits well left of the laptop line, and that even on the most-downloaded list, the one built to favour big data, 80.8% of datasets sit left of the cluster line. The tail is real. It is also rare, and most tooling decisions get made as if the tail were the body.

My rule: a single machine with DuckDB, Polars, or plain SQLite handles everything left of the 16 GB line without ceremony, and most things between 16 GB and 1 TB if you use Parquet and stop loading the whole thing into a dataframe. DuckDB spills to disk. A 200 GB Parquet directory on a laptop is a slow query, not an outage. You need a cluster when the data will not fit on a disk you own, or when hundreds of people query it at once. That is the orange line, and on the recent Hub sample 3% of datasets are past it.

Why teams reach for big tooling anyway

Nobody buys a warehouse because of the histogram. They buy it for the things a laptop does not give you, and those reasons are legitimate:

Sharing. A file on my machine is not a dataset. A table with a name that three teams can query is. The warehouse is a rendezvous point first and a compute engine second.
Governance. Access control, audit logs, PII masking, a place an auditor can point at. A CSV in a Slack thread fails all of these.
Scheduling. Something has to run the job at 6am when I am asleep, retry it, and page someone when it fails. Cron on a laptop is a hobby, not a pipeline.
Memory of the org. The query somebody wrote in 2023 still runs because the table still exists. Laptops get replaced.

Those are coordination problems. They get solved by a shared catalogue, a scheduler, and a permission model. They do not require distributed compute, and yet distributed compute is what the invoice is for. Most teams pay cluster prices for a filing cabinet.

How to size honestly

Measure it. Sum the bytes of every table you actually query monthly. Put the number on a log axis next to 16 GB and 1 TB. Most teams have never done this and are surprised.
Separate storage from compute in your head. Object storage plus Parquet is cheap and shared. The expensive question is what reads it, and for most of the histogram the answer can be one process.
Buy the coordination layer, not the compute layer. A catalogue, a scheduler, and a permission model are the parts that earn their cost. Pick a query engine that runs on one box first and a cluster only when a specific job proves it needs one.
Keep a “biggest table” watch. The tail grows. When one table crosses the line, move that table, not the whole stack.
Say the size out loud in design reviews. “This is 4 GB” ends a lot of architecture arguments in about a second.

Small data did not go away when the big-data decade arrived. It just stopped being the thing we design for, and the histogram says that was a mistake.