The UK's open government data is scattered across dozens of portals with inconsistent, often broken metadata — effectively invisible to both people and AI tools. This project harvests the metadata of every dataset from national, devolved, regional and NHS portals into one normalised, searchable index. We never copy the data itself: every result links to the publisher's own page.
When we harvested the full catalogue, the metadata quality problem was worse than expected:
We've followed the links on 97% of the catalogue. Of those, 36% give you machine-readable data — 12% a direct file, 24% an API you can query. Another 30% just lead to a webpage, where the hunt continues. For 14% the publisher lists no files at all, so there is nothing for us to follow; 10% are genuinely broken, and 10% we couldn't check because the publisher blocked us.
We normalise what can be normalised — hundreds of spellings of file formats and dozens of licence variants collapse to a consistent vocabulary — then collapse duplicates, flag retired records, and stay honest when a search has no good answer, including when we simply hold no data for the place you asked about.
We try hard not to make publishers look worse than they are. Of the "stale" datasets that declare an update schedule, 70% say they were never going to be updated again — a finished one-off publication isn't neglected, so we record the declared cadence rather than treating every quiet dataset as rot. The same care applies to licences (many are recorded in a non-standard field that naive harvesting misses, which is why coverage is far better than it first appears) and to links (a publisher blocking our checker is reported as "not verified", never as "dead").
Collapsing duplicates gets the same treatment. Councils publish dozens of identically-titled datasets — "Council Spending", "Business Rates" — and Rochdale's is not Leeds's. We only merge two records when we can confirm they come from the same organisation, so a duplicate we fail to spot costs you one extra line of results, rather than a wrong merge costing you the dataset entirely.
Those figures cover the whole harvested catalogue, including a long tail of obscure records almost nobody reaches. Among the 60,000-odd datasets you can actually find through search — excluding collapsed duplicates and retired records — licensing is better (29% state none) though the link quality is much the same. We quote the whole-catalogue numbers because they describe the real state of UK open data, not the flattering subset.
All figures are measured from the index and shift as we add sources.
Most catalogue entries don't lead to a data file — they lead to another webpage. So we follow every link and report what's actually behind it:
That last category matters. Checks run from a server, and a fair number of sites block automated requests on sight — so we deliberately never report "dead" unless we actually got a broken response. We also space out requests to any single publisher so we're never the reason a site struggles.
When a search matches poorly we say so, rather than quietly serving the nearest thing. But the reverse doesn't hold: a search that matches well means we understood your words and found datasets on that subject — not that your question has an answer. If nobody publishes the data you want, we can only show you the closest subject, and we'd rather you knew that than assumed the top result settles it.
| Portal | Type | Datasets |
|---|
Know a portal we're missing? The source registry is a single YAML file, so suggesting one is a small pull request — portals running CKAN, DCAT or OpenDataSoft are usually a five-line addition, and CI checks the endpoint before a human reviews it. The code is open source: github.com/dominicm2023/open-data-uk.
Need more than the rate limit allows, or want something the API doesn't do yet? Please don't scrape — the whole index rebuilds from open source in about an hour, or open an issue and we'll sort something out.
Everything the search box does is available as an open JSON API —
see the interactive API documentation. Please keep to
the rate limit (30 searches/minute) and retain the attribution field.
GET /api/search?q=… · GET /api/stats ·
GET /api/sources
We log the search text you type, so we can see which searches come back poor and fix them. We deliberately do not record your IP address, your browser, or any cookie or session identifier — there is nothing stored that links one search to another, or to you. There are no analytics or advertising trackers on this site.
The indexed metadata describes datasets published under the Open Government Licence and other licences as stated per dataset — check each dataset's licence before reuse. This index and its code will be released as open source.