Engineering
How a city goes live
A hand-checked bounding box, one bounded OpenStreetMap query, a nightly run, a publish step. The arithmetic of scaling that to the whole world, in public.
Coverage is the number everyone asks about first, so it is worth explaining what it costs to move it.
Phoo went from two cities to 122 in the catalog in a fortnight, of which 22 are live in the app. Nothing about that was a scraping breakthrough. It was a scheduler.
The thing that is forbidden
There is exactly one query that would make world coverage trivial, and it is the one nobody is allowed to run: a single global Overpass request for amenity=toilets. Public OpenStreetMap infrastructure is donated capacity. Its fair-use guidance is around 10,000 queries and a gigabyte a day, with two concurrent requests, and it explicitly is not a bulk collection mechanism.
So the rule in this codebase is that a query is always bounded to one reviewed city extent, and there is no code path that can express anything else.
The counterintuitive part is that this is not the constraint people assume:
| Scale | Cadence | Queries per day | Share of fair use |
|---|---|---|---|
| 300 cities | weekly | ~43 | ~0.4% |
| 1,000 cities | fortnightly | ~71 | ~0.7% |
| 1,000 cities | weekly | ~143 | ~1.4% |
One bounded query per city per refresh is nothing. The bottleneck was never the API budget. It was that the importer could only be pointed at one city at a time, by hand, via an environment variable.
What replaced the environment variable
The catalog now lives in the database, in two tables that do different jobs.
public.cities is the app-facing catalog: code, display name, country, time zone, center point and the reviewed import extent. The extent matters more than it sounds, because the app resolves “which city am I in” by checking which extent contains the map center. If the extent were the full metropolitan sprawl while the data covered the middle, the app would confidently resolve you into a city that is empty where you are standing. So the box the importer queries and the box the app resolves against are the same box, on purpose.
Alongside it sits a private table of import areas holding everything the importer needs and the app has no business seeing: a guardrail bounding box that any override has to stay inside, the local name tags to prefer, the fallback name for unnamed facilities, the currency to assume for fees, an element cap, and the scheduling state.
Adding a city is now an insert and a review. No deploy.
The nightly queue
One timer, once a night. Each run:
- Asks Postgres for the most stale areas that are due, up to a limit that defaults to 25.
- Processes them one at a time, with a jittered pause of about 45 seconds between cities. Politeness by construction rather than by good intentions.
- Records the outcome per area.
An area is due when it has never been imported, when its interval has elapsed, or when its failure backoff has elapsed. Claiming an area stamps the attempt inside the same statement, using for update skip locked, so two overlapping runs cannot double-import a city. Dry runs read the queue without claiming anything.
A failure does not block the batch. The failed city retries with a linear backoff — one day per consecutive failure, capped at its normal interval — instead of hammering the same endpoint every night forever. Public Overpass endpoints go down; this is a fact rather than a surprise, which is why the importer takes an ordered list of endpoints and fails over rather than a single URL.
At the defaults, one timer sustains a catalog of about 175 cities. Raising the per-night limit takes it to 300 and still uses under two per cent of the published fair-use budget.
What a city goes through
Review. Somebody reads the area’s configuration before it is committed: is the extent the tourist reality or the administrative fiction? Twenty-one of the hundred cities in the second batch needed an override, because municipal boundaries lie in specific ways. Tokyo’s boundary includes islands a thousand kilometres out. Las Vegas’s excludes the Strip. Miami’s excludes Miami Beach. Manila is one city inside a metropolis that behaves as one place.
Dry run. The importer fetches and normalizes without writing, and somebody reads the output: are names coming through in the local script, are fees parsing into the right currency, is the element count plausible for a city that size.
Import. Bounded query, normalize, stage, deduplicate against existing records, preserve source history.
Publish. The city lists itself in the app on its first successful import. There is no per-record editorial pass, and we decided that in the open: at thousands of records per city it does not scale, and pretending otherwise would just mean a slower version of the same data with a nicer story attached. Correctness converges through the community loop instead.
Refresh. The city re-enters the queue and gets picked up again when it is stale.
The part that is designed but not built
Overpass gets Phoo to a few hundred cities comfortably. It does not get us to the world, and the world is the goal.
The plan for that is bulk extracts: Geofabrik country and region files, filtered with osmium tags-filter amenity=toilets, which is roughly 500,000 elements worldwide and zero API usage. Overpass then demotes to what it is genuinely good at — surgical single-city refreshes.
Two things stand between here and there. Staging is currently one call per record, which is fine for a city and absurd for a planet, so it needs batching. And the data partition is currently the city code, which stops making sense once coverage is continuous; that becomes spatial tiles, with the city catalog kept as a user-facing convenience rather than a storage key.
There is also a mundane constraint that shapes the whole design: the box this runs on has 40 GB of disk, and Europe alone is a 30 GB extract. So the pipeline processes country files sequentially — download, filter, delete, next — with a peak footprint under 7 GB, then settles into a persistent filtered world file of about 300 MB kept current with tag-filtered daily diffs and a monthly re-baseline to catch objects that quietly stopped being toilets.
None of this is glamorous. It is, however, the entire difference between a map of 22 cities and a map of everywhere, and it is a scheduler and a disk budget rather than a moonshot.