ajustino.dev
<- Back to notes

The API was broken, so I traced the map that worked

giswfsopen-datadebugging

geo-hazard ingests three public feeds about Spain: AEMET weather warnings, IGN earthquakes and Copernicus EFFIS wildfires. The first two went live in a day. EFFIS took three, and not because the code was hard - because for two of those days I was asking a broken server the right question, at the wrong address.

A 200 that lies, a layer that errors

The EFFIS documentation points at standard OGC services. Reality, captured the hard way: the server hangs on HTTP/2, WFS 2.0.0 returns 502, 1.1.0 hangs, and only WFS 1.0.0 answers. And when I finally asked it for the layer I wanted - effis.nrt.ba.poly, near-real-time burnt areas - it answered instantly, with HTTP 200, containing msPostGISLayerGetItems(): Query error. The hotspot layers just timed out. Same result the next day.

So I paused the wildfire job in production - a one-row UPDATE setting its next_run_at to infinity - and the API served zero wildfire rows. Honest emptiness beats invented data.

Trace the product, not the docs

Then someone (me, showing the project) opened the official EFFIS viewer: fires everywhere, burnt areas from three satellite sources, updating happily. The data existed and was being served. I was just not asking where the viewer asks.

A headless browser with network capture answered it in minutes. The viewer renders fires as raster tiles (WMTS) - which means seeing fires on a map proves nothing about any vector API. Its backing API turned out to be a tiny FastAPI with four endpoints, none of them data. But the tile layer names - nrt.ba.today, s3.hs.today - were the clue. Same server, same MapServer, different mapfile: the viewer feeds from /gwis, the documentation points at /effis. On /gwis, the full family exists and answers: nrt.ba.poly.{today,week,month,season} with fresh polygons, all.hs.* with hotspots from this morning.

The quirks are the contract

What “working” means in twenty-year-old OGC land deserved its own ADR:

  • The TIME parameter is silently ignored. The rolling window is encoded in the layer name - .today, .week, .season are different layers.
  • The GeoJSON output serializes coordinates as [lat, lon] - inverted - while the bbox request parameter wants normal lon,lat order. The axis swap lives in exactly one place in my parser, next to a comment explaining why.
  • Backend failures come back as HTTP 200 with an XML ServiceExceptionReport. Classify by body, not by status code: that one is transient (retry next poll), a 404 is a contract change (page a human).
  • Attribute-rich archive layers exist (viirs.hs.query, with radiative power and confidence) but span the full history since 2019, and any date filter times out. Unusable for sync; noted and walked away.

I captured real payloads as committed fixtures, wrote the parser against those, and the driver went to production the same afternoon.

The first real payload is a test you have not run

The first live sync fetched 7,970 hotspots and 92 burnt-area polygons, and promptly crashed: the number of query arguments cannot exceed 32767. My upsert sent the whole batch as one statement - about 8,000 rows times 9 columns is 72,000 bound parameters, and asyncpg caps a statement at 32,767. The other two sources never came close (230 warnings, 14 quakes), so 174 tests and a fake driver had never exercised the limit. The fix is boring: chunk the batch inside one transaction. The containment worked as designed - failed job, backoff, zero corruption - and the regression test now inserts 4,001 rows because that is the smallest honest number that would have failed.

Three takeaways. When the documented endpoint is broken but the product on top of it works, trace the product. In legacy OGC services the real contract is folklore - capture live payloads as fixtures and write parsers against those, not against the spec. And fake drivers get you to production, but the first real payload is still a test, whether you wrote it or not.