The horizon that lost the west

shade-engine answers “is this spot in the shade right now” by comparing the sun’s position against a precomputed raster: for every square meter of the city, the elevation angle that blocks the sky in each of 64 compass directions. One offline build, millisecond queries. The case study explains the happy path; this note is about the week the artifact lied.

Wrong in one direction only

The symptom had a shape. Mornings looked right: long shadows pointing west, streets on the correct side of the canyon shaded. Evenings were absurd: a June day at eight in the evening, sun 18 degrees above the western horizon, and the shade overlay almost empty - a whole city reportedly standing in the sun at golden hour.

Nothing crashed. No exception, no 500, no log line. That is the failure mode of precomputed artifacts: the API reads a number from a raster, compares it with the sun’s elevation and returns a confident boolean. The number just happened to be wrong - and only for some numbers.

Twenty bands of zeros

The horizon lives in a 64-band raster, one band per compass sector of 5.6 degrees, angles quantized to 8 bits. A per-band histogram takes seconds and screamed: bands 45 to 64 - every sector from west through north-northwest - were identically zero. Toward the west, as far as the engine knew, no building in the city was tall enough to block anything. The sun sets in the west; every evening query compared its elevation against zero and won.

That is also why the symptom clustered so cleanly. The invariant behind the whole engine is one sentence - a pixel is shaded exactly when the sun sits below that pixel’s horizon toward its azimuth - so when a slice of the horizon goes missing, the wrong answers trace the missing azimuths, and nothing else degrades. Impossible geometry is easy to spot once you know which direction to look: a north-south street canyon with the sun low in the west simply cannot be fully sunny.

Rebuilding without drama

The fix was not a patch; a corrupt artifact only has one cure. The horizon sweep re-ran overnight - twelve hours and forty minutes of ray-marching 56 million pixels times 64 sectors on a small VPS - into a staging directory, while production kept serving the broken-but-stable version. The build verifies itself at the end, and an independent verify pass re-checked the artifacts before anything moved.

The swap itself was two renames with the API stopped: about fifteen seconds of downtime for a thirteen-hour rebuild, with the old artifact kept next door as an instant rollback. Then the 83 precomputed tile overlays were regenerated against the healthy horizon and the live checks finally agreed with the sky: eight in the evening, sun at azimuth 286, and the streets of the demo city in the shade where geometry says they must be.

What stays

• Artifacts fail silently. Code throws; rasters just return values. Verification has to live next to the build and next to the swap - not in the moment someone squints at a sunset.
• State the invariant. “Shaded iff sun below the sector horizon” is cheap to write down and cheap to check, and its violations cluster in ways that point at the cause (only west, only evenings).
• Stage, then swap. A rebuild that writes into the serving directory turns hours of build time into hours of inconsistency. Staging plus an atomic rename turned it into fifteen seconds.
• Trust nothing twice. The sweep’s exact mode is validated bit-for-bit against a brute-force oracle in the test suite, and every build ends in verification. The original zeros were never reproduced; the guardrails that would catch them again are permanent.