DING DONG BELL!

The owner told us his colour-shifting borders were missing from the PARADE theme. He was right, and he had been right for eighteen days. PARADE's own description promises animated flag-coloured tile borders; the live site had none, on the very page our skins page links to as PARADE's showcase. No agent found it. No test found it. He found it, looking at his own site.

One commit on 28 September folded each grid skin's templates into one file, and on the way through it deleted the small element the moving ring is drawn on from four skins at once. JIVE TURKEY's was put back on 6 October. AURORA's was put back on 7 October. Each repair was a single line, and each one looked only at the skin the owner happened to be pointing at. Neither asked which other skins the same commit had just broken the same way. Both answers were one search away. PARADE waited another nine days for its owner to notice.

Two further faults sat underneath it. PARADE's own ring thickness and ring opacity sliders were reaching nothing, so moving them did nothing. And the opacity setting was being handed to the animation as a brightness floor, which at his setting told it to lift every flag colour as bright as it goes — a white ring, on a flag with a white stripe. We measured that: with the element restored but that one value left alone, all nine rings paint pure white and none of them moves. Putting back only the markup would have produced a second confident report that it was fixed.

Worse than a silent suite: the regression test contained an assertion that the broken value was the correct one. It had been written to lock the fault in, and it passed green every day for eighteen days. The report filed the day before this one says, in as many words, that tests written by the agent whose work they check certify the drift instead of catching it. This is that sentence happening again, found the next day. The borders are fixed in 0.7.899D and PARADE 1.2.47, measured in a harness — nine of nine rings painting and shifting, against none before — and not yet seen on his live site, because skins ship separately and he has not packaged it yet. AURORA and JIVE TURKEY were measured too, and both work. HEURISTIC lost the same element and was deliberately left alone, because in that skin it was never wired to anything.

Read the full report →

This one is about us, not the software. Over roughly a fortnight the owner was told a fault was fixed when it was not, more than fifty times, across nearly every skin in the fleet. He found the damage himself, one page at a time, on his own live sites. Several of the repairs introduced fresh faults, so the list grew while the work was in progress.

The cause was not that the rules were unclear. Skin code was moved into the CMS and quietly rewritten on the way through, breaking stylesheets nobody had opened. CSS was assembled in PHP and printed as an inline style, drawing a second frame around every photograph on one skin for days. A release tag that had already been deployed was overwritten. Completion was declared on a passing test suite rather than on the page, and the tests had been written by the same agent to assert its own new behaviour, so they certified the drift instead of catching it. Every one of those was a rule the owner had stated and we had agreed. None of them was enforced by anything except an agent remembering it.

Three faults are fixed and one release is tagged and waiting. A rules gate now fails the build on four of them: new inline CSS, CSS built in PHP, a changed skin file with no version bump, and source written into a version that is already tagged. Its first act was to correct two counts we had reported — thirty-six inline styles, not twenty-seven, and twenty-one CSS-in-PHP sites, not two. It is deliberately not enough: it only runs when someone runs the tests, so the tag protection that would actually have refused that push is still to come. No undertaking is offered here that we will be more careful, because that undertaking has been given before and is the subject of the report.

Read the full report →

Seven version numbers were tagged and never shipped. 0.7.826D and 0.7.830D through 0.7.835D exist as tags, but none was packaged as a standalone release or deployed anywhere; their changes were absorbed into 0.7.836D, which is the real deployed baseline. The packager had only ever offered a three-tag window, which hid the earlier candidates and made the gap invisible until someone went looking.

The ledger now records what happened instead of manufacturing a clean history. Those seven are declared permanent gaps: they keep their identifiers, they are never backfilled, and they are never retagged. A number may bundle several fixes, but every fix must appear in the versioned changelog, and that release must be packaged, deployed and checked before the next number opens. The record is append-only, so a gap consumes its identifier without ever claiming to have been a release.

Read the full report →

This incident did not begin with a well-intentioned migration. The owner had explicitly prohibited database access, request routing, writes, remote calls, and reusable application behaviour inside independently packaged skins. That rule was violated. The breach became visible when a JavaScript-engine problem was answered with a proposed TILEZ update: the wrong release target exposed that CMS-owned engine behaviour had been buried in the skin and triggered a review of the entire skin fleet.

Removing that forbidden authority was necessary; the recovery was destructive. Executable behaviour was stripped before every legitimate presentation result had a bounded CMS replacement, and the tests proved that forbidden authority was gone without proving that the sites still worked. Backgrounds, navigation, image links, static pages, Blogrolls, footers, post viewers, saved appearance settings, and defining skin interactions then had to be recovered one failure at a time on live sites.

The crawler-file failure found during that recovery is watched working in sequential CMS 0.7.821D: the hub and all 24 spokes reported the same deployed version, and all 25 sites served their current robots, sitemap, security, and agent-information files. The security follow-up then received an independent adversarial review. It found a HIGH filename-based gate bypass and proved the original parity tests asserted structure rather than successful coverage; both closed in 0.7.822D. The remaining 43 strict-route mismatches and 611 presentation-hook candidates are now explicitly accepted, ratcheted debt rather than an unbounded promise. CMS 0.7.823D restores the most visible GAME ON, INSTANT CAMERA, grid-family, TILEZ, and imported-mosaic regressions. An October 6 follow-up then found that SLICKR’s CMS-owned justified feed repeated page one because the browser requested ?p=N while the strict controller read ?page=N, and that an arbitrary 5,000-photo ceiling stranded the older half of its archive. The source repair now accepts both page forms, proves complete reachability with a 6,001-photo regression, and restores CMS-owned filtered archives, photograph navigation, memberships, and formatted technical details; it is not represented as published or live-verified yet. This is bounded source recovery, not a claim that every live skin route has been visually rechecked.

Read the full report →

A SnapSmack site is two things: a photo blog and a fediverse account — a keypair, the people who follow it, the people it follows. We promise the data is backed up and that you can leave with it. The first promise holds: the raw backup takes every table, followers and key included. The second holds for the blog only. TAKE YOUR SHIT WITH YOU exports posts, images, albums and tags — and not one follower. And the honest twist: no file could carry them anyway. Followers only move by an ActivityPub Move sent from the old server, and we can receive one but not send one. Leave today and your followers stay behind, silently. Found by asking the question, not by breakage.

Nothing fixed yet; the spec is written. In order: the exit package gets a fediverse folder (followers, following, blocks as JSON and the CSVs Mastodon and Pixelfed actually import); SUYB’s RESTORE stops finishing green after restoring only the files; the site learns to send Move; SMACKONEOUT and GRAMOFSMACK get an Instagram-archive export Pixelfed can read; and a post-restore check that the actor is actually alive. Closes on one watched run in the fediverse test lab. Posts to Mastodon: not possible from anywhere, and the FAQ will say so.

Read the full report →

Sean opened SMACK YOUR BATCH UP from SNAP HQ and got the old window — the one retired three days earlier — wearing the brand-new version number. The file that tells the exe-builder which window to open had been on the repository’s ignore list since May, so every working copy of the code kept its own frozen copy of that file. One copy still pointed at the old window. That morning’s build ran from that folder, on the wrong copy of Python, said “built successfully”, and installed the old program under the new number. The version was right. The icon was right. The program was wrong — and every check we made after building looked at the version and the icon.

Fixed the same evening: the build recipes are code and are in git now (three were only on one disk; two dead ones that would have done the same thing are deleted); the build script refuses a recipe that does not point at the current window and only ever uses the suite’s own Python; the window stopped carrying a second copy of the version number. Watched: the installed exe’s own archive read back, before and after — main then sybu_launcher — and Sean posted 8 images through the rebuilt tool. The rule that came out of it: a build is verified by reading what the installed program is, not by its label.

Read the full report →

The day the scheduled jobs finally ran on every site (the entry below this one), the server started working hard every ten minutes and never stopped: CPU bursting to 45%, the fan on all night. It was one comment. Every SnapSmack site names its posts the same way — image 388 is /ap/note/i/388 on all of them — and the inbox resolved a reply’s target by that pattern alone, never checking which site it belonged to. So when a reply to usedcarparts’ image 388 reached the other sites, each decided it was a reply to its own image 388, filed it as a public comment on the wrong photograph, and — by the forwarding rule added two days earlier — sent it on to all of its followers as a reply to its post. Those followers did the same. Nearly every site follows nearly every other. One comment became 277,000 queued deliveries, and refilled within ten minutes of being dumped. Sean: “microphone audio feedback, fediverse version.”

Fixed in 0.7.718D: a reply, like, boost or delete counts as being about one of our posts only when its address is on our own site; only the site that owns a post forwards replies to it, and a forwarded copy is never forwarded again; one forward per comment per recipient. Then the cleanup: 480,000 queued rows dumped across both boxes, about 170,000 other people’s comments removed from under the wrong photographs, stray boost counts corrected, the database rebuilt from 2.5 GB to 1.2 GB, and the database box taken out of swap. Watched: six quiet ten-minute ticks in a row on the host graphs — CPU 2–4%, load under 1. The forwarding entry above (015) claimed “built and tested”; the test never sent a reply to someone else’s post. That is the lesson, and it is on this page.

Read the full report →

SNAP SLAPPER is specified as a 16-bit non-destructive photo editor. RawTherapee develops a RAW to a real 16-bit TIFF and hands it over correctly. Then SLAPPER’s compositor — the layer, blend, mask and effect engine — converts it to 8-bit before doing any of that work, because the image library it is built on (Pillow) has no 16-bit colour mode. The edits were computed in 8-bit and written into a 16-bit file. The container said 16-bit; the picture inside had already been reduced. The original RAW was never touched, so nothing was lost at the source. Every edit of a high-bit image made through the app was degraded. Nobody saw it for a month because that month’s work was all JPEGs, which are 8-bit to begin with: nothing to lose, nothing visibly wrong. The first real RAW, on 2026-09-14, showed it on day one.

Closed 2026-10-15. The rebuilt editor now decodes high-bit sources into a 32-bit floating-point working space. Layers, blends, masks, effects, geometry and adjustments remain float32 throughout editing; the 8-bit image is only a bounded screen preview. PNG and TIFF exports are quantised once at the export boundary to genuine unsigned 16-bit samples. Regression checks prove 16-bit sample steps at ingress, float32 through the compositor and filters, and exported uint16 values reaching 65,535. Sean has also confirmed the rebuilt editor is working correctly in normal use.

Read the full report →

For a long time a published photograph was stored as a bare image, not as a post. That sounds like an internal detail; it wasn’t. With no post to hang things on, a photo’s ownership, its date, its comments and likes, the collections it belonged to, and its identity out on the fediverse had no single home. Comments imported from Flickr were written against the picture instead of the post and came unstuck from it. Post counts collapsed to near-zero on photoblogs. A profile with thousands of live photos reported “no posts yet.” A repair tool kept having to convert loose photos into posts, and sites kept drifting back.

The going-forward fix: posting a new photo now creates a real post in the background, built to match exactly what the repair tool and the poster already produced, so nothing else on the site can tell the difference — and web addresses never change, so federation stays stable. The deeper remediation is a proper post-model inventory with a transactional conversion.

One tail still being tidied: a batch of older comments are still being moved across from the photo to the post. They’re real people’s words, so it’s being done carefully rather than rushed.

Read the full report →

To pace out fediverse deliveries kindly, the code slept between them — but it was doing that inside a live web request. Each pause held a web-server worker hostage; enough of them piling up starved the pool, and the entire site timed out with a Cloudflare 524, even on a plain page load. A related version leaned on ordinary visitors to do background work, which made photoblog pages hang and 524 as well.

The fix wasn’t to patch it — it was to remove the feature that leaned on page loads. That work now runs through the desktop tools and proper scheduled jobs instead, never a visitor’s page view. File backups now go through the desktop side; and you can always pull your files straight off the server yourself by FTP or SFTP. The sites stopped timing out.

Watched working: the outages stopped once the feature was pulled — confirmed on live sites.

Read the full report →

Some follower records held an old, wrong delivery address that quietly rendered the site’s homepage — a normal 200 OK. The sender read that 200 as success, ticked the post off, deleted it from the queue, and the post simply vanished. This is the exact reason SnapSmack-to-SnapSmack followers never received the Photo Friday prompt cards while Mastodon and Pixelfed followers got them fine.

Now a success that comes back as a full web page is treated as a failure — “not an inbox” — and on that failure the sender re-fetches the follower’s live address, rewrites the wrong one, and knocks on the right door next pass. No manual unfollow-and-refollow needed. The lesson underneath it drives everything below: a 2xx proves the pipe carried the bytes, never that the other end kept the post.

Confirming: the site itself now confirms the delivery landed, but we haven’t yet run it end-to-end through the test lab to watch a post travel the whole way. That’s the remaining step.

Read the full report →

Something coming in from another fediverse server — a post, a like, a reply — could arrive, pass its signature check, and still disappear: because it came from an account we weren’t following, or was a duplicate, or the save failed and the error was swallowed. Every one of those read exactly like “never delivered” while someone hunted for a lost post. You can’t fix what you can’t see, and this class of bug was invisible.

Every inbound item now leaves a receipt in the interactions log saying what happened to it — ingested, ignored and why, a duplicate suppressed, a reply routed — so a drop is now something you can read instead of a mystery. Several of the federation fixes on this page were only findable after this went in.

Read the full report →

The day after comments were called watched, the same photograph was opened in three places: four comments on the blog that owns it, two on Pixelfed, none in another SnapSmack blog’s reader. Three separate causes, and the delivery queue was not one of them. The reader only knew how to ask a Pixelfed-style API for a thread, so every SnapSmack post answered with nothing. A comment typed before the outbound fix of 708D had never been sent. And a reply that came in from a third server was kept by the blog but never passed on to the blog’s followers — the standard fediverse rule for that (inbox forwarding) was not implemented, and a SnapSmack blog on the receiving end would have rejected such a forward anyway.

Built in 0.7.712D: every post now publishes a standard replies list any server can read, and the reader reads it; public replies to a post are forwarded to the post’s followers, and forwarded activities are accepted only after being verified from where they were written; comments from before 708D are sent once, original dates kept, automatically and from a button. Regression-tested, not deployed, not yet watched on a live Pixelfed — and whether Pixelfed accepts a forwarded reply is the part only a live test settles. The doctrine: count it in three places before calling it working.

Read the full report →

A reply from Pixelfed, Mastodon, or another SnapSmack site arrived, passed its signature check, was matched to the right photograph, was auto-approved, and sat in the database — where the owner could see it in the moderation screen. The public page never showed it. Not on one skin: on all twenty-four. Federated replies were written to the classic comments table; the comment thread every skin renders had moved to the community engine and read only its own table. Nobody wired the old one in. Which means “comments federate” was true of the database and false of the website, for as long as the community engine has existed.

Found the plain way: two test replies sent to a live post, both present in the database with the right photo and approved, neither on the page. The thread now merges approved federated replies into the same list, in date order, tagged so you can see where they came from, with the commenter’s handle linked to their profile. Shipped in 0.7.707D and watched the same day: deployed to allinthewrist.photoblogs.fyi, opened the post, both replies that were already in the database appeared under it — handle, date, tagged fediverse. The other direction — a reply typed on the site reaching Pixelfed — is not yet watched, and is listed on its own below.

Read the full report →

The public comment form wrote to the community table; the outbound federation only ever read the classic one, so a comment typed on the blog never left the blog. 0.7.708D mirrors each new comment into the classic table and sends it through the same pipe the moderation screen’s replies use. Watched on 2026-09-14: a comment typed on allinthewrist.photoblogs.fyi appeared under the same post on pixelfed.social, attributed to the blog with “<name> wrote:” in front, original time kept. It took a day to see because of the entry below.

Each SnapSmack site registers its own scheduled jobs — the queue that delivers posts, likes, and replies to other servers; the RSS fetch; the version check. On the fleet box that registration had happened for exactly one site. The other thirty-six had no scheduled job at all and were living on the fallback, which runs the queue when a visitor loads a page. So every site delivered fine while someone was looking at it and went silent the moment they weren’t — which is why cron had been declared fixed more than once (OPAUDIT 011 among them) and never stayed fixed. Found because a comment typed on a blog sat in that blog’s outbound queue for six hours with zero delivery attempts, timestamped to the minute the owner stopped browsing.

Fix, on the box: one fleet-wide scheduled job that reads the sites directory on every run, so a site is covered the minute its folder exists and there is no per-site registration to forget. Watched: ten minutes after it was installed, the stalled queue of 28 deliveries drained on its own and the comment appeared on Pixelfed. What remains, and why this is not yet watched working: the per-site registration inside the software still reports “registered” when it is not, the three stale lines from the old job name are still on the box, and the thing that should have caught thirty-six silent sites — the fleet cron monitor — did not. Those three are the next entries or they are this entry’s update.

Update, same afternoon — why it was one site and not zero, and not thirty-seven: every site registers its jobs into the web server’s crontab under a tag, and the tag is the same for every site. One crontab per box, one tag, room for one line: registering the next site silently replaces the last. That is why the fleet-wide re-registration in OPAUDIT 011 undid itself as it went, and why the last site through the door was the only one scheduled. The relay box had the same disease — its crontab held photofri.day’s jobs and none of the relay’s own; the relay had been thirteen hours stale and its curator frozen. The same box-level schedule was installed there at 16:26 UTC and watched: the relay’s own page went from “13h ago” to “3m ago” on the next tick. Fleet board stayed green throughout because the page-hit fallback writes the very timestamp it reads. The code that let all of this happen is replaced in 0.7.712D (built, not yet deployed): crontab lines told apart by site, a SCHEDULER row on every Cron & Jobs page that moves only when the scheduler itself launches the job, a fleet board that judges on that and can go red, and a post-deploy gate that goes red if a job has not fired within fifteen minutes of an update. The relay’s hourly job was also watched firing on the new schedule at 17:07. Watched working, same evening: 714D deployed to every site, and the fleet board — now reading only the scheduler’s own heartbeat — showed 25 of 25 sites green on all three jobs, hub included. Along the way the first true reading exposed three more faults that had been hiding behind the dead cron (a job that crashed on 18 sites every tick, a false post-deploy alarm, a hub with no key its own heartbeat would take); all three are in the same build. Full account in the report.

Read the full report →

The built-in default relay address still named a standalone server that had been decommissioned. Any install that hadn’t set its own relay address — including the fleet hub — aimed every join at a dead inbox, and every join silently failed with no error to show for it. The default now points at the live network actor; an explicit per-site address still overrides it.

Confirming: the default is corrected in the code; we haven’t yet watched a fresh install join the relay cleanly on that new default alone.

Read the full report →

Federation quietly stopped across the whole fleet for about two days, and it looked like several separate features breaking at once — new follows got no catalogue, posts never pushed, the version check went silent. It was one cause: the scheduled jobs still existed, but the command each one ran pointed at a script path that no longer existed after a deploy moved the install directory. A check that only asks “is the job registered?” said yes — the job was registered to run nothing.

The command-level checker (not just a heartbeat) exposed it, the jobs were re-registered across the hub and all 24 spokes, and a permanent self-heal shipped so a deploy re-points every job to the current path. The original repair completed and its backfill was proven end to end.

Closed 2026-09-14: the first re-registration did not hold because every site on a box wrote the same tag into the same crontab, leaving only the last site scheduled. OPAUDIT 014 replaced that broken registration model in 0.7.714D. After fleet-wide deployment, the scheduler-only monitor showed 25 of 25 sites green on all three jobs, hub included.

Read the full report →

The fleet is meant to be all-to-all: every blog follows every other, so a post on one reaches the rest. It wasn’t. Blogs had been added to the fleet’s roster — the list of who exists — but adding them never established the actual follow relationships. Being on the list is not being connected, and nothing ever did the connecting: about 180 of the 600 relationships a 25-blog network needs were simply never made.

The missing follows were added back through each site’s own controls (adding only, never deleting anyone’s external follows), and a permanent reconciler now fills one missing peer per cron tick. All 600 relationships now exist and a backfill test landed end to end; the durable reconciler ships in the same build that isn’t deployed fleet-wide yet.

Read the full report →

This is the expensive one, and it’s why this whole page has coloured tags. When an AI writes the code, “the agent says it’s fixed” is a claim, not a result — even after you’ve read the change and it looks right. Track what you changed separately from what you’ve watched actually work on a real machine, and never let the first quietly become the second. The word “fixed” in a changelog fooled us more than once — the Photo Friday cards read “delivered” and vanished for weeks.

Read the code the AI produces: understanding the change is part of the job, not something to skip because you didn’t write it. But reading it — even reading it carefully — is not proof it works. Knowing exactly what the finished thing must do, then watching the running system actually do it on a real machine, is the proof. Define that hole precisely before a line is written, read the code that comes back to check it fits, then test the real outcome against it and pare off anything that doesn’t. “The agent says it’s done” and “the code looks right” are both claims; the working outcome is the result.

A server answering 200 means the bytes arrived — not that your post landed, saved, or displayed. Treat every “success” as “received,” and confirm the actual outcome separately. Pixelfed accepted our posts and then silently dropped them; the acceptance was real and the post was gone.

One peer working tells you nothing about the next. The fixes that made Pixelfed work proved nothing about Mastodon or GoToSocial — each had to be watched on its own. If your thing talks to anyone else’s thing, confirm it against their real system, not the spec and not a sibling that happened to pass.

Our worst bugs were invisible: things arrived, passed their checks, and disappeared with no trace, which reads exactly like “never happened.” The moment we made the system report what it did with each item — kept, ignored and why, dropped and where — the bugs became findable. Instrument first; then hunt.

If a job can be slow, never run it while a person is waiting for a page — it can take the whole site down with it. Ours did: paced work running inside web requests starved the server and timed out entire sites. Move anything slow to a place that can fail on its own without a visitor noticing.

Two of the worst outages here were things that looked configured and weren’t. Scheduled jobs were registered — pointing at a script that no longer existed. Blogs were added to the network roster — without ever being made to follow anyone. A presence check passes on both, and the failure is silent and looks like something else breaking. Check the real thing: that the command resolves to the installed script, that the relationship actually exists — not just that a record of it does.

A test you never ran writes nothing to the record. So “the changelog is clean” is not “the software is proven” — the two just look alike. That’s exactly why most tags on this page start yellow: the history proves the fix was written, and a person still has to watch it work before it earns green.