DING DONG BELL! — SnapSmack Operability Audits

BUZZERS! is where we show our work on security — whether the software can be abused. This is the other question, the one that’s actually bitten us far more often: does a thing do its job, end to end, watched on a real machine. Almost nothing that hurt this project in the last several months was a break-in. It was plumbing that looked done and wasn’t — a delivery that said “sent” and vanished, a fix written but never watched, a feature declared ready before it was built.

This page is different from BUZZERS! in one important way: it never closes. A security audit gets fixed, verified, and filed. This is a living record. Entries get more accurate over time; they’re never quietly rewritten. And to be clear about the tone: a lot of what’s below is “fixed, and we’re confirming it” — not “broken.” We separate what we’ve genuinely watched work from what we’ve merely fixed, because those are two different claims and running them together is what kept biting us. Saying “we’re still confirming this” out loud isn’t an admission that it’s broken; it’s the honest version of done.

Why publish where we tripped? Because “fixed” in a changelog is the same word that’s fooled us before, and the only cure is to say out loud what’s been watched working versus what we’ve merely been told is done. If you’re running SnapSmack, or building something like it, this is the honest version of the story — not a highlight reel. The one thing you won’t find here is a live, unfixed security hole; those stay in BUZZERS! until they’re closed, then they’re fair game to talk about.

“Ding Dong Bell”? Cockney rhyming slang for hell. This is the page that records every operational hell SnapSmack went through — the stuff we tripped over on the way to something that holds when you lean on it.

How to read the labels

Blew up / Open: It’s broken, or the job it’s meant to do can’t be finished right now. A known problem, stated plainly.
Fixed — confirming: We changed something to fix it, and confirming the real outcome on a real machine is the step still in progress. This isn’t “broken” — it’s “fixed, and we’re making sure.” It stays here until someone actually watches it work, because “fixed” and “watched working” aren’t the same claim.
Watched working: Someone watched the real job succeed on a real machine and named what they saw. Trusted — for that version and that path only.
Eyeball next: This incident pointed at a nearby path worth checking next. A lead we’re following, labelled as a lead — not a known problem.
Confirming: Something we expect to work but haven’t yet stood in front of and watched. Not broken — on the list to confirm, and said out loud rather than assumed done.

Where our own plumbing fell down

Failures inside SnapSmack itself — the content model, background jobs, delivery, and the machinery that’s supposed to keep a site running without anyone watching it.

The site had picture containers but no post containers

the deep one Watched working

For a long time a published photograph was stored as a bare image, not as a post. That sounds like an internal detail; it wasn’t. With no post to hang things on, a photo’s ownership, its date, its comments and likes, the collections it belonged to, and its identity out on the fediverse had no single home. Comments imported from Flickr were written against the picture instead of the post and came unstuck from it. Post counts collapsed to near-zero on photoblogs. A profile with thousands of live photos reported “no posts yet.” A repair tool kept having to convert loose photos into posts, and sites kept drifting back.

The going-forward fix: posting a new photo now creates a real post in the background, built to match exactly what the repair tool and the poster already produced, so nothing else on the site can tell the difference — and web addresses never change, so federation stays stable. The deeper remediation is a proper post-model inventory with a transactional conversion.

One tail still being tidied: a batch of older comments are still being moved across from the photo to the post. They’re real people’s words, so it’s being done carefully rather than rushed.

Read the full report →

Background jobs ran inside web requests and took whole sites down

the 524 outages Watched working

To pace out fediverse deliveries kindly, the code slept between them — but it was doing that inside a live web request. Each pause held a web-server worker hostage; enough of them piling up starved the pool, and the entire site timed out with a Cloudflare 524, even on a plain page load. A related version leaned on ordinary visitors to do background work, which made photoblog pages hang and 524 as well.

The fix wasn’t to patch it — it was to remove the feature that leaned on page loads. That work now runs through the desktop tools and proper scheduled jobs instead, never a visitor’s page view. File backups now go through the desktop side; and you can always pull your files straight off the server yourself by FTP or SFTP. The sites stopped timing out.

Watched working: the outages stopped once the feature was pulled — confirmed on live sites.

Read the full report →

Deliveries that landed on a web page counted as “delivered”

“Wrong Door” Fixed — confirming

Some follower records held an old, wrong delivery address that quietly rendered the site’s homepage — a normal 200 OK. The sender read that 200 as success, ticked the post off, deleted it from the queue, and the post simply vanished. This is the exact reason SnapSmack-to-SnapSmack followers never received the Photo Friday prompt cards while Mastodon and Pixelfed followers got them fine.

Now a success that comes back as a full web page is treated as a failure — “not an inbox” — and on that failure the sender re-fetches the follower’s live address, rewrites the wrong one, and knocks on the right door next pass. No manual unfollow-and-refollow needed. The lesson underneath it drives everything below: a 2xx proves the pipe carried the bytes, never that the other end kept the post.

Confirming: the site itself now confirms the delivery landed, but we haven’t yet run it end-to-end through the test lab to watch a post travel the whole way. That’s the remaining step.

Read the full report →

Incoming fediverse activity arrived, passed checks, then vanished

“Signed Receipt” Fixed — confirming

Something coming in from another fediverse server — a post, a like, a reply — could arrive, pass its signature check, and still disappear: because it came from an account we weren’t following, or was a duplicate, or the save failed and the error was swallowed. Every one of those read exactly like “never delivered” while someone hunted for a lost post. You can’t fix what you can’t see, and this class of bug was invisible.

Every inbound item now leaves a receipt in the interactions log saying what happened to it — ingested, ignored and why, a duplicate suppressed, a reply routed — so a drop is now something you can read instead of a mystery. Several of the federation fixes on this page were only findable after this went in.

Read the full report →

The default network relay pointed at a machine that no longer existed

retired box Fixed — confirming

The built-in default relay address still named a standalone server that had been decommissioned. Any install that hadn’t set its own relay address — including the fleet hub — aimed every join at a dead inbox, and every join silently failed with no error to show for it. The default now points at the live network actor; an explicit per-site address still overrides it.

Confirming: the default is corrected in the code; we haven’t yet watched a fresh install join the relay cleanly on that new default alone.

Read the full report →

The scheduled jobs looked registered while pointing at yesterday’s door

the cron drift Fixed — confirming

Federation quietly stopped across the whole fleet for about two days, and it looked like several separate features breaking at once — new follows got no catalogue, posts never pushed, the version check went silent. It was one cause: the scheduled jobs still existed, but the command each one ran pointed at a script path that no longer existed after a deploy moved the install directory. A check that only asks “is the job registered?” said yes — the job was registered to run nothing.

The command-level checker (not just a heartbeat) exposed it, the jobs were re-registered across the hub and all 24 spokes, and a permanent self-heal shipped so a deploy re-points every job to the current path. The repair was watched working — a hub run completed and a backfill test landed end to end — but the durable fix isn’t deployed fleet-wide yet.

Read the full report →

Adding a blog to the fleet never actually made it follow the others

discovery ≠ connection Fixed — confirming

The fleet is meant to be all-to-all: every blog follows every other, so a post on one reaches the rest. It wasn’t. Blogs had been added to the fleet’s roster — the list of who exists — but adding them never established the actual follow relationships. Being on the list is not being connected, and nothing ever did the connecting: about 180 of the 600 relationships a 25-blog network needs were simply never made.

The missing follows were added back through each site’s own controls (adding only, never deleting anyone’s external follows), and a permanent reconciler now fills one missing peer per cron tick. All 600 relationships now exist and a backfill test landed end to end; the durable reconciler ships in the same build that isn’t deployed fleet-wide yet.

Read the full report →

Where talking to other software fell down

The fediverse is a room full of different implementations, and every assumption about how a peer reads our output was wrong until proven. One peer working proves nothing about the next. So we stopped guessing from the outside: we now run our own instances of the major fediverse software — Pixelfed, Mastodon, GoToSocial — alongside our own SnapSmack fleet, and test against them. Controlling both ends means we get all the data, the raw logs from every server, and see each failure the moment it happens — instead of inferring it from a post that never showed up.

Pixelfed accepted our posts, then silently dropped them

the teacher Fixed — confirming

The saga that taught us the whole doctrine. First, every Follow from Pixelfed was rejected for two weeks — Pixelfed builds its signature check from the path only and dropped the query string our inbox address carried, so the signatures never matched. Then, once that was fixed, deliveries were accepted and no post ever appeared: Pixelfed re-encodes an & when it fetches an object back, so our object address arrived mangled and 404’d — the post dropped after being accepted.

Both of those specific bugs were found and fixed by matching what Pixelfed actually does rather than what the spec says it should: verify against both signature styles, and use plain object addresses with no query string. Posts did land in testing once those were in.

Still confirming — Pixelfed stays the rough one: our Mastodon side is confirmed solid, but Pixelfed still behaves inconsistently and we see drops we haven’t fully explained yet. So we won’t call Pixelfed solid the way we can call Mastodon solid. The known bugs are fixed; the peer itself is still being watched.

Read the full report →

A failed signed fetch dropped every incoming like, boost, and reply

“Second Knock” Watched working

To verify an inbound activity we fetch the sender’s key, and we signed that outbound fetch. When a peer refused the signed fetch, the whole verification failed with “could not fetch signer,” and every like, boost, and reply from that server was dropped — which is why a wall of “signature verify failed” rejections was never actually a crypto problem. A failed signed fetch now retries unsigned, which is what most instances serve anyway. A companion tool re-pulls entries dropped during the outage so nobody had to re-post.

Watched working: boosts now show up properly on our own site’s Pixelfed page — confirmed live.

Eyeball next: a few instances genuinely insist on a signed fetch (authorized-fetch mode) and still refuse ours, so those specific servers aren’t confirmed yet. Named and being worked on.

Read the full report →

Modern Mastodon signs in a format our door couldn’t read

RFC-9421 Watched working

Mainline Mastodon 4.4 and later sign their inbox deliveries with a newer signature standard (RFC-9421) that our inbox verifier couldn’t read, so those deliveries bounced. We first made the rejection log say which scheme arrived — turning every bounce into direct evidence — then built the verifier to accept the new format alongside the old.

Watched working: talking to Mastodon looks flawless now — confirmed live. Mastodon also displays our GRAMOFSMACK carousels and our solo posts correctly.

Read the full report →

Likes from Pixelfed never landed

wrong key Fixed — confirming

A Like from Pixelfed points at the human-readable permalink of a photo, not the machine address our resolver knew how to match — so every like from a Pixelfed follower resolved to nothing and was dropped, each one showing as unresolved in the log. The resolver now also accepts the public permalink and maps it back to the right post or photo. (Notably, this was caught by the inbox log built for exactly this purpose — the drop was visible, so it got fixed.)

Read the full report →

The desktop poster silently dropped ALT text and the colour tag

COLD SNAP Watched working

The site accepted alt text and the colour / black-and-white tag all along, but the COLD SNAP desktop poster never actually sent them — so the ALT you carefully typed went nowhere, quietly. All three posting modes now send both, and every photo travels with its own description and colour tag attached to the image, offline and on the wire.

Read the full report →

What this cost us to learn — for anyone building with AI

SnapSmack is built by one photographer directing AI to write the code — and reading it back. An hour or more a day goes into reading the PHP and Python that comes out, with a second AI on hand to explain the tricky parts. That’s enough to follow what a chunk does and whether it fits, even where writing it from scratch is still out of reach — the way you can order confidently off a menu in a language you don’t yet speak. Not trying to understand the code the AI produces would be the irresponsible move. If that’s you — building a real thing through AI — every lesson below was paid for in one of the incidents above. They’re the part worth stealing.

“Fixed” and “watched working” are two different claims — never let them blur

This is the expensive one, and it’s why this whole page has coloured tags. When an AI writes the code, “the agent says it’s fixed” is a claim, not a result — even after you’ve read the change and it looks right. Track what you changed separately from what you’ve watched actually work on a real machine, and never let the first quietly become the second. The word “fixed” in a changelog fooled us more than once — the Photo Friday cards read “delivered” and vanished for weeks.

Read the code it writes, then watch it run — neither one alone is proof

Read the code the AI produces: understanding the change is part of the job, not something to skip because you didn’t write it. But reading it — even reading it carefully — is not proof it works. Knowing exactly what the finished thing must do, then watching the running system actually do it on a real machine, is the proof. Define that hole precisely before a line is written, read the code that comes back to check it fits, then test the real outcome against it and pare off anything that doesn’t. “The agent says it’s done” and “the code looks right” are both claims; the working outcome is the result.

A success code proves the message was carried, never that the job got done

A server answering 200 means the bytes arrived — not that your post landed, saved, or displayed. Treat every “success” as “received,” and confirm the actual outcome separately. Pixelfed accepted our posts and then silently dropped them; the acceptance was real and the post was gone.

Every assumption about how another system behaves is wrong until you prove it against that exact system

One peer working tells you nothing about the next. The fixes that made Pixelfed work proved nothing about Mastodon or GoToSocial — each had to be watched on its own. If your thing talks to anyone else’s thing, confirm it against their real system, not the spec and not a sibling that happened to pass.

You can’t fix what you can’t see — build the log before you chase the bug

Our worst bugs were invisible: things arrived, passed their checks, and disappeared with no trace, which reads exactly like “never happened.” The moment we made the system report what it did with each item — kept, ignored and why, dropped and where — the bugs became findable. Instrument first; then hunt.

Slow or background work never belongs inside a page load

If a job can be slow, never run it while a person is waiting for a page — it can take the whole site down with it. Ours did: paced work running inside web requests starved the server and timed out entire sites. Move anything slow to a place that can fail on its own without a visitor noticing.

“Set up” is not “working,” and “on the list” is not “connected”

Two of the worst outages here were things that looked configured and weren’t. Scheduled jobs were registered — pointing at a script that no longer existed. Blogs were added to the network roster — without ever being made to follow anyone. A presence check passes on both, and the failure is silent and looks like something else breaking. Check the real thing: that the command resolves to the installed script, that the relationship actually exists — not just that a record of it does.

Git history tells you what got fixed — never what’s still untested

A test you never ran writes nothing to the record. So “the changelog is clean” is not “the software is proven” — the two just look alike. That’s exactly why most tags on this page start yellow: the history proves the fix was written, and a person still has to watch it work before it earns green.

What we haven’t confirmed yet

These aren’t broken and they aren’t known bugs — they’re things we expect to work but haven’t yet stood in front of and watched. A quarter-million-line system confirmed by one photographer’s hands-on testing has a real list of things still to confirm, and we’d rather show you the list and work through it than quietly assume it’s all fine.

Federation with peers we haven’t specifically watched

Confirming

Posting and display are watched working on Mastodon and Pixelfed. GoToSocial and every other implementation start at “not confirmed” for each operation — post, comment, boost, display — until that specific peer is watched. One peer passing never confirms another, because the failures we’ve hit were always in the gap between what a peer should do and what it actually does.

Photo editor files opened in other editors

Watched working Confirming

SNAP SLAPPER into Photoshop is confirmed — a real export was opened in Photoshop and checked against the original with a difference file, and it held up. That’s the one direction we’ve genuinely proven.

Watched working: SNAP SLAPPER export → Photoshop, verified against the original with a difference file.

Still confirming: the other direction (a Photoshop file coming into SNAP SLAPPER) is not yet checked, and neither are other editors like Affinity. Each editor and each direction gets its own confirmation — one passing doesn’t vouch for the rest.

Everything a changelog calls “fixed” that nobody has since watched

Confirming

Git tells us honestly what got fixed and why. It cannot tell us what’s still broken or what was never tested — a test never run writes nothing to history. So most fixes on this page sit at fixed, not watched yet until a named person watches the real outcome on a real machine. We’d rather tell you that than launder a changelog line into a promise.

How “Watched Working” Gets Earned

SnapSmack is early — still closed beta, essentially one operator testing on real boxes. So right now almost everything that earns a green tag here was watched by us, and the honest defaults (“confirming”) are the things one pair of hands hasn’t gotten to yet. That’s the whole reason the labels are split the way they are: a named person watching the real outcome on a real machine is what turns “fixed” into “watched working,” and until the beta widens there simply aren’t many pairs of hands to do the watching.

What one operator does have is a controlled test bench that punches above its headcount. We run our own fediverse lab — Pixelfed, Mastodon and GoToSocial instances plus our own SnapSmack fleet — so we hold both ends of every test, with all the data and the raw logs and failures visible in real time. And we drive it with accelerated agentic testing: AI agents running the same paths over and over, fast, so failures surface quickly and repeatably instead of waiting on a manual pass. That’s how a “confirming” item gets watched to green without needing a crowd.

As more people run it, their named observations — a delivery that failed, a peer that choked, a job watched succeeding — will feed this ledger too. The codebase is public and open to inspection at any time: github.com/baddaywithacamera/snapsmack. Security issues — something that could be abused rather than something that just doesn’t work — belong in BUZZERS! once closed, and should be reported privately, not posted in public.