Blocks were registered permanently. An IP block is a claim about who is
currently behind an address, and that goes stale: VPS addresses are recycled
between customers constantly, and now that the /24 ceiling means most blocks
cover a single address, a permanent block increasingly punishes whoever gets
that address next rather than the actor we saw.
24h is long enough to break a signup run -- the abuse arrives in bursts, not
as a steady trickle -- and short enough that the blocklist stays a live
picture rather than something that only ever grows.
Mastodon expires these itself and reconcile_blocks already reads an elapsed
expires_at as "gone", so expired blocks leave /api/blocks unaided. The one
piece that needed adding is an expiry on our own dedup record: reconcile runs
hourly, so between a block lapsing and the next pass, reputation_block_
installed would otherwise claim a block Mastodon had already dropped and skip
re-installing it.
expires_in is omitted rather than sent as 0 when the expiry is switched off.
Email domain blocks have no expiry in Mastodon and are untouched.
ipapi.is reports the flagged address's whole ASN route, and register_ip_block
blocked it whatever its width. That is the right unit for hosting space, where
a repeat bad actor comes back from a different address in the same VPS block.
It is badly wrong for an ISP aggregate, and the route cache holds routes as
wide as /8, /9 and /11: one `abuser` flag on a residential or mobile address
inside one of those would have walled off millions of consumer addresses.
That was not hypothetical -- 197.211.32.0/19 is registered against
GLOMOBILE-GPRS, a Benin mobile carrier, from a single `abuser` flag: 8,192
addresses of consumer mobile space.
Anything wider than /24 (v4) or /48 (v6) now falls back to blocking the single
address. This costs blocking strength against large hosting allocations -- an
OVH /16 blocks one address now, so a repeat offender from that block needs a
fresh block each time -- and in exchange makes the collateral-damage case
impossible. Both ceilings are env-tunable.
The cap applies to what gets BLOCKED, not to what gets CACHED: the wide route
is still the right cache key, since it genuinely belongs to one operator and
one lookup answers for the whole of it. Narrowing the cache key would multiply
provider lookups for no benefit.
Applied at registration time, so blocks already in force are untouched.
Everything the bot does today is reactive: Mastodon creates the account,
the account.created webhook arrives, we classify the signup IP and email
domain, and then we block the range and suspend what already exists.
POST /api/reputation is the pre-emptive half. A caller asks about an IP
and/or email domain; we classify it, record the answer, register the
matching Admin::IpBlock / EmailDomainBlock, and only then return. A
caller that runs its own block lookup immediately afterwards therefore
finds a block that did not exist when its request started. The intended
caller is Mastodon's signup path, which does exactly that.
It writes, so unlike /api/blocks it is authenticated and refuses to serve
without a key (REPUTATION_API_KEY): an unauthenticated endpoint that
installs IP blocks is a denial-of-service lever, so there is no open mode.
Notes on the pieces that aren't obvious:
- REPUTATION_API_MAX_LOOKUPS_PER_HOUR caps *fresh* provider lookups. The
caller hits this on every view of the signup form, not just on
submission, so without a ceiling a crawler walking /auth/sign_up from a
new range each time would set our ipapi.is bill. Over budget we answer
"unknown" and install nothing -- never "flagged". Cache hits are free
and uncounted, and the webhook path never spends from this budget.
- Blocks installed here are recorded in their own table, keyed by the
CIDR/domain rather than by account (there is no account yet), and are
folded into /api/blocks and the reconcile pass so neither one
under-reports as blocking shifts to the pre-signup path. The published
entry shape is unchanged; a range blocked before anyone signed up from
it simply reads "signups": 0.
- A repeat query for an already-installed block skips the Mastodon round
trip. This runs on a live signup request, so that is saved latency.
ipblock_registered and domain_blocked only ever meant "we posted this
block once" — they said nothing about whether it still exists. When the
API went live it was publishing 24 /32 ranges that had been removed by
hand on 2026-07-21, so ~17% of the IP list didn't block anything. That's
a bad thing to hand another admin an importer for.
app/reconcile_blocks.py fetches Mastodon's live Admin::IpBlock and
Admin::EmailDomainBlock lists and maintains a new block_active column on
both tables: off for rows whose block is gone or whose expires_at has
elapsed, back on for any that reappear. The API serves only active rows.
Re-registering a block reactivates its row, so a range the bot blocks
again after a manual cleanup comes back on its own.
A failed fetch changes nothing — not a 403 from a missing scope, not
Mastodon being down, not a half-failure where one list answers and the
other doesn't. An outage must never be read as "every block was
removed"; the list degrades to over-reporting rather than emptying
itself, and there's a test for each of those paths.
Needs admin:read:ip_blocks + admin:read:email_domain_blocks, added to
the live token in place (scopes updated on the existing Doorkeeper
token, matched by digest) rather than by minting and rotating, so the
token value is unchanged and .env needed no edit.
Mastodon has no RBL/DNSBL support — nothing in it consults an outside
list when deciding whether to accept a signup, confirmed against v4.7.1
(the only outbound lookup in that path is EmailMxValidator's MX
resolution, which feeds the local email_domain_blocks table). So a
shared blocklist has to be imported rather than queried live, the same
way Garden Fence and other community lists propagate. These are the
pieces that make ours importable.
- ?format=txt on both per-list endpoints: one entry per line under a '#'
comment header, the form blocklists are traded in and the form a cron
job can consume without a JSON parser. The combined /api/blocks stays
JSON-only — a text rendering would mix CIDRs with domains in one file,
defeating the point. An unrecognised ?format is a 400 rather than a
silent fallback to JSON, and the enabled-gate still runs first so a
disabled deployment 404s either way.
- Access-Control-Allow-Origin: * on blocklist responses, so browser-side
admin tooling can read them. The data is already public.
- bin/yttrx-blocklist-import: standard-library-only importer another
admin can run against their own instance. Dry run unless --apply;
422 from Mastodon counts as "already blocked", not an error.
The importer defaults IP entries to sign_up_requires_approval rather
than the sign_up_block yttrx uses itself. The entries are whole ASN
routes, which suits yttrx because a flagged signup here is nearly always
datacenter/VPN/proxy space — that calibration isn't transferable, so an
importer gets the reviewable severity by default and has to opt into the
hard one.
Every row blocked before the block_cidr column existed had nothing to
report, so the endpoint fell back to the signup IP's /32 — the address
that signed up rather than the range that was blocked, which is the
opposite of what the list is for.
The range is recoverable from the bot's own ipapi range cache, which
stores the CIDR each classification was cached under — the same value
that was handed to register_ip_block. Run once on admin.yttrx.com:
of 154 blocked rows, 65 gained their real range, 79 genuinely were
single-address blocks (no asn.route from ipapi.is), and 10 with no cache
entry stay as a /32, since under-reporting beats claiming a range is
blocked when it isn't. Rows classified before 2026-07-15 are skipped —
range-blocking didn't exist yet, so only the /32 was ever registered.
Live output went from 150 entries (all /32) to 145, of which 59 are
real networks.
Publishes the IP ranges and email domains the bot has auto-blocked as
JSON: GET /api/blocks, /api/blocks/ip-ranges, /api/blocks/email-domains,
each taking an optional ?limit=N.
The endpoints are unauthenticated, so BLOCKS_API_ENABLED gates them and
defaults to false — with it unset the paths 404 as if they didn't exist.
Publishing which ranges are already blocked also tells a spam signup
which ones to avoid, so opting in is a per-deployment call; yttrx.com
sets it true in the box's .env.
Responses assert only that a block exists: the blocked range/domain, a
signups count, and first/last seen. No account identifiers, and none of
the ipapi.is or check-mail.org classification that drove the decision —
republishing a paid reputation feed's verdicts on an open endpoint would
be redistributing their data rather than ours.
Supporting changes:
- signup_ip gains a block_cidr column (migrated in place on open) so the
API reports the network actually blocked rather than the signup IP.
Rows predating it fall back to that IP's /32 or /128, which is exactly
what was registered before range-blocking started 2026-07-15.
- register_ip_block / register_email_domain_block now return whether the
block landed, and the caller only marks it registered on success, so a
failed POST is no longer published as a block that exists.
ipapi.is free tier returns company/asn as flat strings ("Google LLC",
"AS15169 Google LLC") not nested dicts. The old code assumed nested
objects and crashed with AttributeError when trying to call .get() on
strings. This broke IP-based signup scrutiny for every new signup.
Now handles both formats (string and dict) for backward compat. Route
field handling also simplified since free tier doesn't nest asn data.
Fixes signup accounts falling through without IP classification.
datacenter/proxy/tor/abuser is a much cleaner bulk/bot-signup indicator
than vpn, which also flags plenty of privacy-conscious real users. A
strong-flagged signup now suspends immediately regardless of the email
signal; vpn-only still gets the normal held-welcome/ipblock/lowered-
threshold treatment, falling through to the existing combined-signal
suspend only if also paired with a flagged email domain.
register_ip_block always blocked the exact signup IP as a /32 (or /128
for v6), which is close to pointless for the datacenter/VPN/proxy space
these flags fire on: a repeat bad actor from the same provider almost
never reuses the exact same address, but very often reuses a different
one in the same block.
classify_signup_ip now returns a fourth value, block_cidr — ipapi.is's
asn.route CIDR when it's a valid network that actually contains the
signup IP (already being tracked in ipapi_range_cache purely for lookup
caching), falling back to the address's own /32 or /128 when no usable
route exists. register_ip_block blocks that instead of always deriving
a /32 itself.
Added range_cache_tests() covering the cache/fallback logic directly
(pure sqlite + ipaddress, no network), and updated the existing
ip_scrutiny_tests()/combined_signal_tests() mocks for the new 4-tuple
classify_signup_ip return and register_ip_block arity.
The hourly OR-based sweep (SUSPICIOUS_GRACE_HOURS) still handles a
single-signal flag as before. This adds an additional check in
process_signup(): a signup flagged by BOTH IP-scrutiny and
email-domain scrutiny at once is a stronger signal, so it's suspended
right away instead of waiting out the grace period, skipping the
hold/welcome path and the suspicious_watch entry entirely.
Gated behind new SUSPICIOUS_COMBINED_* env vars, dry-run first per the
usual rollout convention (unlike SUSPICIOUS_DRY_RUN, which shipped
live by design). Falls back to the normal held-welcome path if the
suspend API call fails, and respects ABUSE_ALLOWLIST.
Uses asn.route from ipapi.is to bucket the classification cache by
network block (IPv4 and IPv6), so signups from the same datacenter/ASN
block no longer each burn a separate lookup against the 1000/day quota.
Falls back to a single-address range when route is missing or doesn't
actually contain the queried IP.
fetch_account_counts() was hitting the public /api/v1/accounts/:id
endpoint, which 404s for any account that hasn't confirmed its email
yet. Most flagged signups haven't confirmed at the moment
account.created/approved fires, so the baseline snapshot silently
failed and the account never entered suspicious_watch -- 6 of 8
IP-flagged signups to date missed the sweep entirely. Switch to the
admin endpoint suspicious_sweep.py already uses successfully.
Deploy target was cleaned up to just docker-compose.yml + .env (+ .env.bak-*
snapshots) after switching to registry-pulled images. Fixes references to
git pull, chown waffles, and app/main.py.bak rollback that no longer apply.
Verified from the actual job container logs: act_runner bind-mounts
/var/run/docker.sock into every job container unconditionally, even before
privileged mode was enabled. The previous step's nested dockerd never
actually got used.
act_runner now runs privileged, and the job image ships docker+dockerd, so
the build can happen directly in the job container. Needs vfs as the
storage driver since overlay2 doesn't nest inside overlay2 on this host.
account.approved never fires for a signup that went through open
registration without ever being queued (no prior block on that
address). process_signup was waiting on that event forever, so the
first flagged signup on any new bad IP/email domain never got welcomed
or watched by the suspicious-sweep. Now checks Admin::Account.approved
from the account.created payload and acts immediately if it's already
true.
With the sweep now running every 10 minutes, the per-run 'N due' summary
and per-account skip/clear log lines (already-limited, staff, cleared by
activity, not found) were mostly noise drowning out the actual suspend/
silence actions. Those states are still recorded in suspicious_watch.status
for anyone querying the DB directly; only the taken action, dry-run
simulation, and error paths still log.
Datacenter/hosting-IP-flagged signups are a strong enough spam signal on
their own that a full day of grace was mostly just delaying an inevitable
suspend. Cron cadence tightened from hourly to every 10 minutes so real
latency stays close to the new 1h window.
Watches every account flagged by IP-scrutiny or email-domain scrutiny at
signup and, once it goes live, records a baseline post/follow count. A
scheduled sweep (app/suspicious_sweep.py, run via cron on admin.yttrx.com)
suspends any watch past SUSPICIOUS_GRACE_HOURS with zero new posts and
zero new follows since that baseline; any activity clears the watch.
Works whether yttrx is open-registration or requires moderator approval,
since the watch starts at whichever event actually makes the account live
(account.created vs account.approved), same dual handling the welcome flow
already uses. Needs ABUSE_BOT_TOKEN re-minted with admin:read:accounts.
Classifies each signup's email domain (domain-only, GDPR-friendly) alongside
the existing IP scrutiny signal, with matching held-welcome and auto
email_domain_block behavior. A report against an account with a flagged
domain suspends immediately (no reporter-count threshold) and blocks the
domain, classified live from the report payload rather than any signup-time
record so it also covers pre-existing accounts. Ships CHECK_MAIL_DRY_RUN=true
by default, independent of ABUSE_DRY_RUN, so the new report-triggered suspend
path stays inert until watched.
pmb
waffle2k