diff --git a/CLAUDE.md b/CLAUDE.md index a5cb5eb..29ba0e1 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -19,6 +19,18 @@ A FastAPI webhook server. One Mastodon admin webhook delivers `account.created`, and auto-**silence** young/dormant accounts past the threshold — lowered for accounts with a flagged signup IP (see README). +Alongside the webhook it serves a **reputation query API** — `POST +/api/reputation` — which is the pre-emptive counterpart to all of the above. +Mastodon calls it during signup, immediately before its own IpBlock / +EmailDomainBlock lookups; it classifies the IP/email domain, records it, +registers the block, and only then returns, so the lookup that follows +rejects the signup outright. It **writes**, so it is authenticated +(`X-Reputation-Key`) and 404s unless `REPUTATION_API_ENABLED` *and* +`REPUTATION_API_KEY` are both set. The caller is the +`call-welcomebot-reputation-api` patch on the Mastodon fork — see +`~/secondbrain/yttrx-documentation/mastodon-fork-patches.md`. README.md's +"Reputation query API" section has the full contract. + Alongside the webhook it serves an optional read-only **blocklist API** — `GET /api/blocks`, `/api/blocks/ip-ranges`, `/api/blocks/email-domains` — publishing the IP ranges and email domains the bot has auto-blocked. It is @@ -62,6 +74,17 @@ user-memory note `project_welcomebot` has the current state. check-mail.org's disposable/risk fields back into the response — that would be redistributing a paid reputation feed's data. Account identifiers (handles, ids, emails, raw signup IPs) are out for the usual reasons. +- **The reputation API must never serve unauthenticated.** It installs IP + blocks; an open endpoint that does that is a denial-of-service lever. An + empty `REPUTATION_API_KEY` disables the route rather than opening it, and + that behaviour is asserted in `test_local.py` — don't "fix" it into a + permissive default. +- **Don't remove the block-width cap or the block expiry.** `IP_SCRUTINY_MAX_ + BLOCK_PREFIX_V4`/`_V6` (24/48) and `IP_SCRUTINY_IPBLOCK_EXPIRES_IN` (24h) + exist because ipapi.is reports whole ASN routes: without the cap a single + `abuser` flag on a consumer address blocks its entire ISP route (routes of + /8–/11 are routine, and `197.211.32.0/19` was already registered against a + Benin mobile carrier). Raising them is a policy decision, not a tuning knob. - **`admin.yttrx.com`'s `/root/yttrx-welcomebot/` is not a git repo any more** (cleaned up 2026-07-06). Don't `git init`/clone it back into place — the registry image + `docker-compose.yml`/`.env` is the whole deploy surface now. @@ -217,6 +240,10 @@ needed. Without these scopes the reconcile 403s and exits having changed nothing, which is the deliberate failure mode: never read an outage as "every block was removed". +The reputation API needs no scopes beyond those above — it registers the same +`Admin::IpBlock` / `EmailDomainBlock` records as the webhook path, so +`admin:write:ip_blocks` + `admin:write:email_domain_blocks` cover it. + **The suspicious-signup sweep (`app/suspicious_sweep.py`) needs another scope, `admin:read:accounts`**, to `GET /api/v1/admin/accounts/:id` (fetches current post/follow counts + role/suspended state to decide whether to suspend) — diff --git a/README.md b/README.md index e973d40..a927968 100644 --- a/README.md +++ b/README.md @@ -21,6 +21,12 @@ A small FastAPI webhook server that receives Mastodon **admin webhooks** for - Signatures are HMAC-verified; both flows dedupe in a sqlite store so nobody is welcomed twice and no account is auto-actioned twice. +Both of those are **reactive** — they run after Mastodon has created the +account. The server also exposes a **pre-emptive** path: Mastodon can ask it +about an IP or email domain *during* signup, before consulting its own block +tables, and the answer arrives with the block already installed. See +[Reputation query API](#reputation-query-api-apireputation). + ``` new signup / new report on yttrx └─> Mastodon (mammut) fires admin webhook account.created|approved | report.created @@ -332,6 +338,107 @@ blocks a moderator added by hand in the admin UI are not in it. Entries appear only once the block actually landed on the Mastodon side (a failed `POST /api/v1/admin/ip_blocks` is logged and *not* recorded as blocked). +## Reputation query API (`/api/reputation`) + +The inversion of everything above. `POST /api/reputation` classifies an IP +and/or email domain, records it, registers the matching `Admin::IpBlock` / +`EmailDomainBlock`, and **only then returns** — so a caller that consults its +own block tables immediately afterwards finds a block that did not exist when +its request began. A bad network gets turned away on first contact instead of +cleaned up after the first account out of it. + +``` +signup begins on yttrx + └─> Mastodon (mammut), about to run its own IpBlock lookup + └─> POST https://hooks.yttrx.com/api/reputation (X-Reputation-Key: …) + └─> classify (ipapi.is / check-mail.org, cached) + record in reputation_query + register Admin::IpBlock / EmailDomainBlock + return + └─> IpBlock.severity_sign_up_block.containing(ip).exists? ← now true +``` + +Unlike the blocklist API this one **writes**, so it is authenticated and +refuses to serve without a key: an unauthenticated endpoint that installs IP +blocks is a denial-of-service lever, and there is deliberately no open mode. +With `REPUTATION_API_ENABLED` false or `REPUTATION_API_KEY` empty the path +404s as if it didn't exist. + +```bash +curl -s -X POST https://hooks.yttrx.com/api/reputation \ + -H "Content-Type: application/json" \ + -H "X-Reputation-Key: $REPUTATION_API_KEY" \ + -d '{"ip": "198.51.100.20", "email": "someone@example.com", "source": "sign_up"}' +``` + +```json +{ + "ip": {"ip": "198.51.100.20", "classification": "datacenter+abuser", + "org": "Example Cloud Inc", "flagged": true, "cidr": "198.51.100.0/24", + "blocked": true, "cached": false, "deferred": false}, + "email": {"domain": "example.com", "disposable": false, "risk": 12, + "flagged": false, "blocked": false, "cached": true, "deferred": false}, + "checked_at": "2026-09-13T23:11:58+00:00", + "flagged": true, + "blocked": true +} +``` + +Both request fields are optional (the caller's IP hook has no email address to +offer yet); at least one is required, or the call 400s. Either side may be +`null` if that signal is switched off or had nothing to look at. `blocked` +means "something is blocked on the Mastodon side as a result of this call" — +true both for a block installed just now and for one installed earlier that is +still in force. + +**The caller** is the `call-welcomebot-reputation-api` patch on yttrx's +Mastodon fork (`~/secondbrain/yttrx-documentation/mastodon-fork-patches.md`). +It is best-effort on that side: if this endpoint is slow, down or disabled, +Mastodon logs a warning and proceeds exactly as it would without the patch. + +### Cost control + +The caller hits this when the signup form is merely *rendered*, not only on +submission, so the endpoint is reachable by anyone crawling that page. +`REPUTATION_API_MAX_LOOKUPS_PER_HOUR` (default 500) caps **fresh** provider +lookups per rolling hour; cache hits are free and never counted. Over budget, +an uncached value is answered `"classification": "unknown", "deferred": true` +and nothing is installed — never `flagged`, since the point of the ceiling is +to stop paying, and guessing would either block innocents or waste the guard. + +The webhook path deliberately never spends from this budget: `account.created` +is low-volume and authoritative, and must not be starved by crawler traffic. + +Measured against the live service: ~70ms for a cache hit answered locally, +~700ms from mammut (dominated by TLS setup across the WAN), ~1.3-1.8s on the +first sighting of a bad range or domain, where it also registers the block. + +### Block width and expiry + +Two limits apply to every block this bot registers, from the query API and the +reactive webhook path alike: + +| Setting | Default | Why | +|---|---|---| +| `IP_SCRUTINY_MAX_BLOCK_PREFIX_V4` / `_V6` | `24` / `48` | ipapi.is reports the flagged address's whole ASN route. Right for hosting space, wrong for an ISP aggregate — routes of `/8`–`/11` turn up routinely, and `197.211.32.0/19` was registered against a Benin **mobile carrier** off one `abuser` flag. Anything wider falls back to the single address. | +| `IP_SCRUTINY_IPBLOCK_EXPIRES_IN` | `86400` (24h) | An IP block claims who is behind an address *now*, and VPS addresses are recycled constantly. A day breaks a signup run — which is the shape this abuse takes — without the blocklist accumulating forever. Email domain blocks have no expiry in Mastodon and are unaffected. | + +The cap applies to what gets **blocked**, not what gets **cached**: the wide +route is still the right cache key, since it genuinely belongs to one operator +and one lookup answers for the whole of it. + +### Records + +| Table | Holds | +|---|---| +| `reputation_query` | Audit log of every answer: what was asked, what the providers said, whether it led to a block. Domains only — never a full address. | +| `reputation_block` | One row per block installed here, keyed by the CIDR/domain (there is no account to key it to). Lets a repeat query skip the Mastodon round trip, and carries `expires_at` so the dedup record lapses exactly when the block does. | + +Both feed `/api/blocks` and the reconcile pass alongside the signup-derived +rows, so neither under-reports as blocking shifts to the pre-signup path. The +published entry shape is unchanged; `signups` still counts signups, so a range +blocked before anyone signed up from it reads `"signups": 0`. + ## Layout | Path | Purpose | @@ -341,7 +448,7 @@ only once the block actually landed on the Mastodon side (a failed | `app/reconcile_blocks.py` | Scheduled pass: marks blocks that no longer exist in Mastodon inactive (cron) | | `Dockerfile`, `docker-compose.yml` | Container build + run | | `.env.example` | Config template (copy to `.env`, never commit) | -| `nginx/hooks.yttrx.com.conf` | nginx site for admin.yttrx.com (webhook, healthz, blocklist API) | +| `nginx/hooks.yttrx.com.conf` | nginx site for admin.yttrx.com (webhook, healthz, blocklist API, reputation API) | | `bin/welcomebot-logs`, `bin/welcomebot-signups`, `bin/welcomebot-suspicious` | Ops CLIs installed to `/root/bin` on admin.yttrx.com | | `bin/yttrx-blocklist-import` | Importer other admins can run to pull this list into their own instance | | `bin/backfill-block-cidr.py` | One-shot: reconstruct `block_cidr` for rows predating it (run 2026-09-13) |