README gets the full contract (request/response shape, the cost ceiling and why it exists, the block-width cap and expiry, and the two new tables). CLAUDE.md gets the two rules that are easy to undo by accident: the endpoint must never serve unauthenticated, and the width cap and expiry are policy decisions rather than tuning knobs.
This commit is contained in:
pmb committed
1 parent
6952fbadf6
commit
47e92ca10f
2 files changed
+135
-1
No files matched your search
@@ -19,6 +19,18 @@ A FastAPI webhook server. One Mastodon admin webhook delivers `account.created`,
|
||||
and auto-**silence** young/dormant accounts past the threshold — lowered for
|
||||
accounts with a flagged signup IP (see README).
|
||||
|
||||
Alongside the webhook it serves a **reputation query API** — `POST
|
||||
/api/reputation` — which is the pre-emptive counterpart to all of the above.
|
||||
Mastodon calls it during signup, immediately before its own IpBlock /
|
||||
EmailDomainBlock lookups; it classifies the IP/email domain, records it,
|
||||
registers the block, and only then returns, so the lookup that follows
|
||||
rejects the signup outright. It **writes**, so it is authenticated
|
||||
(`X-Reputation-Key`) and 404s unless `REPUTATION_API_ENABLED` *and*
|
||||
`REPUTATION_API_KEY` are both set. The caller is the
|
||||
`call-welcomebot-reputation-api` patch on the Mastodon fork — see
|
||||
`~/secondbrain/yttrx-documentation/mastodon-fork-patches.md`. README.md's
|
||||
"Reputation query API" section has the full contract.
|
||||
|
||||
Alongside the webhook it serves an optional read-only **blocklist API** —
|
||||
`GET /api/blocks`, `/api/blocks/ip-ranges`, `/api/blocks/email-domains` —
|
||||
publishing the IP ranges and email domains the bot has auto-blocked. It is
|
||||
@@ -62,6 +74,17 @@ user-memory note `project_welcomebot` has the current state.
|
||||
check-mail.org's disposable/risk fields back into the response — that would
|
||||
be redistributing a paid reputation feed's data. Account identifiers
|
||||
(handles, ids, emails, raw signup IPs) are out for the usual reasons.
|
||||
- **The reputation API must never serve unauthenticated.** It installs IP
|
||||
blocks; an open endpoint that does that is a denial-of-service lever. An
|
||||
empty `REPUTATION_API_KEY` disables the route rather than opening it, and
|
||||
that behaviour is asserted in `test_local.py` — don't "fix" it into a
|
||||
permissive default.
|
||||
- **Don't remove the block-width cap or the block expiry.** `IP_SCRUTINY_MAX_
|
||||
BLOCK_PREFIX_V4`/`_V6` (24/48) and `IP_SCRUTINY_IPBLOCK_EXPIRES_IN` (24h)
|
||||
exist because ipapi.is reports whole ASN routes: without the cap a single
|
||||
`abuser` flag on a consumer address blocks its entire ISP route (routes of
|
||||
/8–/11 are routine, and `197.211.32.0/19` was already registered against a
|
||||
Benin mobile carrier). Raising them is a policy decision, not a tuning knob.
|
||||
- **`admin.yttrx.com`'s `/root/yttrx-welcomebot/` is not a git repo any more**
|
||||
(cleaned up 2026-07-06). Don't `git init`/clone it back into place — the
|
||||
registry image + `docker-compose.yml`/`.env` is the whole deploy surface now.
|
||||
@@ -217,6 +240,10 @@ needed. Without these scopes the reconcile 403s and exits having changed
|
||||
nothing, which is the deliberate failure mode: never read an outage as "every
|
||||
block was removed".
|
||||
|
||||
The reputation API needs no scopes beyond those above — it registers the same
|
||||
`Admin::IpBlock` / `EmailDomainBlock` records as the webhook path, so
|
||||
`admin:write:ip_blocks` + `admin:write:email_domain_blocks` cover it.
|
||||
|
||||
**The suspicious-signup sweep (`app/suspicious_sweep.py`) needs another scope,
|
||||
`admin:read:accounts`**, to `GET /api/v1/admin/accounts/:id` (fetches current
|
||||
post/follow counts + role/suspended state to decide whether to suspend) —
|
||||
|
||||
@@ -21,6 +21,12 @@ A small FastAPI webhook server that receives Mastodon **admin webhooks** for
|
||||
- Signatures are HMAC-verified; both flows dedupe in a sqlite store so nobody
|
||||
is welcomed twice and no account is auto-actioned twice.
|
||||
|
||||
Both of those are **reactive** — they run after Mastodon has created the
|
||||
account. The server also exposes a **pre-emptive** path: Mastodon can ask it
|
||||
about an IP or email domain *during* signup, before consulting its own block
|
||||
tables, and the answer arrives with the block already installed. See
|
||||
[Reputation query API](#reputation-query-api-apireputation).
|
||||
|
||||
```
|
||||
new signup / new report on yttrx
|
||||
└─> Mastodon (mammut) fires admin webhook account.created|approved | report.created
|
||||
@@ -332,6 +338,107 @@ blocks a moderator added by hand in the admin UI are not in it. Entries appear
|
||||
only once the block actually landed on the Mastodon side (a failed
|
||||
`POST /api/v1/admin/ip_blocks` is logged and *not* recorded as blocked).
|
||||
|
||||
## Reputation query API (`/api/reputation`)
|
||||
|
||||
The inversion of everything above. `POST /api/reputation` classifies an IP
|
||||
and/or email domain, records it, registers the matching `Admin::IpBlock` /
|
||||
`EmailDomainBlock`, and **only then returns** — so a caller that consults its
|
||||
own block tables immediately afterwards finds a block that did not exist when
|
||||
its request began. A bad network gets turned away on first contact instead of
|
||||
cleaned up after the first account out of it.
|
||||
|
||||
```
|
||||
signup begins on yttrx
|
||||
└─> Mastodon (mammut), about to run its own IpBlock lookup
|
||||
└─> POST https://hooks.yttrx.com/api/reputation (X-Reputation-Key: …)
|
||||
└─> classify (ipapi.is / check-mail.org, cached)
|
||||
record in reputation_query
|
||||
register Admin::IpBlock / EmailDomainBlock
|
||||
return
|
||||
└─> IpBlock.severity_sign_up_block.containing(ip).exists? ← now true
|
||||
```
|
||||
|
||||
Unlike the blocklist API this one **writes**, so it is authenticated and
|
||||
refuses to serve without a key: an unauthenticated endpoint that installs IP
|
||||
blocks is a denial-of-service lever, and there is deliberately no open mode.
|
||||
With `REPUTATION_API_ENABLED` false or `REPUTATION_API_KEY` empty the path
|
||||
404s as if it didn't exist.
|
||||
|
||||
```bash
|
||||
curl -s -X POST https://hooks.yttrx.com/api/reputation \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "X-Reputation-Key: $REPUTATION_API_KEY" \
|
||||
-d '{"ip": "198.51.100.20", "email": "[email protected]", "source": "sign_up"}'
|
||||
```
|
||||
|
||||
```json
|
||||
{
|
||||
"ip": {"ip": "198.51.100.20", "classification": "datacenter+abuser",
|
||||
"org": "Example Cloud Inc", "flagged": true, "cidr": "198.51.100.0/24",
|
||||
"blocked": true, "cached": false, "deferred": false},
|
||||
"email": {"domain": "example.com", "disposable": false, "risk": 12,
|
||||
"flagged": false, "blocked": false, "cached": true, "deferred": false},
|
||||
"checked_at": "2026-09-13T23:11:58+00:00",
|
||||
"flagged": true,
|
||||
"blocked": true
|
||||
}
|
||||
```
|
||||
|
||||
Both request fields are optional (the caller's IP hook has no email address to
|
||||
offer yet); at least one is required, or the call 400s. Either side may be
|
||||
`null` if that signal is switched off or had nothing to look at. `blocked`
|
||||
means "something is blocked on the Mastodon side as a result of this call" —
|
||||
true both for a block installed just now and for one installed earlier that is
|
||||
still in force.
|
||||
|
||||
**The caller** is the `call-welcomebot-reputation-api` patch on yttrx's
|
||||
Mastodon fork (`~/secondbrain/yttrx-documentation/mastodon-fork-patches.md`).
|
||||
It is best-effort on that side: if this endpoint is slow, down or disabled,
|
||||
Mastodon logs a warning and proceeds exactly as it would without the patch.
|
||||
|
||||
### Cost control
|
||||
|
||||
The caller hits this when the signup form is merely *rendered*, not only on
|
||||
submission, so the endpoint is reachable by anyone crawling that page.
|
||||
`REPUTATION_API_MAX_LOOKUPS_PER_HOUR` (default 500) caps **fresh** provider
|
||||
lookups per rolling hour; cache hits are free and never counted. Over budget,
|
||||
an uncached value is answered `"classification": "unknown", "deferred": true`
|
||||
and nothing is installed — never `flagged`, since the point of the ceiling is
|
||||
to stop paying, and guessing would either block innocents or waste the guard.
|
||||
|
||||
The webhook path deliberately never spends from this budget: `account.created`
|
||||
is low-volume and authoritative, and must not be starved by crawler traffic.
|
||||
|
||||
Measured against the live service: ~70ms for a cache hit answered locally,
|
||||
~700ms from mammut (dominated by TLS setup across the WAN), ~1.3-1.8s on the
|
||||
first sighting of a bad range or domain, where it also registers the block.
|
||||
|
||||
### Block width and expiry
|
||||
|
||||
Two limits apply to every block this bot registers, from the query API and the
|
||||
reactive webhook path alike:
|
||||
|
||||
| Setting | Default | Why |
|
||||
|---|---|---|
|
||||
| `IP_SCRUTINY_MAX_BLOCK_PREFIX_V4` / `_V6` | `24` / `48` | ipapi.is reports the flagged address's whole ASN route. Right for hosting space, wrong for an ISP aggregate — routes of `/8`–`/11` turn up routinely, and `197.211.32.0/19` was registered against a Benin **mobile carrier** off one `abuser` flag. Anything wider falls back to the single address. |
|
||||
| `IP_SCRUTINY_IPBLOCK_EXPIRES_IN` | `86400` (24h) | An IP block claims who is behind an address *now*, and VPS addresses are recycled constantly. A day breaks a signup run — which is the shape this abuse takes — without the blocklist accumulating forever. Email domain blocks have no expiry in Mastodon and are unaffected. |
|
||||
|
||||
The cap applies to what gets **blocked**, not what gets **cached**: the wide
|
||||
route is still the right cache key, since it genuinely belongs to one operator
|
||||
and one lookup answers for the whole of it.
|
||||
|
||||
### Records
|
||||
|
||||
| Table | Holds |
|
||||
|---|---|
|
||||
| `reputation_query` | Audit log of every answer: what was asked, what the providers said, whether it led to a block. Domains only — never a full address. |
|
||||
| `reputation_block` | One row per block installed here, keyed by the CIDR/domain (there is no account to key it to). Lets a repeat query skip the Mastodon round trip, and carries `expires_at` so the dedup record lapses exactly when the block does. |
|
||||
|
||||
Both feed `/api/blocks` and the reconcile pass alongside the signup-derived
|
||||
rows, so neither under-reports as blocking shifts to the pre-signup path. The
|
||||
published entry shape is unchanged; `signups` still counts signups, so a range
|
||||
blocked before anyone signed up from it reads `"signups": 0`.
|
||||
|
||||
## Layout
|
||||
|
||||
| Path | Purpose |
|
||||
@@ -341,7 +448,7 @@ only once the block actually landed on the Mastodon side (a failed
|
||||
| `app/reconcile_blocks.py` | Scheduled pass: marks blocks that no longer exist in Mastodon inactive (cron) |
|
||||
| `Dockerfile`, `docker-compose.yml` | Container build + run |
|
||||
| `.env.example` | Config template (copy to `.env`, never commit) |
|
||||
| `nginx/hooks.yttrx.com.conf` | nginx site for admin.yttrx.com (webhook, healthz, blocklist API) |
|
||||
| `nginx/hooks.yttrx.com.conf` | nginx site for admin.yttrx.com (webhook, healthz, blocklist API, reputation API) |
|
||||
| `bin/welcomebot-logs`, `bin/welcomebot-signups`, `bin/welcomebot-suspicious` | Ops CLIs installed to `/root/bin` on admin.yttrx.com |
|
||||
| `bin/yttrx-blocklist-import` | Importer other admins can run to pull this list into their own instance |
|
||||
| `bin/backfill-block-cidr.py` | One-shot: reconstruct `block_cidr` for rows predating it (run 2026-09-13) |
|
||||
|
||||
Reference in new issue
Block a user