yttrx-welcomebot/README.md

# yttrx welcome-bot + abuse-bot

A small FastAPI webhook server that receives Mastodon **admin webhooks** for
[yttrx.com](https://yttrx.com) and reacts to two events:

- **`account.created` / `account.approved`** — direct-messages a welcome toot
  to every new local signup (the welcome bot). Both events are handled so it
  works whether registration approval is on or off; dedup welcomes once.
- **`report.created`** — evaluates the reported account and, when enough
  **distinct** reporters have open reports against a *young* or *dormant*
  account, auto-**silences** it (reversible) and DMs a moderator for review
  (the abuse bot). Special case: a report against an account whose email
  domain is flagged high-risk (see below) auto-**suspends** it immediately
  (no reporter-count threshold) and blocks the domain.

- **Runs on:** `admin.yttrx.com` (Docker container, nginx-proxied at
  `https://hooks.yttrx.com`)
- **Mastodon side:** `mammut` — one admin webhook subscribing to both events,
  a welcome bot account (posts the DMs), and a separate **moderator** bot
  account whose token carries the admin scopes used to silence accounts.
- Signatures are HMAC-verified; both flows dedupe in a sqlite store so nobody
  is welcomed twice and no account is auto-actioned twice.

Both of those are **reactive** — they run after Mastodon has created the
account. The server also exposes a **pre-emptive** path: Mastodon can ask it
about an IP or email domain *during* signup, before consulting its own block
tables, and the answer arrives with the block already installed. See
[Reputation query API](#reputation-query-api-apireputation).

```
new signup / new report on yttrx
   └─> Mastodon (mammut) fires admin webhook  account.created|approved | report.created
         └─> POST https://hooks.yttrx.com/webhook   (X-Hub-Signature: sha256=…)
               └─> nginx (admin) -> 127.0.0.1:8087 -> bot container
                     ├─ account.created/approved -> POST /api/v1/statuses (welcome DM)
                     └─ report.created  -> classify target, count distinct
                                           reporters, maybe silence + DM mod
```

## Abuse-bot policy

On `report.created` against a **local**, not-already-limited account that is
**not staff** (no Admin/Owner/Moderator role — `ABUSE_SKIP_PRIVILEGED`) and not
on `ABUSE_ALLOWLIST`:

1. **Classify the account** by its authored posts (reblogs excluded):
   - `young` — oldest post is newer than `ABUSE_YOUNG_MAX_DAYS` (30d)
   - `dormant` — newest post is older than `ABUSE_DORMANT_MIN_DAYS` (30d)
   - `no-posts` — reported account with no authored statuses
   - `active` — everything else (posting for >1mo and posted recently)
2. **Count distinct reporters** with *open* reports against the target
   (self-reports excluded — one person filing repeatedly counts once).
3. **Silence** the account when distinct reporters ≥ the tier threshold:
   - young / dormant / no-posts → `ABUSE_SOURCES_NEWDORMANT` (2)
   - active → `ABUSE_SOURCES_ACTIVE` (3)
4. The action is `silence` (reversible) and is applied **without** a
   `report_id`, so the report **stays open** in the moderation queue. The bot
   then DMs `MOD_ALERT_ACCT` a summary with a link to the report, and DMs the
   **silenced user** a link to the appeals/help page (`ABUSE_HELP_URL`,
   `https://welcome.yttrx.com/posts/account-limited/`).

`ABUSE_DRY_RUN=true` (the shipped default) logs + DMs what *would* happen
without touching any account — keep it on until you've watched it for a while.

## IP-based signup scrutiny

Every `account.created` delivery already carries the signup IP for free
(`Admin::Account.ip`). On each new local signup, the bot:

1. **Classifies the IP** via [ipapi.is](https://ipapi.is/) (cached in sqlite;
   free, keyless tier is 1,000 req/day — `IP_SCRUTINY_IPAPI_KEY` raises the
   limit if needed) as any combination of `datacenter`, `vpn`, `proxy`, `tor`,
   and `abuser` (ipapi.is's own abuse score), joined with `+`, or `clean` if
   none matched. API failures classify as `unknown` and are never flagged.
2. If **any signal matched**, the signup is treated with more scrutiny:
   - **Moderator DM** with the IP, org, and classification.
   - **Held welcome** (`IP_SCRUTINY_HOLD_WELCOME`) — the welcome DM is skipped
     on `account.created` and only sent when `account.approved` fires, i.e.
     once a human clears yttrx's existing approval-required registration
     gate. If the signup is rejected instead, no welcome is ever sent.
   - **Auto-registered IP block** (`IP_SCRUTINY_AUTO_IPBLOCK`) — the *network*
     the signup IP belongs to (ipapi.is's `asn.route`, when it's a valid CIDR
     that actually contains the IP and falls inside the allowed width window;
     otherwise the `/24` — `/48` for v6 — containing the address, never the
     bare `/32`/`/128`) is added to Mastodon's native `Admin::IpBlock` at
     `IP_SCRUTINY_IPBLOCK_SEVERITY` (default `sign_up_block` — a hard reject
     of future signups from the range at registration, no moderator queue,
     no appeal via this bot; reversible from the admin UI). Blocking the
     whole network rather than
     the single address matters here specifically because a flagged signup is
     almost always datacenter/VPN/proxy space — a repeat bad actor is far more
     likely to come back from a different address in the same block than the
     exact one, unlike a residential IP where that reasoning wouldn't hold.
   - **Lowered abuse-bot threshold** — if this account is later reported, the
     usual `ABUSE_SOURCES_*` distinct-reporter threshold is replaced by
     `IP_SCRUTINY_ABUSE_THRESHOLD` (whichever is lower), since a flagged
     signup IP plus a report is a stronger combined signal than either alone.
3. If the matched classification includes `datacenter`, `proxy`, `tor`, or
   `abuser` — a *strong* signal, gated by `IP_SCRUTINY_STRONG_SUSPEND_ENABLED`
   — the signup is **suspended immediately** at signup time instead of just
   getting the held-welcome/ipblock/lowered-threshold treatment above, no
   matter what the email-domain signal says. `vpn` alone is deliberately
   excluded from this list (it also flags plenty of privacy-conscious real
   users, and is a weaker bulk/bot-signup indicator on its own) — a vpn-only
   flag still falls through to the normal path, though it's still caught by
   the combined-signal suspend below if the email domain is *also* flagged.
   `IP_SCRUTINY_STRONG_SUSPEND_ACTION` (default `suspend`) is the action
   taken; `IP_SCRUTINY_STRONG_SUSPEND_DRY_RUN=true` (the shipped default)
   logs + DMs what would happen without acting. Skips the held welcome and
   suspicious-watch entry entirely, same shape as the combined-signal suspend
   (§ below) — the account is already gone.

`IP_SCRUTINY_DRY_RUN=true` (the shipped default) classifies and DMs a
moderator without holding any welcome or writing any ip_block — keep it on
until you've watched the false-positive rate of the hosting/mobile keyword
regexes for a while. Registering ip_blocks requires the `ABUSE_BOT_TOKEN` to
carry the `admin:write:ip_blocks` scope in addition to its existing scopes
(see `CLAUDE.md`).

## Disposable/high-risk email signup scrutiny

Every `account.created` delivery already carries the signup email for free
(`Admin::Account.email`). On each new local signup, the bot:

1. **Classifies the domain only** (never the full email address — GDPR-
   friendly) via [check-mail.org](https://check-mail.org/), cached in sqlite.
   Flagged if the API marks it `is_disposable` or its `risk` score (0-100) is
   at or above `CHECK_MAIL_RISK_THRESHOLD` (default 80).
2. If flagged, same scrutiny shape as the IP signal:
   - **Moderator DM** with the domain and risk score.
   - **Held welcome** (`CHECK_MAIL_HOLD_WELCOME`) — same semantics as
     `IP_SCRUTINY_HOLD_WELCOME`.
   - **Auto-registered email domain block** (`CHECK_MAIL_AUTO_DOMAIN_BLOCK`)
     — the domain is added to Mastodon's native `Admin::EmailDomainBlock`,
     blocking future signups from it.

A **report** against an account whose email domain is flagged is a special,
stronger case (not merely a lowered threshold like the IP signal): it
auto-**suspends** the account on the very first report — no distinct-reporter
threshold — and (re-)registers the email domain block. This is classified
**live** from the report payload's `target_account.email` (a full
`Admin::Account`, always present), not from any record made at signup time —
so it also catches accounts that signed up before this feature existed or
while `CHECK_MAIL_ENABLED`/`CHECK_MAIL_API_KEY` was off. Gated by
`ABUSE_DRY_RUN` **or** `CHECK_MAIL_DRY_RUN` (either one holds it) —
deliberately independent of `ABUSE_DRY_RUN` alone, so this brand-new action
ships inert by `CHECK_MAIL_DRY_RUN`'s own default even once the general
abuse-bot is live.

`CHECK_MAIL_DRY_RUN=true` (the shipped default) classifies and DMs a
moderator without holding any welcome or writing any email_domain_block —
keep it on until you've watched the false-positive rate for a while.
Registering email_domain_blocks requires the `ABUSE_BOT_TOKEN` to carry the
`admin:write:email_domain_blocks` scope in addition to its existing scopes
(see `CLAUDE.md`).

## Junk-username purge

On `account.created`, a username that fully matches one of
`JUNK_USERNAME_PATTERNS` (whitespace-separated regexes, case-insensitive;
default `bp[0-9a-f]{16}`) is **suspended and deleted** immediately, before
any IP/email lookup, welcome, or moderator DM. Mastodon's own username
blocks only do equals/contains, hence regexes here.

Added 2026-10-03 for a campaign of ~100/day API signups named `bp` + 16 hex,
from residential JP IPs, never confirmed, using dead JP carrier addresses
and plus-tagged real icloud/gmail inboxes — every confirmation email (and
re-send) bounced into the yttrx inbox or bombed a stranger.

- The suspend is sent with `send_email_notification=false`, so no warning
  email goes to the junk address. The signup's own confirmation email has
  usually already gone out by the time the webhook lands; this stops the
  re-sends and the pile-up, not that first send.
- Mastodon's suspend registers a **canonical email block** on the address
  with `+tag` and dots stripped, so every later plus-variant of the same
  inbox is rejected at signup.
- The follow-up `DELETE /api/v1/admin/accounts/:id` needs the existing
  `admin:write:accounts` scope and an Admin role on `bot`.
- `ABUSE_ALLOWLIST` is honoured. If the purge call fails the signup falls
  through to the normal path. `JUNK_USERNAME_DRY_RUN=true` (shipped default)
  only logs `[DRY-RUN] would purge`.

## Suspicious-signup sweep

A scheduled companion to the two signup-scrutiny signals above (`app/suspicious_sweep.py`,
run via `docker exec yttrx-welcomebot python -m app.suspicious_sweep`, e.g. hourly cron on
`admin.yttrx.com`):

1. Any account flagged by **either** IP-scrutiny (datacenter/vpn/proxy/tor/
   abuser) or email-domain scrutiny (disposable/high-risk) has a grace-period clock
   started the moment it goes live — `account.created` if signups are open
   (or auto-approved), `account.approved` if this instance requires
   moderator approval. Same dual-event handling the welcome flow already
   uses, so it works in either registration mode without extra config. A
   baseline snapshot of `statuses_count`/`following_count` is recorded at
   that moment (`app.main.maybe_start_suspicious_watch`, table
   `suspicious_watch`).
2. The sweep looks for watches past `SUSPICIOUS_GRACE_HOURS` (default 1;
   lowered from the original 24 on 2026-07-06 — a flagged-IP
   signup is already a strong enough spam signal that a full day of grace
   was mostly just delaying an inevitable suspend) and compares current
   counts to the baseline:
   - **Any new post or new follow** since the baseline → cleared, never
     rechecked.
   - **Zero of either** → `SUSPICIOUS_ACTION` (default `suspend`) is applied,
     the moderator is DMed, and the watch is marked resolved.
3. Already-suspended/silenced accounts, staff (`ABUSE_SKIP_PRIVILEGED`), and
   `ABUSE_ALLOWLIST` handles are skipped exactly as elsewhere in this bot —
   the sweep reuses those same settings rather than defining its own.

Unlike the report-driven abuse-bot (default `silence`), this sweep defaults
to **`suspend`** — the reasoning is that zero organic activity in a full day
is a stronger bulk/bot-signup signal than a report alone, and Mastodon
suspensions have a server-side undo window. It also ships **live**
(`SUSPICIOUS_DRY_RUN=false`) rather than with the usual dry-run rollout step;
`SUSPICIOUS_DRY_RUN=true` remains available as a kill switch if you want to
pause it without redeploying.

Requires `ABUSE_BOT_TOKEN` to additionally carry the **`admin:read:accounts`**
scope (fetches live counts + role/suspended state via
`GET /api/v1/admin/accounts/:id`) — see `CLAUDE.md`'s moderator-token-gotcha
section for the remint procedure.

Inspect the watch list: `welcomebot-suspicious` / `welcomebot-suspicious --pending`
(see `bin/welcomebot-suspicious`).

## Blocklist API (`/api/blocks`)

An optional, **read-only** JSON view of what the bot has auto-blocked. Three
endpoints, all `GET`, all accepting an optional `?limit=N`:

| Path | Returns |
|---|---|
| `/api/blocks` | both lists in one document |
| `/api/blocks/ip-ranges` | networks registered as `Admin::IpBlock` |
| `/api/blocks/email-domains` | domains registered as `Admin::EmailDomainBlock` |

```json
{
  "generated_at": "2026-09-13T00:00:00+00:00",
  "ip_ranges": {
    "count": 1,
    "items": [
      {"cidr": "198.51.100.0/24", "signups": 3,
       "first_blocked_at": "2026-07-15 09:12:04",
       "last_blocked_at": "2026-08-30 22:41:18"}
    ]
  },
  "email_domains": {
    "count": 1,
    "items": [
      {"domain": "temp-mail.org", "signups": 2,
       "first_blocked_at": "2026-07-03 11:02:55",
       "last_blocked_at": "2026-07-19 16:30:09"}
    ]
  }
}
```

### Plain-text format

Both per-list endpoints also take `?format=txt`, which renders one entry per
line under a `#` comment header — the form blocklists are normally traded in,
and the form another admin's cron job can consume without a JSON parser:

```bash
curl -s 'https://hooks.yttrx.com/api/blocks/ip-ranges?format=txt'   | grep -v '^#'
curl -s 'https://hooks.yttrx.com/api/blocks/email-domains?format=txt' | grep -v '^#'
```

The combined `/api/blocks` is JSON-only: a text rendering would put CIDRs and
domains in one file, and the point of the text form is that it pipes straight
into something expecting one kind of entry. Responses carry
`Access-Control-Allow-Origin: *` so browser-side tooling can read them.

### Importing it elsewhere (`bin/yttrx-blocklist-import`)

Mastodon has **no RBL/DNSBL support** — nothing in it consults an outside list
when deciding whether to accept a signup. (The only outbound lookup in that
path is `EmailMxValidator`'s MX resolution, and that feeds the *local*
`email_domain_blocks` table.) So another instance can't point at this feed
live; it has to import, the same way [Garden Fence](https://gardenfence.github.io)
and other community lists propagate.

`bin/yttrx-blocklist-import` does that — standard library only, no install:

```bash
# dry run (the default — nothing is written)
./bin/yttrx-blocklist-import --instance https://your.instance --token "$TOKEN"
# actually write the blocks
./bin/yttrx-blocklist-import --instance https://your.instance --token "$TOKEN" --apply
```

The token needs `admin:write:ip_blocks` and `admin:write:email_domain_blocks`
(the script's `--help` has a Rails-console snippet that mints one). Entries
already present come back `422` and are counted as "already blocked", not as
errors. `--min-signups N` skips one-off entries, `--ips-only`/`--domains-only`
take one list, `--limit` caps each.

**On severity:** the IP entries are whole networks, because on yttrx a flagged
signup is almost always datacenter/VPN/proxy space where the next attempt comes
from a neighbouring address. That's calibrated to yttrx's traffic, not someone
else's, so the importer defaults to `sign_up_requires_approval` — a flagged
signup lands in the importer's moderation queue rather than being rejected —
even though yttrx runs these at `sign_up_block` itself. `--ip-severity
sign_up_block` opts into the harder behaviour.

**Off by default.** These paths are served *without authentication*, so they
404 unless `BLOCKS_API_ENABLED=true` is set — publishing which ranges and
domains are already blocked also tells a would-be spam signup which ones to
avoid, and that's a per-deployment call. yttrx.com opts in; the shipped
default doesn't. `BLOCKS_API_MAX_ITEMS` (default 5000) caps the response size
whatever `?limit` asks for, and `BLOCKS_API_CACHE_SECONDS` (default 300) sets
the `Cache-Control: public, max-age=…` on responses.

What the response deliberately does **not** contain:

- **No account identifiers** — no handles, account ids, emails or signup IPs.
  Repeat signups out of one range collapse into a single entry with a
  `signups` count.
- **No classification data.** The ipapi.is verdict (`datacenter`/`proxy`/…),
  the org name, and check-mail.org's disposable flag and risk score all stay
  out of the response. They're what *drove* each decision, but republishing a
  paid reputation feed's verdicts on an open endpoint would be redistributing
  their data rather than ours. An entry asserts only that the block exists.

Entries report the **network** that was blocked (ipapi.is's `asn.route` for
the signup IP, clamped to the width window, which is what `register_ip_block`
registers), not the address that signed up. A `/32` or `/128` entry means a
single address really is what was blocked — a historical row now, since every
block registered from 2026-09-21 covers at least a `/24`: either the row
predates the width floor, or it predates range-blocking (2026-07-15). The `block_cidr` column recording
this arrived with the API itself, so existing rows were backfilled once from
the ipapi range cache by `bin/backfill-block-cidr.py`; ten rows with no cache
entry left are shown as their `/32`, since a narrower entry is a safer error
than claiming a range is blocked when it isn't.

### Staying accurate (`app/reconcile_blocks.py`)

`ipblock_registered` / `domain_blocked` only ever meant "we posted this block
once" — they say nothing about whether it's still there. A moderator clearing
entries by hand, or an `expires_at` elapsing, would leave the endpoint
publishing blocks that no longer block anything. (This was not hypothetical:
when the API first went live it was publishing 24 `/32` ranges that had been
removed on 2026-07-21.)

The reconcile pass fetches Mastodon's live `Admin::IpBlock` and
`Admin::EmailDomainBlock` lists and flips `block_active` off for rows whose
block is gone or expired, back on for any that reappear. The API serves only
active rows, so `?format=txt` hands an importer only blocks actually in force.

```bash
docker exec yttrx-welcomebot python -m app.reconcile_blocks --dry-run
docker exec yttrx-welcomebot python -m app.reconcile_blocks
```

Runs hourly by cron on admin.yttrx.com, alongside the suspicious-signup sweep.
It needs `admin:read:ip_blocks` and `admin:read:email_domain_blocks` on
`ABUSE_BOT_TOKEN`. **A failed fetch changes nothing** — a 403 from a missing
scope, or Mastodon being down, must never be read as "every block was
removed", so rows keep their existing state and the list degrades to
over-reporting rather than emptying itself.

Scope: this is the **bot's own record**, not Mastodon's full blocklist —
blocks a moderator added by hand in the admin UI are not in it. Entries appear
only once the block actually landed on the Mastodon side (a failed
`POST /api/v1/admin/ip_blocks` is logged and *not* recorded as blocked).

## Reputation query API (`/api/reputation`)

The inversion of everything above. `POST /api/reputation` classifies an IP
and/or email domain, records it, registers the matching `Admin::IpBlock` /
`EmailDomainBlock`, and **only then returns** — so a caller that consults its
own block tables immediately afterwards finds a block that did not exist when
its request began. A bad network gets turned away on first contact instead of
cleaned up after the first account out of it.

```
signup begins on yttrx
   └─> Mastodon (mammut), about to run its own IpBlock lookup
         └─> POST https://hooks.yttrx.com/api/reputation  (X-Reputation-Key: …)
               └─> classify (ipapi.is / check-mail.org, cached)
                   record in reputation_query
                   register Admin::IpBlock / EmailDomainBlock
                   return
         └─> IpBlock.severity_sign_up_block.containing(ip).exists?  ← now true
```

Unlike the blocklist API this one **writes**, so it is authenticated and
refuses to serve without a key: an unauthenticated endpoint that installs IP
blocks is a denial-of-service lever, and there is deliberately no open mode.
With `REPUTATION_API_ENABLED` false or `REPUTATION_API_KEY` empty the path
404s as if it didn't exist.

```bash
curl -s -X POST https://hooks.yttrx.com/api/reputation \
  -H "Content-Type: application/json" \
  -H "X-Reputation-Key: $REPUTATION_API_KEY" \
  -d '{"ip": "198.51.100.20", "email": "[email protected]", "source": "sign_up"}'
```

```json
{
  "ip": {"ip": "198.51.100.20", "classification": "datacenter+abuser",
         "org": "Example Cloud Inc", "flagged": true, "cidr": "198.51.100.0/24",
         "blocked": true, "cached": false, "deferred": false},
  "email": {"domain": "example.com", "disposable": false, "risk": 12,
            "flagged": false, "blocked": false, "cached": true, "deferred": false},
  "checked_at": "2026-09-13T23:11:58+00:00",
  "flagged": true,
  "blocked": true
}
```

Both request fields are optional (the caller's IP hook has no email address to
offer yet); at least one is required, or the call 400s. Either side may be
`null` if that signal is switched off or had nothing to look at. `blocked`
means "something is blocked on the Mastodon side as a result of this call" —
true both for a block installed just now and for one installed earlier that is
still in force.

**The caller** is the `call-welcomebot-reputation-api` patch on yttrx's
Mastodon fork (`~/secondbrain/yttrx-documentation/mastodon-fork-patches.md`).
It is best-effort on that side: if this endpoint is slow, down or disabled,
Mastodon logs a warning and proceeds exactly as it would without the patch.

### Cost control

The caller hits this when the signup form is merely *rendered*, not only on
submission, so the endpoint is reachable by anyone crawling that page.
`REPUTATION_API_MAX_LOOKUPS_PER_HOUR` (default 500) caps **fresh** provider
lookups per rolling hour; cache hits are free and never counted. Over budget,
an uncached value is answered `"classification": "unknown", "deferred": true`
and nothing is installed — never `flagged`, since the point of the ceiling is
to stop paying, and guessing would either block innocents or waste the guard.

The webhook path deliberately never spends from this budget: `account.created`
is low-volume and authoritative, and must not be starved by crawler traffic.

Measured against the live service: ~70ms for a cache hit answered locally,
~700ms from mammut (dominated by TLS setup across the WAN), ~1.3-1.8s on the
first sighting of a bad range or domain, where it also registers the block.

### Block width and expiry

Two limits apply to every block this bot registers, from the query API and the
reactive webhook path alike:

| Setting | Default | Why |
|---|---|---|
| `IP_SCRUTINY_MAX_BLOCK_PREFIX_V4` / `_V6` | `24` / `48` | **Ceiling** — the widest network one block may cover. ipapi.is reports the flagged address's whole ASN route. Right for hosting space, wrong for an ISP aggregate — routes of `/8`–`/11` turn up routinely, and `197.211.32.0/19` was registered against a Benin **mobile carrier** off one `abuser` flag. Anything wider falls back to the floor width below. |
| `IP_SCRUTINY_MIN_BLOCK_PREFIX_V4` / `_V6` | `24` / `48` | **Floor** — the narrowest network one block may cover. A `/32` blocks the address we saw and nothing the same actor moves to next door, which is exactly how these ranges get used; anything narrower than this (missing route, `/32` fallback, small hosting allocation) is widened to the network of this width containing the address. At the default this equals the ceiling, so every v4 block is exactly the `/24`. What that costs an innocent neighbour depends on the two policies around it — at `sign_up_block` severity with a month-long `IP_SCRUTINY_IPBLOCK_EXPIRES_IN` (what yttrx runs) one flagged signup hard-refuses registration from 256 addresses for a month; `sign_up_requires_approval` or a shorter expiry buys that back. Set to `32`/`128` to restore blocking exactly what the route says. |
| `IP_SCRUTINY_IPBLOCK_EXPIRES_IN` | `86400` (24h) | An IP block claims who is behind an address *now*, and VPS addresses are recycled constantly. A day breaks a signup run — which is the shape this abuse takes — without the blocklist accumulating forever. Email domain blocks have no expiry in Mastodon and are unaffected. |

The cap applies to what gets **blocked**, not what gets **cached**: the wide
route is still the right cache key, since it genuinely belongs to one operator
and one lookup answers for the whole of it.

### Records

| Table | Holds |
|---|---|
| `reputation_query` | Audit log of every answer: what was asked, what the providers said, whether it led to a block. Domains only — never a full address. |
| `reputation_block` | One row per block installed here, keyed by the CIDR/domain (there is no account to key it to). Lets a repeat query skip the Mastodon round trip, and carries `expires_at` so the dedup record lapses exactly when the block does. |

Both feed `/api/blocks` and the reconcile pass alongside the signup-derived
rows, so neither under-reports as blocking shifts to the pre-signup path. The
published entry shape is unchanged; `signups` still counts signups, so a range
blocked before anyone signed up from it reads `"signups": 0`.

### Watching it

`welcomebot-reputation` (installed at `/root/bin` on admin.yttrx.com) reads
both tables directly, so it survives log rotation:

```bash
welcomebot-reputation                    # last 20 answers
welcomebot-reputation --stats            # volume, flag/block rate, by source
welcomebot-reputation --blocks           # what it currently has blocked
welcomebot-reputation --blocked --since '-24 hours'
welcomebot-reputation --deferred         # did the hourly budget ceiling bite?
welcomebot-reputation --ip | --email     # one side only
```

`--blocks` marks a row `GONE` once the reconcile pass finds the block no
longer on the Mastodon side (moderator cleared it, or it expired), so the
difference between "we installed this" and "this is in force" stays visible.

For the raw log lines rather than the tables, `welcomebot-logs --reputation`
filters to pre-blocks, budget refusals and width-cap narrowings:

```bash
welcomebot-logs --reputation -f          # follow pre-signup blocking live
welcomebot-logs --reputation --since 24h
```

On the Mastodon side the patch logs one line per pre-block at `info`
(`WelcomebotReputation: pre-blocked signup attempt`), plus a warning for every
failed or timed-out lookup — `docker logs live-web-1 | grep WelcomebotReputation`
on mammut.

## Layout

| Path | Purpose |
|---|---|
| `app/main.py` | The FastAPI app |
| `app/suspicious_sweep.py` | Scheduled sweep: suspends flagged signups with no activity (run via cron, not the webserver) |
| `app/reconcile_blocks.py` | Scheduled pass: marks blocks that no longer exist in Mastodon inactive (cron) |
| `Dockerfile`, `docker-compose.yml` | Container build + run |
| `.env.example` | Config template (copy to `.env`, never commit) |
| `nginx/hooks.yttrx.com.conf` | nginx site for admin.yttrx.com (webhook, healthz, blocklist API, reputation API) |
| `bin/welcomebot-logs`, `bin/welcomebot-signups`, `bin/welcomebot-suspicious`, `bin/welcomebot-reputation` | Ops CLIs installed to `/root/bin` on admin.yttrx.com |
| `bin/yttrx-blocklist-import` | Importer other admins can run to pull this list into their own instance |
| `bin/backfill-block-cidr.py` | One-shot: reconstruct `block_cidr` for rows predating it (run 2026-09-13) |
| `test_local.py` | Offline smoke test (no network) |

## Configuration

Copy `.env.example` to `.env` and fill in:

| Var | What |
|---|---|
| `WEBHOOK_SECRET` | The secret Mastodon shows when you create the webhook |
| `BOT_ACCESS_TOKEN` | Access token of the bot account (scope `write:statuses`) |
| `MASTODON_BASE_URL` | `https://yttrx.com` |
| `WELCOME_MESSAGE` | Template; `{acct}` → new user's handle |
| `LOCAL_ONLY` | `true` — only welcome accounts local to yttrx |
| `DB_PATH` | `/data/welcomed.db` (matches the compose volume) |
| `ABUSE_BOT_TOKEN` | Moderator bot token (admin scopes); blank disables abuse handling |
| `ABUSE_ENABLED` | Master switch for report handling |
| `ABUSE_DRY_RUN` | `true` — log/DM only, take no action (rollout safety) |
| `ABUSE_ACTION` | `silence` (default, reversible) or `suspend` |
| `ABUSE_LOCAL_ONLY` | `true` — only auto-act on local accounts |
| `ABUSE_YOUNG_MAX_DAYS` / `ABUSE_DORMANT_MIN_DAYS` | Tier windows (30 / 30) |
| `ABUSE_SOURCES_NEWDORMANT` / `ABUSE_SOURCES_ACTIVE` | Distinct-reporter thresholds (2 / 3) |
| `ABUSE_MAX_STATUS_PAGES` | Backward status-scan cap (5 × 40 statuses) |
| `ABUSE_SKIP_PRIVILEGED` | `true` — never auto-act on staff (Admin/Owner/Moderator), via the payload role |
| `ABUSE_ALLOWLIST` | Extra handles never to auto-act on (comma-separated backstop) |
| `MOD_ALERT_ACCT` | Handle to DM after an auto-action; blank disables the DM |
| `ABUSE_HELP_URL` | Appeals/help page the silenced user is linked to |
| `ABUSE_USER_DM` | Template DMed to the silenced user (`{acct}`, `{help_url}`); blank disables |
| `IP_SCRUTINY_ENABLED` | Master switch for IP-based signup scrutiny |
| `IP_SCRUTINY_IPAPI_KEY` | Optional ipapi.is API key for higher rate limits; blank uses the anonymous 1,000 req/day tier |
| `IP_SCRUTINY_DRY_RUN` | `true` — classify + DM only, no held welcome, no ip_block write |
| `IP_SCRUTINY_HOLD_WELCOME` | `true` — hold the welcome for a flagged signup until `account.approved` |
| `IP_SCRUTINY_ABUSE_THRESHOLD` | Distinct-reporter threshold used (if lower) for accounts with a flagged signup IP |
| `IP_SCRUTINY_AUTO_IPBLOCK` | Auto-register a flagged signup's network (ipapi.is route clamped to the width window, or the `/24`/`/48` containing the address if there's no usable route) into Mastodon's `Admin::IpBlock` |
| `IP_SCRUTINY_IPBLOCK_SEVERITY` | `sign_up_block` (default) — hard reject, no queue; other options: `sign_up_requires_approval` (soft, moderator queue), `no_access` |
| `IP_SCRUTINY_STRONG_SUSPEND_ENABLED` | Master switch — immediately suspend a signup whose IP is flagged `datacenter`/`proxy`/`tor`/`abuser` (not `vpn` alone), regardless of the email signal |
| `IP_SCRUTINY_STRONG_SUSPEND_ACTION` | `suspend` (default) or `silence`, applied immediately on a strong IP signal |
| `IP_SCRUTINY_STRONG_SUSPEND_DRY_RUN` | `true` (ships dry-run-first) — set `false` to act for real once the moderator DMs look right |
| `CHECK_MAIL_ENABLED` | Master switch for disposable/high-risk email signup scrutiny |
| `CHECK_MAIL_API_KEY` | check-mail.org API key; blank disables the check |
| `CHECK_MAIL_DRY_RUN` | `true` — classify + DM only, no held welcome, no email_domain_block write, no report-triggered suspend |
| `CHECK_MAIL_RISK_THRESHOLD` | Flag if `is_disposable` or `risk` ≥ this (default 80) |
| `CHECK_MAIL_HOLD_WELCOME` | `true` — hold the welcome for a flagged signup until `account.approved` |
| `CHECK_MAIL_AUTO_DOMAIN_BLOCK` | Auto-register a flagged domain into Mastodon's `Admin::EmailDomainBlock`; also gates the report-triggered override |
| `SUSPICIOUS_SWEEP_ENABLED` | Master switch for the suspicious-signup sweep (`app/suspicious_sweep.py`) |
| `SUSPICIOUS_GRACE_HOURS` | Hours a flagged signup has to post or follow someone before it's swept (default 1) |
| `SUSPICIOUS_ACTION` | `suspend` (default) or `silence`, applied to a flagged signup with zero activity in the grace window |
| `SUSPICIOUS_DRY_RUN` | `false` (ships live by design) — set `true` to log/DM only, no real action |

## Local test

```bash
python3 -m venv .venv && . .venv/bin/activate
pip install -r requirements.txt
WEBHOOK_SECRET=testsecret BOT_ACCESS_TOKEN=x python test_local.py   # -> ALL TESTS PASSED
```

---

## Deploy runbook

> Every step that changes the running yttrx system must be logged in
> `yttrx-documentation/changelog.md` (auto-publishes on `git push origin main`).

### 1. Create the bot account + token (on yttrx, via the web UI)

1. Register/choose a bot account, e.g. `@welcome` (set "This is a bot account"
   in Preferences → Profile).
2. Preferences → **Development** → **New application**.
   - Scopes: **`write:statuses`** (untick the rest).
   - Save, open it, copy **"Your access token"** → `BOT_ACCESS_TOKEN` in `.env`.

### 1b. Create the moderator bot account + token (for the abuse bot)

The abuse bot silences accounts, which needs **admin** privileges — keep this
off the public welcome bot.

1. Register a dedicated account, e.g. `@modbot` (mark it a bot account).
2. As an admin, give it a **role** that includes the **Manage Users** and
   **Manage Reports** permissions (Administration → Roles, then assign it on
   the account). Without these the API calls 403.
3. As `@modbot`: Preferences → **Development** → **New application** with
   scopes **`admin:write:accounts`**, **`admin:read:reports`**,
   **`read:statuses`**, **`write:statuses`**. Copy its access token →
   `ABUSE_BOT_TOKEN` in `.env`.
4. Set `MOD_ALERT_ACCT` to the handle that should receive review DMs (e.g.
   `pete`). Keep `ABUSE_DRY_RUN=true` for the initial rollout.

### 2. Build + run the container (on admin.yttrx.com)

```bash
# copy this project to admin (rsync/scp/git clone), then:
cd ~/yttrx-welcomebot          # wherever it lands on admin
cp .env.example .env           # fill in BOT_ACCESS_TOKEN now; WEBHOOK_SECRET in step 4
docker compose up -d --build
curl -s localhost:8087/healthz # -> {"ok":true}
```

The container listens on `127.0.0.1:8087` only.

### 3. nginx + TLS for hooks.yttrx.com (on admin.yttrx.com)

DNS: point `hooks.yttrx.com` (A/AAAA, or proxied via Cloudflare) at admin first.

```bash
sudo cp nginx/hooks.yttrx.com.conf /etc/nginx/sites-available/hooks

# issue the cert (standalone — same pattern as the other admin sites)
sudo systemctl stop nginx
sudo certbot certonly --standalone -d hooks.yttrx.com
sudo systemctl start nginx

sudo ln -s ../sites-available/hooks /etc/nginx/sites-enabled/hooks
sudo nginx -t && sudo systemctl reload nginx
curl -s https://hooks.yttrx.com/healthz   # -> {"ok":true}
```

Renewal piggybacks on the existing `0 2 * * * certbot renew --nginx` cron.

### 4. Create the webhook in Mastodon (on yttrx, admin UI)

Administration → **Webhooks** → **New webhook**:

- **URL:** `https://hooks.yttrx.com/webhook`
- **Events:** check **`account.created`**, **`account.approved`** *and*
  **`report.created`** (one webhook, all events → one shared secret, one
  endpoint). Both account events are handled so the welcome fires whether or
  not registration approval is enabled; the dedup store welcomes once.
- Save, then copy the generated **secret** → `WEBHOOK_SECRET` in `.env` on
  admin, and `docker compose up -d` to restart with it.
- Use the webhook's **"Send test"** / re-enable it; confirm a `200` in
  `docker compose logs -f welcomebot`.

> If you'd rather roll the abuse bot out separately, create a *second* webhook
> for `report.created` only — but then it has its **own** secret, so you'd need
> a second endpoint/secret. Subscribing one webhook to both events is simpler.

> ⚠️ Verify the exact event name and that the signing header is
> `X-Hub-Signature` against this instance (v4.5.11) when you wire it up —
> the admin UI lists the available events, and the server logs the header it
> receives. Adjust `app/main.py` if your version differs.

### 5. Smoke test end to end

Register a throwaway test account on yttrx and confirm the `@welcome` bot DMs
it. Then delete the test account (`tootctl accounts delete` on mammut) and the
test toot.

### 6. Cron for the suspicious-signup sweep (on admin.yttrx.com)

The sweep is not part of the always-running webhook server — it's invoked
on a schedule via `docker exec`. `ABUSE_BOT_TOKEN` must additionally carry
the `admin:read:accounts` scope (remint via the Rails-console snippet in
`CLAUDE.md`, same procedure as the other scope additions) before installing
this cron entry, or every sweep run 403s per-account:

```bash
crontab -l | { cat; echo '*/10 * * * * docker exec yttrx-welcomebot python -m app.suspicious_sweep >> /var/log/welcomebot-suspicious-sweep.log 2>&1'; } | crontab -
```

Every 10 minutes (tightened from hourly on 2026-07-06 to match
`SUSPICIOUS_GRACE_HOURS` dropping from 24 to 1) — with a 1h grace window an
hourly cadence could add up to another hour of latency on top of the grace
period itself; a 10-minute cadence keeps the gap between crossing the
deadline and being swept small relative to the window.

## Operations

```bash
welcomebot-logs -f --abuse             # convenience CLI on admin (see bin/welcomebot-logs)
welcomebot-signups --flagged           # signup-IP classification history (see bin/welcomebot-signups)
welcomebot-suspicious --pending        # suspicious-sweep watch list (see bin/welcomebot-suspicious)
docker exec yttrx-welcomebot python -m app.suspicious_sweep   # run the sweep manually
docker compose logs -f welcomebot      # watch deliveries
docker compose restart welcomebot      # after editing .env
docker compose down && docker compose up -d --build   # redeploy after code change
```

Dedup store: `welcomebot-data` volume → `/data/welcomed.db`. To re-welcome
someone (e.g. after a failed send that got marked), delete their row:

```bash
docker compose exec welcomebot \
  python -c "import sqlite3; sqlite3.connect('/data/welcomed.db').execute(\
  'DELETE FROM welcomed WHERE acct=?', ('alice',)).connection.commit()"
```

## Rollback

```bash
# stop the bot
cd ~/yttrx-welcomebot && docker compose down
# disable nginx site
sudo rm /etc/nginx/sites-enabled/hooks && sudo nginx -t && sudo systemctl reload nginx
# in Mastodon admin UI: disable or delete the account.created webhook
```