pmb/yttrx-welcomebot
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add a reputation query API for pre-signup blocking
docker-build-push / build-push (push) Successful in 6s
Details

Browse Source 
Everything the bot does today is reactive: Mastodon creates the account,
the account.created webhook arrives, we classify the signup IP and email
domain, and then we block the range and suspend what already exists.

POST /api/reputation is the pre-emptive half. A caller asks about an IP
and/or email domain; we classify it, record the answer, register the
matching Admin::IpBlock / EmailDomainBlock, and only then return. A
caller that runs its own block lookup immediately afterwards therefore
finds a block that did not exist when its request started. The intended
caller is Mastodon's signup path, which does exactly that.

It writes, so unlike /api/blocks it is authenticated and refuses to serve
without a key (REPUTATION_API_KEY): an unauthenticated endpoint that
installs IP blocks is a denial-of-service lever, so there is no open mode.

Notes on the pieces that aren't obvious:

- REPUTATION_API_MAX_LOOKUPS_PER_HOUR caps *fresh* provider lookups. The
  caller hits this on every view of the signup form, not just on
  submission, so without a ceiling a crawler walking /auth/sign_up from a
  new range each time would set our ipapi.is bill. Over budget we answer
  "unknown" and install nothing -- never "flagged". Cache hits are free
  and uncounted, and the webhook path never spends from this budget.

- Blocks installed here are recorded in their own table, keyed by the
  CIDR/domain rather than by account (there is no account yet), and are
  folded into /api/blocks and the reconcile pass so neither one
  under-reports as blocking shifts to the pre-signup path. The published
  entry shape is unchanged; a range blocked before anyone signed up from
  it simply reads "signups": 0.

- A repeat query for an already-installed block skips the Mastodon round
  trip. This runs on a live signup request, so that is saved latency.
This commit is contained in:
pmb committed 2026-09-13 16:02:26 -07:00
1 parent 8490048d2d
commit 57234ca548
5 files changed
+758 -4
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

app/main.py
+431 -4
View File
@@ -34,6 +34,16 @@ Alongside the webhook, an optional read-only blocklist API publishes what the
bot has auto-blocked — ``GET /api/blocks``, ``/api/blocks/ip-ranges``,
``/api/blocks/email-domains``. It is unauthenticated and therefore off unless
``BLOCKS_API_ENABLED`` is set; see that config entry.
Everything above is *reactive*: it acts on an account Mastodon has already
created. ``POST /api/reputation`` is the pre-emptive counterpart — Mastodon
asks about an IP/email domain during signup, before creating anything, and
this classifies it, records it, and registers the Admin::IpBlock /
EmailDomainBlock *before returning*, so the lookup Mastodon runs immediately
afterwards rejects the signup outright. It writes, so it is authenticated
(``X-Reputation-Key``) and off unless ``REPUTATION_API_ENABLED`` and
``REPUTATION_API_KEY`` are both set. The caller is the
``call-welcomebot-reputation-api`` patch on yttrx's Mastodon fork.
"""
from __future__ import annotations
@@ -45,12 +55,14 @@ import logging
import os
import sqlite3
import threading
import time
from contextlib import contextmanager
from datetime import datetime, timedelta, timezone
import httpx
from fastapi import BackgroundTasks, FastAPI, Header, HTTPException, Request
from fastapi.responses import JSONResponse, PlainTextResponse
from pydantic import BaseModel
logging.basicConfig(
level=logging.INFO,
@@ -248,6 +260,54 @@ BLOCKS_API_PUBLIC_URL = os.environ.get(
# intermediate cache hold it for a few minutes costs nothing.
BLOCKS_API_CACHE_SECONDS = int(os.environ.get("BLOCKS_API_CACHE_SECONDS", "300"))
# --- Reputation query API (pre-signup pre-blocking) -------------------------
# POST /api/reputation lets Mastodon ask "what do you know about this IP /
# email domain?" *before* it runs its own IpBlock / EmailDomainBlock lookup at
# signup time. The handler classifies, records the answer, registers the block
# on the Mastodon side, and only then returns — so the lookup Mastodon makes
# immediately afterwards finds the block this call just installed. That turns
# the bot from purely reactive (suspend after the account exists) into
# pre-emptive: a bad range is rejected on first contact rather than cleaned up
# afterwards.
#
# Unlike the blocklist API this endpoint *writes* (it installs blocks), so it
# is authenticated and refuses to serve at all without a key. See the
# yttrx patch `call-welcomebot-reputation-api` on the Mastodon fork for the
# caller.
REPUTATION_API_ENABLED = os.environ.get(
"REPUTATION_API_ENABLED", "false").lower() in ("1", "true", "yes")
# Shared secret the caller presents as `X-Reputation-Key`. No key, no service:
# an unauthenticated endpoint that installs IP blocks is a denial-of-service
# lever, so an empty key disables the route entirely (404) rather than
# defaulting to open.
REPUTATION_API_KEY = os.environ.get("REPUTATION_API_KEY", "")
# Whether a flagged lookup actually registers the block on the Mastodon side.
# Off = classify, record and answer only, installing nothing — the dry-run
# equivalent for this path, and the way to watch what it *would* block before
# letting it act. The per-signal IP_SCRUTINY_DRY_RUN / CHECK_MAIL_DRY_RUN and
# IP_SCRUTINY_AUTO_IPBLOCK / CHECK_MAIL_AUTO_DOMAIN_BLOCK switches still apply
# on top of this, so a signal that is dry-run for the webhook path is dry-run
# here too.
REPUTATION_API_INSTALL_BLOCKS = os.environ.get(
"REPUTATION_API_INSTALL_BLOCKS", "true").lower() in ("1", "true", "yes")
# Ceiling on *fresh* (cache-missing) external provider lookups this endpoint
# may make per rolling hour. Mastodon calls it on every view of the signup
# form, not just on submission, so a crawl of /auth/sign_up from many distinct
# ranges would otherwise burn paid ipapi.is / check-mail.org quota at whatever
# rate the crawler chooses. Over budget, an uncached value answers "unknown"
# (never "flagged") and nothing is installed. Cache hits are free and never
# counted; 0 disables the ceiling.
#
# The webhook path deliberately does NOT consult this budget — account.created
# is low-volume, authoritative, and must never be starved by crawler traffic.
REPUTATION_API_MAX_LOOKUPS_PER_HOUR = int(
os.environ.get("REPUTATION_API_MAX_LOOKUPS_PER_HOUR", "500"))
# DM MOD_ALERT_ACCT when this endpoint installs a block it has never installed
# before. Bounded by how many genuinely new bad ranges/domains show up, not by
# request volume — repeat hits on an already-installed block are silent.
REPUTATION_API_DM_MODERATOR = os.environ.get(
"REPUTATION_API_DM_MODERATOR", "true").lower() in ("1", "true", "yes")
if not WEBHOOK_SECRET:
log.warning("WEBHOOK_SECRET is empty — signature verification will reject all requests.")
if not BOT_ACCESS_TOKEN:
@@ -256,6 +316,8 @@ if ABUSE_ENABLED and not ABUSE_BOT_TOKEN:
log.warning("ABUSE_BOT_TOKEN is empty — report.created handling is disabled.")
if CHECK_MAIL_ENABLED and not CHECK_MAIL_API_KEY:
log.warning("CHECK_MAIL_API_KEY is empty — disposable-email signup scrutiny is disabled.")
if REPUTATION_API_ENABLED and not REPUTATION_API_KEY:
log.warning("REPUTATION_API_KEY is empty — the reputation query API stays disabled.")
app = FastAPI(title="yttrx welcome-bot + abuse-bot")
@@ -368,6 +430,49 @@ def _init_db() -> None:
" cached_at TEXT DEFAULT CURRENT_TIMESTAMP"
")"
)
# Audit log of every /api/reputation answer: what was asked about,
# what the providers said, and whether it led to a block. The webhook
# path has signup_ip/signup_email for this; a pre-signup query has no
# account to hang a row off, so it gets its own table. Never stores a
# full email address — the domain only, same rule as signup_email.
conn.execute(
"CREATE TABLE IF NOT EXISTS reputation_query ("
" id INTEGER PRIMARY KEY AUTOINCREMENT,"
" kind TEXT NOT NULL," # 'ip' | 'email'
" value TEXT NOT NULL," # the IP, or the email domain
" classification TEXT,"
" org TEXT,"
" risk INTEGER,"
" flagged INTEGER,"
" target TEXT," # cidr / domain the block covers
" blocked INTEGER DEFAULT 0,"
" source TEXT," # caller-supplied tag
" queried_at TEXT DEFAULT CURRENT_TIMESTAMP"
")"
)
conn.execute(
"CREATE INDEX IF NOT EXISTS reputation_query_kind_value "
"ON reputation_query (kind, value)"
)
# One row per block installed by the query API, keyed by what is
# actually blocked. Two jobs: skip re-POSTing a block we already
# installed (Mastodon answers 422, but the round trip is on the signup
# request path), and give /api/blocks + the reconcile pass the same
# view of these as of the signup-derived ones.
conn.execute(
"CREATE TABLE IF NOT EXISTS reputation_block ("
" kind TEXT NOT NULL," # 'ip' | 'email'
" target TEXT NOT NULL," # cidr, or domain
" classification TEXT,"
" org TEXT,"
" queries INTEGER DEFAULT 1,"
" first_blocked_at TEXT DEFAULT CURRENT_TIMESTAMP,"
" last_seen_at TEXT DEFAULT CURRENT_TIMESTAMP,"
" block_active INTEGER DEFAULT 1,"
" block_checked_at TEXT,"
" PRIMARY KEY (kind, target)"
")"
)
conn.execute(
"CREATE TABLE IF NOT EXISTS suspicious_watch ("
" account_id TEXT PRIMARY KEY,"
@@ -592,6 +697,63 @@ def start_suspicious_watch(account_id: str, acct: str, reasons: str,
)
def record_reputation_query(kind: str, value: str, classification: str, org: str,
risk: int, flagged: bool, target: str, blocked: bool,
source: str) -> None:
"""Append one /api/reputation answer to the audit log."""
with _db() as conn:
conn.execute(
"INSERT INTO reputation_query "
"(kind, value, classification, org, risk, flagged, target, blocked, source) "
"VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)",
(kind, value, classification, org, int(risk), int(flagged),
target or None, int(blocked), source or None),
)
def reputation_block_installed(kind: str, target: str) -> bool:
"""True if the query API already installed this block and the reconcile
pass hasn't since found it gone. Lets a repeat query skip the Mastodon
round trip — this runs on the signup request path, so a saved POST is
saved latency."""
with _db() as conn:
row = conn.execute(
"SELECT 1 FROM reputation_block "
"WHERE kind = ? AND target = ? AND block_active = 1",
(kind, target),
).fetchone()
return row is not None
def record_reputation_block(kind: str, target: str, classification: str, org: str) -> None:
"""Record that this block is installed, or that an installed one was hit
again. Reactivates a row the reconcile pass had marked gone, same as
mark_ipblock_registered does for the signup-derived rows."""
with _db() as conn:
conn.execute(
"INSERT INTO reputation_block "
"(kind, target, classification, org) VALUES (?, ?, ?, ?) "
"ON CONFLICT(kind, target) DO UPDATE SET "
" queries = queries + 1,"
" last_seen_at = CURRENT_TIMESTAMP,"
" block_active = 1,"
" classification = COALESCE(excluded.classification, classification),"
" org = COALESCE(excluded.org, org)",
(kind, target, classification or None, org or None),
)
def reputation_blocks(kind: str) -> list[tuple[str, str, str]]:
"""(target, first_blocked_at, last_seen_at) for every still-active block
the query API installed, newest first."""
with _db() as conn:
return conn.execute(
"SELECT target, first_blocked_at, last_seen_at FROM reputation_block "
"WHERE kind = ? AND block_active = 1 ORDER BY last_seen_at DESC",
(kind,),
).fetchall()
def _fallback_cidr(ip: str) -> str:
"""The single-address CIDR for ip, used for signup_ip rows written before
block_cidr existed — that's the block that was registered for them."""
@@ -644,7 +806,8 @@ def blocked_ip_ranges(limit: int = 0) -> list[dict]:
entry["signups"] += 1
entry["first_blocked_at"] = classified_at
return _capped(list(grouped.values()), limit)
_merge_reputation_blocks(grouped, "ip", "cidr")
return _capped(_newest_first(grouped), limit)
def blocked_email_domains(limit: int = 0) -> list[dict]:
@@ -682,7 +845,44 @@ def blocked_email_domains(limit: int = 0) -> list[dict]:
entry["signups"] += 1
entry["first_blocked_at"] = classified_at
return _capped(list(grouped.values()), limit)
_merge_reputation_blocks(grouped, "email", "domain")
return _capped(_newest_first(grouped), limit)
def _merge_reputation_blocks(grouped: dict[str, dict], kind: str, key: str) -> None:
"""Fold the query API's blocks into a signup-derived blocklist, in place.
Both paths register the same kind of Admin::IpBlock / EmailDomainBlock, so
the published list would silently under-report as more blocking moves to
the pre-signup path. Entries keep the existing keys — no contract change
for anyone already consuming this — which means `signups` continues to
count *signups* specifically. A range blocked before anyone signed up from
it therefore reads `"signups": 0`, which is exactly right: blocked, and
nothing got through.
"""
for target, first_at, last_at in reputation_blocks(kind):
if not target:
continue
entry = grouped.get(target)
if entry is None:
grouped[target] = {
key: target,
"signups": 0,
"first_blocked_at": first_at,
"last_blocked_at": last_at,
}
else:
entry["first_blocked_at"] = min(
filter(None, (entry["first_blocked_at"], first_at)), default=None)
entry["last_blocked_at"] = max(
filter(None, (entry["last_blocked_at"], last_at)), default=None)
def _newest_first(grouped: dict[str, dict]) -> list[dict]:
"""Most recently blocked first. The signup rows arrive in that order
already, but merged-in query-API entries have their own timestamps, so the
combined list has to be re-sorted to keep the documented ordering true."""
return sorted(grouped.values(), key=lambda e: e["last_blocked_at"] or "", reverse=True)
def _capped(items: list[dict], limit: int) -> list[dict]:
@@ -1148,7 +1348,8 @@ def classify_signup_ip(ip: str) -> tuple[str, str, bool, str]:
return classification, intel["org"], bool(reasons), intel.get("cidr") or f"{ip}/32"
def register_ip_block(ip: str, acct: str, org: str, cidr: str = "") -> bool:
def register_ip_block(ip: str, acct: str, org: str, cidr: str = "",
comment: str = "") -> bool:
"""Register a flagged signup's network in Mastodon's native Admin::IpBlock.
cidr should be classify_signup_ip's block_cidr — the ASN/route-level
@@ -1157,6 +1358,10 @@ def register_ip_block(ip: str, acct: str, org: str, cidr: str = "") -> bool:
Falls back to a single-address block if no cidr is given or ip itself
doesn't parse (defensive only; classify_signup_ip always supplies one).
comment overrides the moderator-facing note stored with the block; the
reputation query API passes its own, since a pre-signup block has no
account to name.
Returns True if the range is blocked on the Mastodon side afterwards
(including the 422 "already blocked" case), so the caller only records it
as blocked when it really is — /api/blocks publishes those rows.
@@ -1168,7 +1373,8 @@ def register_ip_block(ip: str, acct: str, org: str, cidr: str = "") -> bool:
log.warning("skipping ip_block registration for unparseable ip=%r (acct=%s)", ip, acct)
return False
cidr = f"{ip}/{prefix_len}"
comment = f"welcomebot: flagged signup @{acct} ({org or 'unknown org'})"[:200]
comment = (comment or
f"welcomebot: flagged signup @{acct} ({org or 'unknown org'})")[:200]
try:
resp = httpx.post(
f"{MASTODON_BASE_URL}/api/v1/admin/ip_blocks",
@@ -1186,6 +1392,181 @@ def register_ip_block(ip: str, acct: str, org: str, cidr: str = "") -> bool:
return True
# --- Reputation query API (pre-signup pre-blocking) -------------------------
# The pre-emptive half of the bot: Mastodon asks about an IP/email *before*
# creating anything, and this installs the block in time for Mastodon's own
# lookup to find it. Everything here runs on a live signup request, so it is
# written to be fast on the common path (sqlite cache hit) and to fail open —
# never to make a signup wait on, or fail because of, a provider outage.
_lookup_budget_lock = threading.Lock()
_lookup_budget: list[float] = []
def _external_lookup_allowed() -> bool:
"""Claim one slot from the rolling hourly budget for a fresh provider
lookup. Returns False when the budget is spent, in which case the caller
must answer "unknown" rather than making the call.
Only the query API spends budget — see REPUTATION_API_MAX_LOOKUPS_PER_HOUR.
Mastodon calls it on every *view* of the signup form, so without a ceiling
a crawler walking /auth/sign_up from a fresh range each time sets our paid
provider spend.
"""
if REPUTATION_API_MAX_LOOKUPS_PER_HOUR <= 0:
return True
now = time.monotonic()
with _lookup_budget_lock:
cutoff = now - 3600
while _lookup_budget and _lookup_budget[0] < cutoff:
_lookup_budget.pop(0)
if len(_lookup_budget) >= REPUTATION_API_MAX_LOOKUPS_PER_HOUR:
return False
_lookup_budget.append(now)
return True
def _reputation_ip(ip: str, source: str) -> dict | None:
"""Classify one IP for the query API and install the block if it's flagged.
Returns None when the IP side is switched off or the address doesn't parse.
"""
if not (IP_SCRUTINY_ENABLED and ip):
return None
try:
ipaddress.ip_address(ip)
except ValueError:
log.warning("reputation query: unparseable ip=%r", ip)
return None
cached = cached_ip_intel(ip) is not None
if not cached and not _external_lookup_allowed():
# Budget spent. Answer "unknown", never "flagged" — the whole point of
# the ceiling is to stop paying, and guessing in either direction here
# would either block innocents or waste the guard.
log.warning("reputation query: hourly lookup budget spent, ip=%s answered unknown", ip)
bounds = _ip_range_bounds(ip)
return {
"ip": ip, "classification": "unknown", "org": "", "flagged": False,
"cidr": bounds[3] if bounds else "", "blocked": False,
"cached": False, "deferred": True,
}
classification, org, flagged, cidr = classify_signup_ip(ip)
blocked = False
if flagged:
installable = (REPUTATION_API_INSTALL_BLOCKS and IP_SCRUTINY_AUTO_IPBLOCK
and not IP_SCRUTINY_DRY_RUN)
if not installable:
log.info("reputation query: would block ip=%s cidr=%s (%s) — installing is off",
ip, cidr, classification)
elif reputation_block_installed("ip", cidr):
# Already ours and still in force; Mastodon's own lookup will find
# it. Skip the round trip, just count the hit.
record_reputation_block("ip", cidr, classification, org)
blocked = True
elif register_ip_block(ip, "", org, cidr, comment=(
f"welcomebot: pre-signup block, {classification} "
f"({org or 'unknown org'})")):
record_reputation_block("ip", cidr, classification, org)
blocked = True
log.warning("reputation query: pre-blocked ip=%s cidr=%s classification=%s org=%s source=%s",
ip, cidr, classification, org, source or "-")
if REPUTATION_API_DM_MODERATOR:
dm_moderator(
f"🛡️ Pre-blocked {cidr} before signup — {classification} "
f"({org or 'unknown org'})."
)
record_reputation_query("ip", ip, classification, org, 0, flagged, cidr, blocked, source)
return {
"ip": ip, "classification": classification, "org": org, "flagged": flagged,
"cidr": cidr, "blocked": blocked, "cached": cached, "deferred": False,
}
def _reputation_email(email: str, source: str) -> dict | None:
"""Classify one signup email's *domain* for the query API and install the
domain block if it's flagged. The full address is never stored or logged —
same rule as the webhook path (see classify_email_domain).
Returns None when the email side is switched off or there's no domain.
"""
if not (CHECK_MAIL_ENABLED and CHECK_MAIL_API_KEY and email and "@" in email):
return None
domain = email.rsplit("@", 1)[-1].strip().lower()
if not domain:
return None
cached = cached_check_mail(domain) is not None
if not cached and not _external_lookup_allowed():
log.warning("reputation query: hourly lookup budget spent, domain=%s answered unknown",
domain)
return {
"domain": domain, "disposable": False, "risk": 0, "flagged": False,
"blocked": False, "cached": False, "deferred": True,
}
is_disposable, risk = classify_email_domain(domain)
flagged = is_disposable or risk >= CHECK_MAIL_RISK_THRESHOLD
blocked = False
if flagged:
installable = (REPUTATION_API_INSTALL_BLOCKS and CHECK_MAIL_AUTO_DOMAIN_BLOCK
and not CHECK_MAIL_DRY_RUN)
if not installable:
log.info("reputation query: would block domain=%s (disposable=%s risk=%d) "
"— installing is off", domain, is_disposable, risk)
elif reputation_block_installed("email", domain):
record_reputation_block("email", domain, "", "")
blocked = True
elif register_email_domain_block(domain, "(pre-signup)"):
record_reputation_block("email", domain, "", "")
blocked = True
log.warning("reputation query: pre-blocked domain=%s disposable=%s risk=%d source=%s",
domain, is_disposable, risk, source or "-")
if REPUTATION_API_DM_MODERATOR:
dm_moderator(
f"🛡️ Pre-blocked email domain {domain} before signup "
f"(disposable={is_disposable}, risk={risk})."
)
classification = "disposable" if is_disposable else ("high-risk" if flagged else "clean")
record_reputation_query("email", domain, classification, "", risk, flagged,
domain, blocked, source)
return {
"domain": domain, "disposable": is_disposable, "risk": risk, "flagged": flagged,
"blocked": blocked, "cached": cached, "deferred": False,
}
def reputation_check(ip: str = "", email: str = "", source: str = "") -> dict:
"""Classify an IP and/or email domain, install any block they earn, and
report what happened. The write half of the bot's pre-emptive path.
Each side is independent: either may be absent from the request, switched
off by config, or fail its provider lookup, without affecting the other.
A provider failure classifies "unknown"/not-flagged (classify_signup_ip and
classify_email_domain both swallow their own errors) — we never block on
our own lookup errors.
`blocked` in the response means "something is blocked on the Mastodon side
as a result of this call", which is what the caller acts on: it is true
both for a block installed just now and for one we installed earlier that
is still in force.
"""
result = {
"ip": _reputation_ip(ip.strip(), source),
"email": _reputation_email(email.strip(), source),
"checked_at": _now_iso(),
}
sides = [side for side in (result["ip"], result["email"]) if side]
result["flagged"] = any(side["flagged"] for side in sides)
result["blocked"] = any(side["blocked"] for side in sides)
return result
def count_distinct_reporters(target_id: str) -> int:
"""Count distinct accounts with an OPEN report against the target.
@@ -1507,6 +1888,52 @@ def api_blocks_email_domains(limit: int = 0, format: str = "json"):
})
# --- Reputation query API ---
# POST /api/reputation — authenticated, and it WRITES (installs blocks), which
# is what separates it from the read-only /api/blocks above. Mastodon calls it
# from the signup path immediately before its own IpBlock/EmailDomainBlock
# lookup; see the `call-welcomebot-reputation-api` patch on the fork.
class ReputationRequest(BaseModel):
"""What Mastodon knows at the moment it asks. Both fields are optional —
the IP-block choke point has no email address to offer, the email
validator has both."""
ip: str = ""
email: str = ""
source: str = ""
def _reputation_guard(key: str | None) -> None:
"""404 unless the endpoint is switched on AND has a key (a route that
installs blocks must never be reachable unauthenticated); 401 on a bad
key. Disabled is indistinguishable from absent, same as /api/blocks."""
if not (REPUTATION_API_ENABLED and REPUTATION_API_KEY):
raise HTTPException(status_code=404, detail="Not Found")
if not key or not hmac.compare_digest(key, REPUTATION_API_KEY):
raise HTTPException(status_code=401, detail="invalid key")
@app.post("/api/reputation")
def api_reputation(
body: ReputationRequest,
x_reputation_key: str | None = Header(default=None),
):
"""Look up an IP and/or email domain, record the answer, install whatever
block it earns, and return — in that order, so that a caller running its
own block lookup straight afterwards sees the block from this call.
Declared as a plain `def` on purpose: FastAPI runs those in a threadpool,
so the blocking provider calls underneath never tie up the event loop and
stall webhook deliveries.
"""
_reputation_guard(x_reputation_key)
if not body.ip and not body.email:
raise HTTPException(status_code=400, detail="ip or email is required")
return reputation_check(ip=body.ip, email=body.email, source=body.source)
@app.post("/webhook")
async def webhook(
request: Request,