M27 — Takedown tracking and abuse.ch submission

Status: proposal for review. Builds on M21 (collector, egress gate), M22 (reports), M25 (sending), M26 (outcomes). Everything is off by default: TAKEDOWN_ENABLED=false, and abuse.ch submission needs both ABUSECH_SUBMIT_ENABLED=true and the owner’s switch in Settings (mode off by default). CLAUDE.md §5 requires owner approval for automatic reporting: the owner approved abuse.ch submission, and this design keeps it an explicit switch in the UI. Placeholders only: documentation ranges, documentation ASNs, example.* names.

1. Goal and invariants

The project’s main goal is getting C2 servers and malware-distribution hosts taken down. M27 measures it. For each reported target it tracks whether the target is still up, how long it took to go down, and how each provider performs. M27 also adds an optional, controlled submission of captured malware URLs to URLhaus (and C2 addresses to ThreatFox).

#	Invariant	Enforced by
T1	No counter-scan, no hacking back. The only request to a target is one bounded HTTP request to a URL that already served malware to the collector. C2 servers and attacking addresses are never contacted, and neither is a URL hosted on one: http_eligible is false when the URL host is a sessions.src_ip seen in the last 30 days, a sample_links C2 address or a reported c2 / attack_source target (a Mirai loader on its own address). Ingest decides it when the target is made and again every sync cycle, so a host that starts attacking later is dropped; such targets keep only passive signals.	§2 table, http_eligible set only from fetch_targets with a stored sample, collector recheck selects kind = 'url' AND http_eligible only, hostile_hosts in ingest/ingest/takedown/sync.py, static test
T2	Re-checks run in the collector (the only service allowed to contact hostile hosts), through the egress client, with the M21 URL validator on the URL and every redirect hop.	collector/collector/recheck.py, reuse of validate.normalize_url / resolve_redirect
T3	A re-check stores no payload unless the hash changed.	§4.3
T4	Everything outbound goes through the VPN gate. Gate closed → no check, no quota spent, no row.	egress_client, EgressClosed handling
T5	Quotas are bounded: a daily HTTP re-check quota, a share of the abuse.ch and AbuseIPDB lookup quotas, a daily submission cap.	enrichment_quota providers takedown_http, takedown_intel, takedown_abuseipdb, abusech_submit
T6	Every submission body passes the identity guard. A failure sends nothing.	§7.5
T7	abuse.ch submission is off unless the environment flag and the owner’s UI switch are both on. Submissions are delivered at most once, and the table is their audit log.	§7
T8	Nothing is ever reported to AbuseIPDB. M27 only reads it, within quota.	§4.5, static test

2. What may be contacted

Action	Allowed	Why
One GET, bounded (size, time, redirects), to a URL that served a stored sample to the collector	Yes	Same request the collector already made; same identity and caps. A malware distribution URL is public by nature
HEAD instead of GET	No (not needed)	Many malware hosts mishandle HEAD; a bounded GET answers “still serving the same file?” in one request
GET to other paths on the same host (directory listing, /bins/)	No	Crawling is exploration
TCP connect, TLS handshake or banner grab on a C2 address or port	No	That is a probe of attacker infrastructure: a counter-scan
Any packet to an attacking IP (ping, connect, traceroute)	No	Counter-scan
Port scans, DNS brute force, reverse DNS of targets from the stack	No	Counter-scan; ingest cannot resolve names anyway (M21)
Intel lookups (URLhaus, ThreatFox, AbuseIPDB) about a target	Yes, within quota	Passive, to fixed intel APIs, already allowed by M13
Reading our own sensors’ data for an attacking IP	Yes	No traffic at all
Submitting a URL or ip:port to abuse.ch	Yes, opt-in (§7)	Owner-approved sharing

CLAUDE.md §5 gains one sentence, with the owner’s approval: “Takedown re-checks (M27) may send one bounded GET to a URL that already served malware to the collector; nothing else is ever contacted.”

3. Targets

3.1 Where targets come from

The ingest takedown worker syncs targets every cycle (TAKEDOWN_ENABLED, interval 600 s).

Source	Target kind / role	reported_at	Channel
Subject of a sent/closed report of kind attack_source	ip / attack_source	earliest sent_at of the reports that name it	mail
Subject of a payload_host report (an IP or a domain)	ip or domain / payload_host	same	mail
Each URL of that host in fetch_targets with status = 'fetched' and a sample (≤ 20 per host, latest first)	url / payload_url, parent_id = the host target	parent’s reported_at	mail
Subject of a c2 report	ip or domain / c2	same	mail
A submitted URLhaus submission (§7)	url / payload_url (with parent host target created without a report)	submitted_at	urlhaus
A submitted ThreatFox submission	ip / c2	submitted_at	threatfox

UNIQUE (kind, role, value). A target reported by several channels keeps the earliest reported_at. channels lists every channel. takedown_target_links records each report or submission.

• Demo reports and demo indicators produce demo targets only when ABUSE_REPORTS_INCLUDE_DEMO=true (CI, screenshots). Demo targets are never HTTP-checked or looked up.
• Tracking lasts TAKEDOWN_TRACK_DAYS (30) from reported_at (track_until). An admin can stop tracking a target early.

3.2 Provider of a target

provider_key is as<asn> when the ASN is known:

• from abuse_networks.asn of the report subject’s network_id;
• for URL children, the parent’s ASN;
• for a URL submitted with an IP-literal host, the GeoLite2 ASN of that IP.

For a domain with no hosting ASN (M22 reports domains to the registrar), it is recipient:<id>, the registrar’s abuse recipient. provider_name comes from abuse_networks.name, the GeoLite2 org or the recipient name. country is the network’s country.

4. Checks

4.1 Methods

Method	Runs in	For	Outcome codes
http	collector	kind = 'url' AND http_eligible	same, changed, changed_not_stored, up_unhashed, http_404, http_410, http_4xx, http_5xx, suspended_page, empty, unreachable, timeout, refused, redirect_limit, bad_redirect
urlhaus	ingest	URLs (url query) and hosts (host query)	online, offline, unknown_status, not_listed
threatfox	ingest	C2 ip/domain (search_ioc)	listed, not_listed
abuseipdb	ingest	attack_source IPs	reported_after (another reporter after our reported_at), quiet, not_listed
sightings	ingest (DB only)	attack_source IPs	seen (a non-benign session after reported_at in the last 72 h), not_seen
samples	ingest (DB only)	C2	new_sample (a sample first seen after reported_at links to this C2), none

Every check is one takedown_checks row. detail holds only constants and small numbers (e.g. {"sessions": 3}, {"last_online_age_h": 12}), never a server answer.

4.2 Schedule

HTTP_OFFSETS = (0, 1 h, 6 h, 24 h, 48 h) from reported_at. Then one check every 24 h until track_until. While the state is down, a check every 72 h catches a host that comes back. LISTING_OFFSETS = (0, 24 h, 7 d, 30 d) for urlhaus/threatfox/abuseipdb. sightings and samples are evaluated every worker cycle (cheap SQL). A target created more than one offset late (e.g. from an old report when M27 is first enabled) starts at the next offset in the future, after one immediate baseline check. Both schedules are pure functions (collector/collector/recheck.py next_http_at(step, reported_at, state, now); ingest/ingest/takedown/schedule.py), tested on fixed clocks. At most about 35 HTTP checks per URL over 30 days.

4.3 HTTP re-check without storing unchanged payloads (collector)

collector/collector/recheck.py, class Rechecker, called from CollectorService.step() after the normal fetch batch when TAKEDOWN_ENABLED and COLLECTOR_ENABLED are both true. It shares the service’s egress client, semaphore, per-host lock and _host_last spacing.

1. Pick ≤ 5 due targets: kind = 'url' AND http_eligible AND tracking AND http_next_at <= now() AND NOT demo.

2. Re-validate the URL (normalize_url with the deny lists). If refused: outcome refused, http_eligible = false (it may now point somewhere private), and nothing is sent.

3. Reserve one unit of takedown_http quota (TAKEDOWN_HTTP_DAILY_QUOTA, 200).

4. GET with the same headers as the fetch (user_agent_for, Accept: */*, Accept-Encoding: identity), redirects by hand (≤ COLLECTOR_MAX_REDIRECTS, each hop validated), verify=False, connect/total timeouts as the collector.

5. On 2xx, stream the raw body with aiter_raw(), hashing sha256 incrementally, capped at COLLECTOR_MAX_BYTES:

   • If quarantine has room: write to a temp file (Quarantine.open_temp).
   • If quarantine is full: hash only, keep nothing.
   • The first 512 bytes are kept in memory for the HTML check.

6. Decide (pure function classify_body(prev_sha, new_sha, head, size, prev_kind)):

   Case	Outcome	Storage
   new_sha == last_sha256	same	temp aborted (unlinked)
   body empty	empty	aborted
   head is HTML (<!doctype html / <html in the first 512 bytes, case-insensitive) and the previous sample kind is not HTML	suspended_page	aborted, nothing stored
   different hash, room in quarantine	changed	committed with Quarantine.commit, samples + sample_sources rows as in service._record_sample; the analyzer picks it up
   different hash, quarantine full	changed_not_stored	nothing
   body over the cap	up_unhashed	aborted

7. Non-2xx: 404 → http_404, 410 → http_410, other 4xx → http_4xx, 5xx → http_5xx.

8. Exceptions:

   • EgressClosed: refund the quota and write no row (nothing was sent).
   • httpx.ProxyError, or a proxy answer 502/503/504 to CONNECT (DNS failure or refusal at the target): unreachable.
   • A connection error to the proxy itself: refund, no row, gate.note_proxy_error(). That is our side, not the target.
   • Timeout: timeout.

9. Write takedown_checks (method http, http_status, sha256 if hashed, bytes, duration_ms). Update the collector-owned columns http_step, http_last_at, http_next_at, last_sha256 (on changed only).

baseline_sha256 is set by ingest when the target is created (fetch_targets.last_sha256). The “changed” state compares against it (§5).

4.4 Listing checks (ingest)

The takedown worker calls the existing clients: IntelWorker.urlhaus(kind, value), IntelWorker.threatfox(...), ReputationWorker.check_abuseipdb(ip). These are refactored so a single lookup can be called without the candidate loop (no behaviour change). Each lookup reserves:

• the provider’s own daily quota (reserve_quota, as today), and
• a share quota: takedown_intel (TAKEDOWN_INTEL_DAILY_QUOTA, 100) or takedown_abuseipdb (TAKEDOWN_ABUSEIPDB_DAILY_QUOTA, 50).

Takedown tracking can therefore never take the whole intel budget. Answers also refresh indicator_intel (24 h cache): a fresh cached answer (< 12 h) is reused instead of a call. URLhaus url_status and last_online are read when present.

4.5 Attacking addresses

No packet is sent. sightings reads sessions (src_ip = value, NOT benign, scanner_name IS NULL, started_at > reported_at). abuseipdb is read at the listing offsets; a lastReportedAt after reported_at means other victims still see it. A static test asserts that the takedown package never calls an AbuseIPDB report endpoint (only the existing check client).

5. States and time to takedown

state ∈ up, down, changed, unknown. Computed by ingest (ingest/ingest/takedown/state.py, pure decide(target, checks, now) -> Decision) after every cycle for targets with new checks.

Target	up	changed	down	unknown
url, http_eligible	latest HTTP check same or up_unhashed	latest HTTP check 2xx and its hash ≠ baseline_sha256 (still serving, other file)	confirmed: the last two HTTP checks are down codes and at least 1 h apart when both are strong (http_404, http_410, http_4xx, suspended_page, empty), or 6 h apart otherwise (unreachable, timeout, http_5xx)	no HTTP check yet
url, not eligible (TFTP, refused, never fetched)	URLhaus online	—	URLhaus offline	otherwise
ip/domain payload_host	any child up/changed	—	every child down (or no children and URLhaus host offline)	otherwise
c2	new_sample after reported_at, or ThreatFox/URLhaus listing refreshed after reported_at	—	never (we do not contact C2s; UI says so)	otherwise
attack_source	seen in the last 72 h, or reported_after in the last 72 h	—	not seen since reported_at and now - max(reported_at, last session) >= TAKEDOWN_QUIET_DAYS (7)	otherwise

The UI wording for attack_source is “Active”, “Quiet” and “Unknown”, never “taken down”. For C2s, the wording explains that only passive signals exist.

A single failing check never changes up to down; the UI shows “Not answering since hh:mm”.

Times stored on the target (ingest):

• last_up_at: the latest up check.
• first_down_at: the first check of the first confirmed down streak.
• resurrections: incremented on down → up.
• down_before_report: the baseline check (step 0) was already down. These targets are excluded from takedown statistics.

Time to takedown (TTD) = first_down_at - reported_at. It is an upper bound (the takedown happened between last_up_at and first_down_at). The UI shows “within 6 h” style ranges and last_up_at on hover.

6. Statistics

Computed by SQL in the API from targets (live data) and from outcome_rollup_monthly (history older than report retention). Excluded: demo targets, down_before_report, and c2/attack_source from takedown percentages (shown separately as “quiet” rates).

• Taken down within H (H = 48 h, 7 d, 30 d):
  • Numerator: targets with first_down_at - reported_at <= H.
  • Denominator: targets with reported_at <= now() - H, plus those already down within H.
  • Targets not yet observed for H are left out of the denominator (right-censoring). This makes “12 of 31 reported malware URLs went down within 48 h” exact.
• Median TTD: median of TTD among targets that went down within TAKEDOWN_TRACK_DAYS, shown with its count (“median 19 h, 12 URLs”). If fewer than 5 values: “not enough data yet”.
• Still up after 30 days: tracked to the end and never down.
• Per provider (provider_key): targets, % down within 48 h / 30 d, median TTD, still up, and M26’s reports sent, % answered, median first reply (reports joined through takedown_target_links and abuse_report_subjects.network_id).
• Per channel: mail only, URLhaus only, both. URLhaus notifies hosting providers itself, so the page shows the split and does not claim credit.

7. abuse.ch submission

7.1 Settings (Settings > Reporting > abuse.ch)

Table abusech_submission (single row, migration 0027):

Column	Meaning
mode	off (default), review, auto
threatfox_enabled	Also submit C2 ip:port to ThreatFox (default false)
anonymous	Ask abuse.ch to hide the submitter (default true)
auto_hold_minutes	10 to 1440, default 60
daily_cap	1 to 200, default 20
key_ciphertext, key_nonce, key_set, key_last4	Auth-Key, AES-GCM with SecretBox, AAD abusech_submission, never returned (last 4 characters shown, as M11)
updated_at, updated_by

• A dedicated key: submissions are attributed to the abuse.ch account behind it. The card warns: “Use an abuse.ch account whose public name does not identify you.” The lookup key (M11 provider abusech) is unchanged and is used for the dedupe lookups.
• PUT and DELETE are admin-only, CSRF, re-auth. GET is admin-only.
• Mode auto needs a key and ABUSECH_SUBMIT_ENABLED=true. When the flag is off, the card reads “Disabled on this server (ABUSECH_SUBMIT_ENABLED)” and the switch is disabled.

7.2 Modes

mode	behaviour
off	candidates are not even computed
review	candidates are listed in the Submissions tab; the admin clicks Submit (one or up to 20 selected); confirmation dialog
auto	each new candidate becomes pending with scheduled_at = now + auto_hold_minutes, shown as “Submitting at hh:mm” with Cancel; at most daily_cap per UTC day

7.3 Candidates (ingest, ingest/ingest/abusech/)

URLhaus (platform = 'urlhaus', ioc_type = 'url', threat = 'malware_download') requires all of:

• fetch_targets.status = 'fetched', and a sample_sources.last_fetched within 48 h (it served malware to us recently);
• the sample kind is not HTML or empty;
• scheme http/https, URL ≤ 1,024 characters;
• indicator not demo;
• host not in the never-list (suckerpot_db.blocklist_values: resolvers, big SaaS and CDN, owner values) and no self term in it;
• not already in abusech_submissions for this platform.

ThreatFox (only with threatfox_enabled; ioc_type = 'ip:port', threat_type = 'botnet_cc') requires:

• sample_links.kind = 'c2_ip' with a port and confidence = 'high';
• the sample family maps to a Malpedia name in ingest/rules/abusech/malpedia.yaml (e.g. mirai: elf.mirai, gafgyt: elf.bashlite). No mapping → not a candidate (ThreatFox requires a family);
• confidence_level 75;
• domains are not submitted in M27.

7.4 Dedupe (lookup first)

Before a candidate becomes visible (review) or scheduled (auto), the worker asks URLhaus (url) or ThreatFox (search_ioc) through the existing lookup client. A fresh indicator_intel answer (< 12 h) counts.

• Known: status skipped_known, with the link built locally (URLHAUS_LINK_PREFIX / THREATFOX_LINK).
• Lookup impossible (quota, no lookup key): the candidate stays candidate with lookup = 'pending' and is not submittable until the lookup is done.

7.5 Body and identity guard

Built by ingest/ingest/abusech/body.py (pure) from a fixed allow-list of fields:

Platform	Body (JSON, Auth-Key header)
URLhaus	{"anonymous": "1" or "0", "submission": [{"url": <url>, "threat": "malware_download", "tags": [≤ 5]}]}
ThreatFox	{"query": "submit_ioc", "threat_type": "botnet_cc", "ioc_type": "ip:port", "malware": <malpedia>, "confidence_level": 75, "anonymous": 1 or 0, "tags": [≤ 5], "iocs": [<ip:port>]}; no reference, no comment

• Tags come from a fixed vocabulary only: the family name, the architecture (elf, arm, mips, mipsel, x86, x86-64, powerpc, sh4), sh for shell scripts. Pattern [a-z0-9.-]{1,32}.
• No free text, no comment, no reference URL, no date, no session data.
• The serialized body (headers excluded; the key is not in the body) goes through guard(body_bytes, terms, fmt="json", ctx=GuardContext(internal_ids=…)). The context carries the internal ids of the sessions that saw the URL.
• A URL that embeds the monitored address (e.g. http://198.51.100.9/x.sh?t=<self address>) fails with self_address and is never submitted: status guard_failed:<codes>.
• no_self_address fails closed too (as in M22).

Endpoint constants live in one place (ingest/ingest/abusech/client.py): URLHAUS_SUBMIT_URL, THREATFOX_API. The implementer verifies the current endpoint, method and field names against abuse.ch’s published API documentation and records a fixture of the expected answer. The client uses make_client (egress gate) and allows only the hosts urlhaus.abuse.ch, urlhaus-api.abuse.ch and threatfox-api.abuse.ch. It never follows redirects, caps the body at 64 KiB and parses JSON with the jsondepth guard. Answers become constant codes:

inserted, already_known, invalid, blacklisted, auth_failed, rate_limited, http_error, unknown_outcome, egress_closed, guard_failed.

7.6 Delivery at most once

The same pattern as abuse_send_requests:

• ingest commits status = 'submitting' (started_at) before the request;
• only pending rows are picked;
• a submitting row older than 5 minutes becomes failed / unknown_outcome and is never retried;
• EgressClosed before the request returns the row to pending.

The daily cap counts rows that reached submitting today (UTC), quota provider abusech_submit.

7.7 Status values and trigger

candidate, pending, submitting, submitted, duplicate, rejected, failed, cancelled, skipped_known, skipped_never_list, guard_failed.

Trigger abusech_submissions_guard:

• The API may only move candidate → pending (sets approved_by, approved_at; requires mode review or auto) and pending → cancelled.
• Ingest inserts candidates and moves everything else.
• Final statuses are immutable.
• Demo rows never leave candidate.

7.8 Audit

The abusech_submissions table is the log of every submission, including automatic ones. Rows are immutable once final and kept for 730 days. API audit entries:

• abusech.settings.update (field names only), abusech.settings.delete;
• abusech.submit (submission ids, platform), abusech.submit.cancel.

The key and the body are never logged.

8. Monthly roll-up (history and M29)

outcome_rollup_monthly is recomputed by the takedown worker once a day for the current and previous month. It is kept forever and is not pruned with reports:

month, provider_key, asn, country, kind (attack_source|payload_host|payload_url|c2), channel (mail|urlhaus|threatfox), reports_sent, reports_answered, auto_ack, ticket_opened, resolved, rejected, bounced, human_reply, targets, targets_eligible_48h, targets_down_48h, targets_eligible_7d, targets_down_7d, targets_eligible_30d, targets_down_30d, ttd_p50_s, first_reply_p50_s, computed_at.

It holds aggregates only: no values, no URLs, no addresses. The Results page uses it for months whose reports were already deleted by retention.

9. Migration 0027_takedown

CREATE TABLE takedown_targets (
  id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
  kind text NOT NULL CHECK (kind IN ('url', 'ip', 'domain')),
  role text NOT NULL CHECK (role IN ('attack_source', 'payload_host', 'payload_url', 'c2')),
  value text NOT NULL CHECK (char_length(value) BETWEEN 1 AND 1024),
  target_key text NOT NULL CHECK (target_key ~ '^[0-9a-f]{64}$'),   -- sha256(kind || ':' || value)
  parent_id bigint REFERENCES takedown_targets (id) ON DELETE CASCADE,
  fetch_target_id bigint REFERENCES fetch_targets (id) ON DELETE SET NULL,
  network_id bigint REFERENCES abuse_networks (id) ON DELETE SET NULL,
  asn integer, country text CHECK (country IS NULL OR char_length(country) = 2),
  provider_key text NOT NULL CHECK (char_length(provider_key) BETWEEN 1 AND 64),
  provider_name text CHECK (char_length(provider_name) <= 200),
  reported_at timestamptz NOT NULL,
  channels text[] NOT NULL DEFAULT '{}' CHECK (channels <@ ARRAY['mail', 'urlhaus', 'threatfox']),
  demo boolean NOT NULL DEFAULT false,
  tracking boolean NOT NULL DEFAULT true,
  track_until timestamptz NOT NULL,
  stopped_at timestamptz, stopped_by text CHECK (char_length(stopped_by) <= 120),
  -- state (ingest)
  state text NOT NULL DEFAULT 'unknown' CHECK (state IN ('up', 'down', 'changed', 'unknown')),
  state_source text CHECK (state_source IN
    ('http', 'urlhaus', 'threatfox', 'abuseipdb', 'sightings', 'samples', 'children')),
  state_since timestamptz, state_updated_at timestamptz,
  down_before_report boolean NOT NULL DEFAULT false,
  last_up_at timestamptz, first_down_at timestamptz,
  resurrections smallint NOT NULL DEFAULT 0,
  listing_step smallint NOT NULL DEFAULT 0, listing_next_at timestamptz,
  -- HTTP re-check (http_eligible, baseline by ingest; the rest by the collector)
  http_eligible boolean NOT NULL DEFAULT false,
  baseline_sha256 text CHECK (baseline_sha256 ~ '^[0-9a-f]{64}$'),
  last_sha256 text CHECK (last_sha256 ~ '^[0-9a-f]{64}$'),
  http_step smallint NOT NULL DEFAULT 0, http_next_at timestamptz, http_last_at timestamptz,
  created_at timestamptz NOT NULL DEFAULT now(),
  UNIQUE (kind, role, value),
  CHECK (NOT http_eligible OR kind = 'url'),
  CHECK (role <> 'payload_url' OR kind = 'url'));
CREATE INDEX ix_takedown_http_due ON takedown_targets (http_next_at)
  WHERE http_eligible AND tracking AND NOT demo;
CREATE INDEX ix_takedown_listing_due ON takedown_targets (listing_next_at) WHERE tracking AND NOT demo;
CREATE INDEX ix_takedown_provider ON takedown_targets (provider_key, reported_at);
CREATE INDEX ix_takedown_state ON takedown_targets (state, reported_at DESC);
CREATE INDEX ix_takedown_parent ON takedown_targets (parent_id) WHERE parent_id IS NOT NULL;

CREATE TABLE takedown_checks (
  id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
  target_id bigint NOT NULL REFERENCES takedown_targets (id) ON DELETE CASCADE,
  checked_at timestamptz NOT NULL DEFAULT now(),
  method text NOT NULL CHECK (method IN ('http', 'urlhaus', 'threatfox', 'abuseipdb', 'sightings', 'samples')),
  outcome text NOT NULL CHECK (outcome ~ '^[a-z_0-9]{1,32}$'),
  http_status smallint, sha256 text CHECK (sha256 IS NULL OR sha256 ~ '^[0-9a-f]{64}$'),
  bytes bigint, duration_ms integer,
  detail jsonb NOT NULL DEFAULT '{}' CHECK (octet_length(detail::text) <= 1024));
CREATE INDEX ix_takedown_checks_target ON takedown_checks (target_id, checked_at DESC);

CREATE TABLE abusech_submission (
  id smallint PRIMARY KEY DEFAULT 1 CHECK (id = 1),
  mode text NOT NULL DEFAULT 'off' CHECK (mode IN ('off', 'review', 'auto')),
  threatfox_enabled boolean NOT NULL DEFAULT false,
  anonymous boolean NOT NULL DEFAULT true,
  auto_hold_minutes integer NOT NULL DEFAULT 60 CHECK (auto_hold_minutes BETWEEN 10 AND 1440),
  daily_cap integer NOT NULL DEFAULT 20 CHECK (daily_cap BETWEEN 1 AND 200),
  key_ciphertext bytea, key_nonce bytea,
  key_set boolean GENERATED ALWAYS AS (key_ciphertext IS NOT NULL) STORED,
  key_last4 text CHECK (key_last4 IS NULL OR char_length(key_last4) = 4),
  updated_at timestamptz NOT NULL DEFAULT now(),
  updated_by text CHECK (char_length(updated_by) <= 120),
  CHECK ((key_ciphertext IS NULL) = (key_nonce IS NULL)),
  CHECK (mode <> 'auto' OR key_ciphertext IS NOT NULL));

CREATE TABLE abusech_submissions (
  id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
  platform text NOT NULL CHECK (platform IN ('urlhaus', 'threatfox')),
  ioc_type text NOT NULL CHECK (ioc_type IN ('url', 'ip:port')),
  value text NOT NULL CHECK (char_length(value) BETWEEN 1 AND 1024),
  threat text NOT NULL CHECK (threat IN ('malware_download', 'botnet_cc')),
  malware text CHECK (char_length(malware) <= 64),
  tags text[] NOT NULL DEFAULT '{}' CHECK (cardinality(tags) <= 5),
  sample_sha256 text REFERENCES samples (sha256) ON DELETE SET NULL,
  fetch_target_id bigint REFERENCES fetch_targets (id) ON DELETE SET NULL,
  demo boolean NOT NULL DEFAULT false,
  lookup text NOT NULL DEFAULT 'pending' CHECK (lookup IN ('pending', 'unknown', 'known')),
  status text NOT NULL DEFAULT 'candidate' CHECK (status IN ('candidate', 'pending', 'submitting',
    'submitted', 'duplicate', 'rejected', 'failed', 'cancelled', 'skipped_known',
    'skipped_never_list', 'guard_failed')),
  result_code text CHECK (char_length(result_code) <= 120),
  reference text CHECK (reference IS NULL OR char_length(reference) <= 256),  -- built locally
  auto boolean NOT NULL DEFAULT false,
  created_at timestamptz NOT NULL DEFAULT now(),
  approved_at timestamptz, approved_by text CHECK (char_length(approved_by) <= 120),
  scheduled_at timestamptz, started_at timestamptz, submitted_at timestamptz, done_at timestamptz,
  UNIQUE (platform, value));
CREATE INDEX ix_abusech_submissions_status ON abusech_submissions (status, created_at DESC);
CREATE INDEX ix_abusech_submissions_pending ON abusech_submissions (scheduled_at) WHERE status = 'pending';

CREATE TABLE takedown_target_links (
  target_id bigint NOT NULL REFERENCES takedown_targets (id) ON DELETE CASCADE,
  report_id bigint REFERENCES abuse_reports (id) ON DELETE CASCADE,
  submission_id bigint REFERENCES abusech_submissions (id) ON DELETE CASCADE,
  linked_at timestamptz NOT NULL DEFAULT now(),
  CHECK ((report_id IS NULL) <> (submission_id IS NULL)));
CREATE UNIQUE INDEX ux_takedown_links_report ON takedown_target_links (target_id, report_id)
  WHERE report_id IS NOT NULL;
CREATE UNIQUE INDEX ux_takedown_links_submission ON takedown_target_links (target_id, submission_id)
  WHERE submission_id IS NOT NULL;

CREATE TABLE outcome_rollup_monthly (
  month date NOT NULL CHECK (extract(day FROM month) = 1),
  provider_key text NOT NULL CHECK (char_length(provider_key) <= 64),
  asn integer, country text CHECK (country IS NULL OR char_length(country) = 2),
  kind text NOT NULL CHECK (kind IN ('attack_source', 'payload_host', 'payload_url', 'c2')),
  channel text NOT NULL CHECK (channel IN ('mail', 'urlhaus', 'threatfox')),
  reports_sent integer NOT NULL DEFAULT 0, reports_answered integer NOT NULL DEFAULT 0,
  auto_ack integer NOT NULL DEFAULT 0, ticket_opened integer NOT NULL DEFAULT 0,
  resolved integer NOT NULL DEFAULT 0, rejected integer NOT NULL DEFAULT 0,
  bounced integer NOT NULL DEFAULT 0, human_reply integer NOT NULL DEFAULT 0,
  targets integer NOT NULL DEFAULT 0,
  targets_eligible_48h integer NOT NULL DEFAULT 0, targets_down_48h integer NOT NULL DEFAULT 0,
  targets_eligible_7d integer NOT NULL DEFAULT 0, targets_down_7d integer NOT NULL DEFAULT 0,
  targets_eligible_30d integer NOT NULL DEFAULT 0, targets_down_30d integer NOT NULL DEFAULT 0,
  ttd_p50_s integer, first_reply_p50_s integer,
  computed_at timestamptz NOT NULL DEFAULT now(),
  PRIMARY KEY (month, provider_key, kind, channel));

• Trigger abusech_submissions_guard (§7.7). Like 0022, it detects the role through a column privilege only ingest holds (UPDATE on result_code).
• Trigger takedown_targets_guard: kind, role, value, target_key, reported_at may only move earlier, never later; demo is immutable.
• Procedure suckerpot_takedown_prune(job_id, config), scheduled daily by suckerpot_db.retention:
  • delete takedown_checks older than check_days (TAKEDOWN_CHECK_RETENTION_DAYS, 400);
  • delete targets whose track_until is older than 400 days (links and checks cascade);
  • delete abusech_submissions older than 730 days;
  • never touch outcome_rollup_monthly.

fetch_targets/samples deletion by the M21 retention sets fetch_target_id / sample_sha256 to NULL and leaves the target in place.

Models in db/suckerpot_db/models.py: TakedownTargetRow, TakedownCheckRow, TakedownTargetLinkRow, AbusechSubmissionSettingsRow, AbusechSubmissionRow, OutcomeRollupRow.

10. Roles

Table	suckerpot_ingest	suckerpot_collector	suckerpot_api
takedown_targets	SELECT, INSERT, UPDATE (all but the collector columns), DELETE	SELECT; UPDATE (http_step, http_next_at, http_last_at, last_sha256, http_eligible)	SELECT; UPDATE (tracking, stopped_at, stopped_by, http_next_at, listing_next_at) for stop and recheck
takedown_checks	SELECT, INSERT	INSERT	SELECT
takedown_target_links	SELECT, INSERT, DELETE	—	SELECT
abusech_submission	SELECT (all incl. key)	—	SELECT (all but key_ciphertext, key_nonce), INSERT, UPDATE, DELETE
abusech_submissions	SELECT, INSERT, UPDATE	—	SELECT; UPDATE (status, approved_at, approved_by) trigger-limited
outcome_rollup_monthly	SELECT, INSERT, UPDATE, DELETE	—	SELECT
enrichment_quota / enrichment_status	existing	RLS predicates extended: provider collector or takedown_http; status names collector, egress_collector, takedown_http	existing
fetch_targets, samples, sample_sources, sample_links	SELECT (confirm; ingest’s M22 candidates already read sample_links/samples)	existing (recheck reuses the sample insert path)	existing

11. API (api/api/takedown.py, api/api/abusech.py)

Method	Path	Who	Notes
GET	/api/v1/takedown/summary	any role	since/until, optional kind, channel. Answer: urls {tracked, eligible_48h, down_48h, eligible_7d, down_7d, eligible_30d, down_30d, ttd_p50_s, ttd_n, still_up_30d}, the same for hosts; attack_sources {tracked, quiet, active}; c2 {tracked, still_used}; reports (from M26 /abuse/outcomes totals)
GET	/api/v1/takedown/curve	any role	Cumulative % down by hours since report (points at 1, 6, 24, 48 h, 3, 7, 14, 30 d) per kind, censored as §6
GET	/api/v1/takedown/providers	any role	Per provider_key: name, ASN, country, targets, % down 48 h / 30 d, ttd_p50_s, still up, reports sent, % answered, first reply p50. Sort, limit ≤ 200; history months merged from outcome_rollup_monthly
GET	/api/v1/takedown/targets	any role	Cursor pagination (limit ≤ 100). Filters state, kind, role, provider, channel, tracking, q. URLs returned raw; the UI defangs
GET	/api/v1/takedown/targets/{id}	any role	Target, children, links (report public_ids, submission ids), checks timeline (≤ 200, newest first), computed TTD and bounds
POST	/api/v1/takedown/targets/{id}/recheck	admin	Sets http_next_at/listing_next_at to now, if the last check is older than 1 h (else 429 too_soon). Audit takedown.recheck
POST	/api/v1/takedown/targets/{id}/stop	admin	tracking = false. Audit takedown.stop
GET	/api/v1/abusech/submission	admin	Settings without key, key_set, key_last4, server_enabled (env flag), submitted_today
PUT	/api/v1/abusech/submission	admin, re-auth	{mode, threatfox_enabled, anonymous, auto_hold_minutes, daily_cap, key?, current_password}; 422 server_disabled when the mode is not off and the flag is off; 422 key_required for auto without a key. Audit
DELETE	/api/v1/abusech/submission	admin, re-auth	Removes the key, mode off. Audit
GET	/api/v1/abusech/submissions	any role	Cursor pagination; filters status, platform, since. Fields: value, platform, status, result code, tags, reference link, times. Counts per status for the tabs
POST	/api/v1/abusech/submissions/submit	admin	{ids: [≤ 20]} → candidate → pending (scheduled_at NULL, picked at the next cycle); 409 when the mode is off or the lookup is not unknown. Audit abusech.submit
POST	/api/v1/abusech/submissions/{id}/cancel	admin	pending → cancelled. Audit abusech.submit.cancel

setup_status.py gains two optional items: “Takedown tracking” (TAKEDOWN_ENABLED) and “abuse.ch submission” (mode). Neither blocks the checklist.

12. UI (follow web/DESIGN.md; CLAUDE.md §4; no new hue)

The “Abuse reports” hub gets the tabs Reports (/defend/abuse), Submissions (/defend/submissions) and Results (/defend/results).

Results page (web/src/pages/results.tsx, extended from M26). Every card opens with a sentence built by web/src/lib/results-sentences.ts. Filters (since, kind, channel, provider) live in the URL.

1. Header finding:
   • “Since 12 September you reported 31 malware URLs; 12 went down within 48 hours and 19 within 30 days.”
   • Empty: “Nothing reported yet. Takedowns appear here after your first report or submission.”
   • Disabled: “Takedown tracking is off (TAKEDOWN_ENABLED).”
2. Stat tiles: Reports sent, Answered, Taken down within 48 h (“12 of 31”), Median time to takedown (“19 h, 12 URLs”).
3. How fast do things go down?, e.g. “12 of 31 reported malware URLs went down within 48 h.” Step chart of the cumulative % down vs time since report (Recharts, existing chart wrappers in web/src/components/charts), one line per kind (URLs, hosts), with a legend that has text labels.
4. Did anyone answer? and Reports over time: the M26 cards.
5. Providers table, e.g. “AS64500 took down 5 of 6 URLs, the fastest median in your reports (7 h).” Columns: provider (ASN chip, clickable to /networks), country flag, targets, down within 48 h, median TTD, still up after 30 days, reports, answered, first reply. Sortable, virtualized.
6. Targets table: target (defanged text, never a link), kind badge, provider, reported (relative time), channel badges (Mail, URLhaus, ThreatFox), state badge (icon + word: Up, Down, Changed, Unknown; for attacking addresses Active, Quiet, Unknown), TTD. Rows open the target page.
7. abuse.ch card, e.g. “18 URLs were submitted to URLhaus this month; 6 were already known.” Links to the Submissions tab.

Target page /defend/results/targets/:id:

• Header: the defanged value, provider, reported time, state, and TTD with its bounds (“down between 14:00 and 20:00, 19 h after the report”).
• A sentence, e.g. “Still serving the same file 3 days after the report.” / “Serves a different file since 2 Nov (a new sample).” / “We never contact C2 servers; this shows whether new malware still points to it.”
• A Timeline as a vertical step list (reuse the campaign story steps): report sent, replies (M26), each check with its method, outcome in words, and HTTP status. A changed hash links to the new sample.
• Children URLs for a host. Admin buttons “Check now” and “Stop tracking”.

Colour:

• Up uses the severity “severe” token only for payload_url/payload_host (malware still being served, §4 “red only for severe”).
• Changed uses the warning tone. Down uses the success/neutral tone.
• Attacking addresses and C2s never use red.
• Every state carries an icon and a word.

Submissions tab (web/src/pages/submissions.tsx):

• Opening sentence: “Submission is off.” / “You review each URL before it goes to URLhaus.” / “New URLs go to URLhaus automatically after 60 minutes; 4 of 20 sent today.”
• Status tabs: To review, Scheduled, Submitted, Skipped, Failed.
• Rows: defanged URL, platform, tags, lookup result, status (icon + word), result code in words, URLhaus link (external, rel="noreferrer noopener"; built server-side from fixed prefixes).
• Admin: checkbox selection plus “Submit selected” (confirmation dialog listing the URLs); “Cancel” on scheduled rows.
• Viewers: read-only.

Settings > Reporting > abuse.ch card (web/src/components/abuse/abusech-card.tsx):

• Mode as a segmented control (Off, Review each, Automatic), ThreatFox switch, Anonymous switch, hold and cap.
• Write-only key field (last 4 shown), the account warning, the server-flag notice. Re-auth dialog on save.

Mocks: web/src/mocks/takedown.ts, web/src/mocks/abusech.ts (documentation addresses, example.net URLs, documentation ASNs, generated hashes).

13. Configuration

Variable	Default	Used by
TAKEDOWN_ENABLED	false	ingest (worker), collector (re-checks; also needs COLLECTOR_ENABLED)
TAKEDOWN_TRACK_DAYS	30 (7 to 90)	ingest, collector
TAKEDOWN_HTTP_DAILY_QUOTA	200	collector
TAKEDOWN_INTEL_DAILY_QUOTA	100	ingest
TAKEDOWN_ABUSEIPDB_DAILY_QUOTA	50	ingest
TAKEDOWN_QUIET_DAYS	7	ingest
TAKEDOWN_CHECK_RETENTION_DAYS	400	migrate
ABUSECH_SUBMIT_ENABLED	false	ingest (worker), api (reported in GET, refuses non-off modes)

docker-compose.yml passes these to ingest, collector, api and migrate (environment only, no network or volume change). .env.example documents them. The abuse.ch key is not an environment variable.

14. Tests

• ingest/ (ingest/tests/takedown/, ingest/tests/abusech/):
  • decide() table tests: every row of §5, including the 1 h / 6 h confirmation spacing, resurrection, down_before_report, C2 never down, attack source quiet vs active;
  • schedule functions on fixed clocks;
  • target sync from sent reports (dedupe, earliest reported_at, child URLs ≤ 20, demo rules, provider key);
  • listing checks within quotas (share quota exhausted → no call), EgressClosed → no row, no quota;
  • roll-up SQL on synthetic data (integration);
  • candidates: each refusal reason; dedupe (known → skipped_known and zero submit calls);
  • guard: a URL containing the planted self address → guard_failed, zero calls;
  • mode off or env flag off → zero calls; daily cap; at-most-once (crash after submitting → unknown_outcome, never resent); anonymous flag in the body; tags only from the vocabulary;
  • the key never in logs or rows (caplog); response fixtures → result codes;
  • static test: ingest/ingest/takedown/** and ingest/ingest/abusech/** pass test_egress_static.py, contain no AbuseIPDB report call, and the abusech client’s host allow-list is exactly the three hosts.
• collector/ (collector/tests/test_recheck.py):
  • same hash → no file left in quarantine (including .incoming), no samples row;
  • changed → committed sample and sample_sources row;
  • suspended HTML → nothing stored;
  • quarantine full → changed_not_stored, nothing written;
  • oversize → up_unhashed;
  • redirect to a private address → refused and http_eligible=false;
  • gate closed → no row, quota refunded;
  • proxy itself unreachable → no row;
  • CONNECT 502 → unreachable;
  • only kind='url' AND http_eligible rows are ever selected;
  • per-host spacing shared with fetches;
  • the collector static test still passes.
• db/: upgrade and downgrade 0026 ↔ 0027; grants (collector cannot change state; API cannot submit; demo submission cannot leave candidate; final submission immutable); RLS on enrichment_quota for takedown_http; prune keeps the roll-up.
• api/: authz walk (viewer 403 on recheck, stop, submit, cancel, settings); re-auth on PUT and DELETE; key never returned; server_disabled and key_required; summary and curve censoring on a synthetic fixture where half the targets are younger than 48 h (integration); recheck rate limit.
• web/: vitest for every sentence branch (zero, one, many; censoring wording “of 31 old enough to tell”); state badge words per kind; submissions read-only for viewers; settings key write-only. Screenshots 1440/390 dark/light of Results, the target page, Submissions and Settings > abuse.ch.

15. Task split

Every prompt repeats: no value from ENVIRONMENT.local.md or .env; documentation ranges, ASNs and example.* names only; no request to anything but the URL being re-checked and the fixed intel and abuse.ch hosts.

Wave 1 (parallel)

A. Migration and roles (sonnet, backend-engineer)

• Owns: db/migrations/versions/0027_takedown.py, db/suckerpot_db/{models,roles,retention}.py, db/tests/**.
• Contracts: §9, §10.
• Verify: just lint typecheck test, just test-integration.

E. Web (sonnet, frontend-engineer, on mocks; after M26’s web is merged)

• Owns:
  • web/src/pages/{results,results-target,submissions}.tsx, web/src/app/{nav.ts,nav.test.tsx,router.tsx};
  • web/src/components/results/**, web/src/components/abuse/abusech-card.tsx, web/src/pages/settings.tsx (card mount only);
  • web/src/lib/results-sentences.ts, web/src/lib/api*;
  • web/src/mocks/{takedown,abusech,handlers}.ts, and the tests.
• Contracts: §11 shapes, §12.
• Verify: just web-test web-build screenshots; ui-designer review.

Wave 2 (after A)

B. Collector re-checks (sonnet, backend-engineer)

• Owns: collector/collector/{recheck,service,config}.py, collector/tests/**.
• Contracts: §4.2, §4.3, T1 to T4.
• Verify: just lint typecheck test, just test-integration.

C. Ingest takedown worker (sonnet, backend-engineer)

• Owns:
  • ingest/ingest/takedown/** (sync, schedule, listing checks, state.decide, roll-up);
  • ingest/ingest/enrichment/{intel,reputation}.py (single-lookup refactor only);
  • ingest/ingest/features.py, ingest/ingest/main.py (wiring for C and D: D exposes AbusechSubmitWorker(factory, terms, gate, box) with the FeatureWorker shape);
  • ingest/tests/takedown/**;
  • docker-compose.yml (environment of ingest, collector, api, migrate only), .env.example.
• Contracts: §3 to §6, §8.
• Verify: just lint typecheck test, just test-integration.

D. abuse.ch submission (sonnet, backend-engineer; parallel with C, separate directory)

• Owns: ingest/ingest/abusech/** (candidates, body, client, worker), ingest/rules/abusech/malpedia.yaml, ingest/tests/abusech/**.
• Contracts: §7, T6, T7. The endpoint and fields are verified against current abuse.ch documentation; record the fixtures.
• Verify: just lint typecheck test.

Wave 3

F. API (sonnet, backend-engineer; can start after A, on fixtures)

• Owns: api/api/takedown.py, api/api/abusech.py, api/api/routes.py, api/api/setup_status.py, api/tests/**.
• Contracts: §11.
• Verify: just lint typecheck test, just test-integration.

G. Docs (haiku, docs-writer)

• docs/takedowns.md (new: what is checked and what is never contacted, how to read the Results page, censoring in plain words, abuse.ch setup and the account warning).
• docs/api.md, docs/ui.md, docs/configuration.md, docs/retention.md, docs/ARCHITECTURE.md, docs/SECURITY.md (“Takedown re-checks are not scans”).
• docs/index.md (toctree design/m27, takedowns).
• The orchestrator amends CLAUDE.md §3 (workers) and §5 (the sentence of §2), with the owner’s approval.

Reviews: security-reviewer (opus) on B, C, D and F together. The review hunts for:

• any path that contacts a C2, an attacking address, or a URL that never served malware;
• a re-check that stores an unchanged payload;
• a submission that bypasses the guard, the mode, the env flag or the cap;
• the key leaking;
• quota exhaustion by one attacker.

ui-designer on E.

16. What M29 (community network) reuses

• takedown_targets.target_key (sha256 of kind:value): a hub can group the same URL or C2 reported by several nodes without nodes sending values in clear (opt-in nodes may also share the value for public malware URLs, an M29 decision).
• provider_key (as<asn> / registrar recipient) and the state, outcome and check-outcome codes are stable constants: the hub’s grouping keys.
• outcome_rollup_monthly is exactly what a node shares: per month, provider, kind and channel, counts and medians, no values. It is kept forever, so a node that joins late can share its history.
• reported_at, first_down_at and channels let a hub compute cross-node TTD per provider and avoid double-counting a takedown that several nodes report.
• abusech_submissions (platform, value) uniqueness and the lookup-first dedupe extend naturally to “the hub says another node already submitted it”.
• Never shared: report text, replies, tokens, the abuse.ch key, quarantine contents, session data.

17. Risks and open questions

1. abuse.ch API details (endpoint, anonymous field, answer shapes) are to be verified at implementation. The client is isolated behind two constants and fixtures.
2. Attribution on URLhaus: submissions show the account’s public name unless anonymous is honoured. The owner needs an account that does not identify him; the card says so.
3. Timing correlation: a URL submitted minutes after it hit the honeypot tells the attacker that someone they attacked reports. The hold time (auto_hold_minutes) adds delay; the home address is never exposed.
4. Re-check visibility: the payload host sees the VPN exit fetch the file again about 35 times in 30 days. It may block the VPN exit, which reads as http_4xx/unreachable, i.e. “down”. The 6 h confirmation for weak codes and the URLhaus cross-check reduce false takedowns. Accepted residual.
5. C2 state is passive only by design, so C2s are reported as “still used / unknown”, never “down”.
6. ThreatFox domains are not submitted in M27.
7. CLAUDE.md §5 amendment (one sentence, §2) needs the owner’s approval before merge.