M26 — Report outcomes: did anyone answer?

Status: proposal for review. Builds on M22 (reports), M25 (send queue and SMTP). Default behaviour is unchanged: no mailbox is contacted until the owner sets ABUSE_REPLIES_ENABLED=true and saves and enables a mailbox in Settings. Without a mailbox, everything else in this milestone works: the owner can paste a reply (or upload an .eml), set an outcome by hand, and the Results page shows what was sent. Every address, mailbox and name below is a placeholder (documentation ranges, example.org, example.net, documentation ASNs AS64496 to AS64511).

1. Goal and invariants

The owner wants to see how many reports were sent (per provider, ASN, country, over time), whether anyone answered, what kind of answer it was, and how long it took.

#	Invariant	Enforced by
O1	The mailbox is reached only through the VPN proxy. The gate is checked before any socket exists; with the gate closed nothing is attempted.	suckerpot_egress.TunnelIMAP4 / TunnelIMAP4_SSL (§4.1), ingest/tests/test_egress_static.py (imaplib clients forbidden elsewhere)
O2	Mail is never deleted, written or answered. The only changes are \Seen or UID MOVE to a folder. No \Deleted, EXPUNGE, APPEND, DELETE, RENAME, no SMTP from the replies package.	SafeMailbox wrapper (§4.2), ingest/tests/test_replies_static.py
O3	Incoming mail is hostile. Sizes are capped, only headers and a bounded partial body are fetched, parsing is stdlib email with compat32 on bounded bytes, attachments are never decoded or opened, HTML is never rendered (tags stripped to text when there is no text part), no link is followed or made clickable.	§5
O4	Mail unrelated to a report is never stored and never flagged. The body is fetched only after the headers show a link to a report or a delivery status notification (DSN).	§4.3
O5	Only metadata and a bounded plain-text excerpt (≤ 2,000 characters) are stored. Excerpts are deleted after ABUSE_REPLY_EXCERPT_DAYS.	§9, §13
O6	The reply token is random and opaque. It is independent of every internal id and appears only in the outgoing Subject (and, if chosen, one body line). It is never in X-ARF, CSV, STIX, MISP, file names or the Message-ID.	§2, guard tests §15
O7	A reply changes a report’s outcome automatically only when it is verified: matched by our own Message-ID, or authenticated (DKIM/DMARC pass) from the recipient’s domain. Anything else waits for the owner.	§7.3
O8	Off by default: ABUSE_REPLIES_ENABLED=false, and report_imap.enabled=false even when a profile exists.	ingest.features, §3

2. The reply token

2.1 Generation

abuse_reports.reply_token text NOT NULL UNIQUE, default left(replace(gen_random_uuid()::text, '-', ''), 12), CHECK (reply_token ~ '^[0-9a-f]{12}$'). The first 12 hex digits of a v4 UUID are random (48 bits, from pg_strong_random). Existing rows are backfilled by the default when the column is added. The token is not derived from id, public_id or id_seed. suckerpot_reporting.ids.new_reply_token() (secrets.token_hex(6)) exists for tests and for any code path that inserts the column explicitly. The transition trigger adds reply_token to the immutable identity columns.

2.2 Where it appears

report_settings.reference_style (subject default, subject_and_body, off), set in Settings > Reporting:

• The subject template gains a {ref} placeholder. If the template has no {ref} and the style is not off, the renderer appends (ref 3f9a1c7b2e4d) to the subject.
• subject_and_body also adds a last body line Ref: 3f9a1c7b2e4d.
• Never in the X-ARF YAML part, evidence.csv, STIX, MISP, .eml file names or download names.

The SMTP Message-ID stays <token_urlsafe(18)@sender-domain> (M25), unrelated to reply_token. Ingest now stores it in abuse_send_requests.message_id when it claims the request (status sending), before connecting. Manual sends (send queue, mailto:) have no Message-ID we know of: the subject token is their matching key.

2.3 What it reveals

48 random bits, one per report: nothing about the owner, the monitored system or the project. The shape (ref <12 hex>) is a common ticket-reference shape; the residual fingerprint risk (the format is in this public repo) is stated in Settings, and the owner can move it ({ref} anywhere in the template) or turn it off (off keeps Message-ID matching for SMTP sends only).

2.4 Guard changes (reporting/)

• GuardContext.internal_ids also receives id_seed.hex (32 chars) from ingest (ingest/ingest/abuse/build.py, reportmail/sender.py) and the API (abuse_core.guard_context): the seed must never leave, and a token accidentally built from it is caught.
• message_subject() and render_text() take reply_token: str | None and reference_style. render_xarf() passes the token to the human part only; render_csv, stix, misp never receive it.
• No new reason code. A token never matches existing codes because it is 12 hex characters and the shortest internal id (session id, UUID) is longer.

3. Mailbox profile (Settings > Reporting > Replies)

One profile in table report_imap (single row, migration 0026). Same pattern as report_smtp:

• Columns: enabled (default false), host, port, security (ssl default port 993, starttls port 143; no plain-text option), username, password (AES-GCM with the existing SecretBox, AAD report_imap, never returned; password_set generated column), folder (default INBOX), processed_action (mark_read default, move), move_folder, poll_minutes (5 to 1440, default 15), since (date; mail with an earlier INTERNALDATE is ignored; defaults to the day the profile is first saved), updated_at, updated_by.
• host must pass api.abuse_smtp.valid_host (a public DNS name, no IP literal, no internal suffix). folder and move_folder: printable ASCII, 1 to 100 chars, no ", \, *, %, CR or LF (always sent quoted). move_folder must differ from folder.
• Write routes are admin-only, CSRF, and ask for the account password again. GET is admin-only too (it holds the owner’s mailbox) and returns everything but the password.
• Recommended in the UI: a dedicated mailbox or alias for reports. The reply contact of M22 is where replies arrive; the profile should read that mailbox.
• Proton: IMAP needs Proton Bridge, which is not part of the stack (open question 1). The card says so.
• Nothing in the profile goes through the identity guard: it is sent only to the owner’s own mail provider, through the tunnel. The client never sends the IMAP ID command.

Ingest keeps its own state in report_imap_state (single row): profile_fingerprint (sha256 of host|port|username|folder), uidvalidity, last_uid, move_supported, last_poll_at, last_ok_at, last_error, counters (today’s fetched, relevant, matched, ignored, oversize). When the fingerprint changes (another mailbox or folder), ingest resets uidvalidity and last_uid.

“Test connection” and “Check now” are rows in report_imap_requests (kind: test | poll), the API-asks-ingest-does pattern of abuse_send_requests.

4. Polling worker (ingest/ingest/replies/, flag ABUSE_REPLIES_ENABLED)

4.1 Egress adapter (egress/suckerpot_egress/imap.py, new)

class TunnelIMAP4(imaplib.IMAP4):           # plain IMAP; STARTTLS is mandatory before LOGIN
    def __init__(self, gate: EgressGate, host: str, port: int, *, timeout: float) -> None
    def _create_socket(self, timeout: float | None) -> socket.socket
        # open_tunnel_sync(self._gate, self.host, self.port, timeout=timeout or 15.0)

class TunnelIMAP4_SSL(imaplib.IMAP4_SSL):   # implicit TLS; certificate and host name verified
    def __init__(self, gate: EgressGate, host: str, port: int, *,
                 ssl_context: ssl.SSLContext, timeout: float) -> None
    def _create_socket(self, timeout: float | None) -> socket.socket
        # raw = open_tunnel_sync(...); return self.ssl_context.wrap_socket(raw, server_hostname=self.host)

Exported from suckerpot_egress/__init__.py. The module lowers imaplib._MAXLINE to 65,536 (process wide; only ingest imports imaplib). debug stays 0: server answers are never logged.

ingest/tests/test_egress_static.py: add imaplib.IMAP4, imaplib.IMAP4_SSL, poplib.POP3, poplib.POP3_SSL to FORBIDDEN_CALLS and imaplib, poplib to FORBIDDEN_MODULES (the replies package imports TunnelIMAP4* from suckerpot_egress, never imaplib clients itself; it may import imaplib.Internaldate2tuple through a helper in suckerpot_egress.imap, so the replies package has no imaplib import at all).

4.2 SafeMailbox (ingest/ingest/replies/imap.py)

A thin wrapper that owns the private imaplib object and exposes only:

Method	IMAP	Notes
connect()	TCP via tunnel, TLS or STARTTLS, CAPABILITY	STARTTLS missing → tls_failed; never LOGIN in clear
login()	LOGIN	password from the decrypted profile, never logged
select(folder, readonly)	SELECT / EXAMINE	returns UIDVALIDITY, EXISTS; EXAMINE for tests
search_new(last_uid, since)	UID SEARCH UID n:* SINCE dd-Mon-yyyy	never UNSEEN: the owner may have read mail already
fetch_headers(uids)	UID FETCH … (UID RFC822.SIZE INTERNALDATE BODY.PEEK[HEADER.FIELDS (…)])	batches of 25
fetch_partial(uid, n)	UID FETCH uid (BODY.PEEK[]<0.n>)	.PEEK: fetching never sets \Seen
mark_seen(uid)	UID STORE uid +FLAGS.SILENT (\Seen)	the only flag ever written
move(uid, folder)	UID MOVE (RFC 6851)	only when MOVE is in the capabilities; else falls back to mark_seen and records move_supported=false
list_folders()	LIST "" "*"	test only; ≤ 50 names, sanitised
logout()	LOGOUT	always, in finally

Header fields fetched: FROM SUBJECT DATE MESSAGE-ID IN-REPLY-TO REFERENCES AUTO-SUBMITTED PRECEDENCE X-AUTOREPLY X-AUTORESPOND CONTENT-TYPE AUTHENTICATION-RESULTS RETURN-PATH.

Blocking imaplib runs in asyncio.to_thread, with the same deadline technique as reportmail.outgoing.deliver: socket timeouts of min(15 s, time left) and a timer that shuts the socket down after ABUSE_REPLY_POLL_DEADLINE_S (120 s) per poll.

4.3 One poll

1. Load the profile. If none, enabled=false, or the password cannot be decrypted, publish status not_configured / paused and stop.
2. gate_open() false → status egress_closed, nothing attempted (no row, no socket).
3. Connect, log in, SELECT folder. On a UIDVALIDITY change: last_uid = 0. The since date and the unique (source, uidvalidity, uid) and message_id_sha256 keys prevent re-processing.
4. search_new → at most ABUSE_REPLY_MAX_PER_POLL (50) UIDs, oldest first, and at most ABUSE_REPLY_DAILY_CAP (500) per UTC day. Any extra mail waits for the next poll.
5. fetch_headers. A header block larger than 64 KiB is skipped as oversize. Relevance from headers only (§7.1 steps 1, 2 and 5, plus “looks like a DSN”: Content-Type: multipart/report, or a From local part of mailer-daemon / postmaster). Not relevant → nothing stored, flags untouched, last_uid advances, counter ignored.
6. Relevant: check RFC822.SIZE. If it is above ABUSE_REPLY_MAX_BYTES (256 KiB), fetch only the first ABUSE_REPLY_MAX_BYTES and set truncated=true. Parse (§5), classify (§6), match (§7), store (abuse_replies), and update the outcome (§8). All of this happens in one DB transaction per message.
7. After the commit: mark_seen or move, according to the profile. If the flag or move fails, only last_error is set; the stored reply stays.
8. Commit last_uid, last_poll_at, counters. Logout.

Pasted replies (abuse_reply_imports, §11) are processed in the same worker cycle, without IMAP: same parser, classifier and matcher, source='paste'. The raw bytes are then cleared (raw = ''::bytea, status = 'done'). This works with no mailbox configured.

Error codes (report_imap_state.last_error, report_imap_requests.error_code): not_configured, paused, egress_closed, tls_failed, auth_failed, no_folder, no_move_folder, timeout, network, protocol, internal. Server answers are never stored.

5. Parsing hostile mail (ingest/ingest/replies/parse.py, pure)

Input is at most ABUSE_REPLY_MAX_BYTES of raw message. Output is a frozen ParsedReply dataclass.

Step	Rule
Parser	email.parser.BytesParser(policy=email.policy.compat32).parsebytes(raw): legacy policy, no structured header objects
Headers	≤ 200 header lines considered; each value cut at 4 KiB before decoding; email.header.decode_header in try, charset allowlist (below), errors replaced
MIME walk	Iterative, ≤ 50 parts, depth ≤ 5; stop silently beyond
Decoded parts	Only text/plain, text/html, message/delivery-status, text/rfc822-headers / message/rfc822-headers, and the header block of a message/rfc822 part. Each decoded payload is capped at 128 KiB
Attachments	Never decoded. Metadata only: {"count", "types": [≤ 10 content types], "bytes"}. File names are not stored
Charsets	utf-8, us-ascii, iso-8859-1..15, windows-1250..1258, koi8-r, gb2312, gbk, gb18030, big5, shift_jis, euc-jp, iso-2022-jp, euc-kr; anything else is decoded as UTF-8 with errors="replace"
HTML	Used only when there is no text/plain part: a html.parser.HTMLParser subclass keeps text outside script/style, ≤ 128 KiB input. Never rendered, no URL kept as a link
DSN	message/delivery-status: Action, Status (^[245]\.\d{1,3}\.\d{1,3}$), Final-Recipient (domain only). Original Message-ID and Subject read from the returned headers part
From	email.utils.parseaddr, ASCII mailbox regex as in outgoing._MAILBOX; domain lower-cased; else from_address = NULL
Auth	The topmost Authentication-Results header only (the owner’s provider adds it first): dmarc=pass header.from=<d>, dkim=pass header.d=<d>. auth = pass when aligned with the From domain (equal or a subdomain), fail on dmarc=fail, else none
Excerpt	Text body → drop lines starting with >; cut at the first line that is -----Original Message-----, ends with wrote:, or starts with From: after a blank line; NFKC; remove C0/C1 controls (except \n, \t), zero-width and bidi characters; collapse blank runs; cut at ABUSE_REPLY_EXCERPT_CHARS (2,000). Unicode is kept (non-English desks)
Subject	Decoded, controls removed, ≤ 300 chars
Message-ID	Stored only as sha256(lower(value)) for dedupe

All of this runs in asyncio.to_thread. The bounded input and structure keep the parse linear. Fuzz tests (§15) prove each limit.

6. Classification rules (ingest/rules/replies/*.yaml)

6.1 Engine

Condition nodes reuse the tagging grammar (all, any, not, leaf operators equals, in, contains, contains_any, startswith, endswith, glob, exists, gte, lte). There are no regular expressions, so a rule cannot backtrack. The engine change is compile_condition(raw, where, *, fields: frozenset[str] | None = None): when fields is given, it replaces the event/session whitelist. It also gets a public evaluate(cond, get) -> bool. Existing callers are unchanged.

Fields (whitelist REPLY_FIELDS): from_address, from_domain, from_local, subject, body (the decoded text, ≤ 8 KiB considered), auto_submitted, precedence, x_autoreply, x_autorespond, content_type, report_type, dsn_action, dsn_status, has_in_reply_to, attachment_count, matched (bool).

6.2 File format

ingest/rules/replies/20-resolved.yaml

rules:
  - id: resolved-action-taken
    outcome: resolved              # auto_ack | ticket_opened | resolved | rejected | bounce | human_reply | unknown
    priority: 20                   # lower runs first; first match wins
    description: The provider says it acted (suspended, terminated, removed, null-routed).
    when:
      any:
        - {field: body, contains_any: ["has been suspended", "has been terminated", "we have taken action",
                                       "action has been taken", "has been removed", "null-routed",
                                       "the server has been shut down", "issue has been resolved"]}
        - {field: subject, contains_any: ["resolved", "closed - action taken"]}
tickets:                           # ticket id extraction, linear scan, no regex
  - id: generic-ticket
    after: ["ticket #", "ticket#", "ticket:", "ticket id:", "case #", "case:", "incident #",
            "[#", "reference:", "ref#", "ticket number"]
    fields: [subject, body]
    min_len: 4
    max_len: 32
    chars: "A-Za-z0-9-_"            # charset of the id; a run must contain at least one digit

The loader (ingest/ingest/replies/rules.py) refuses unknown keys and outcomes, duplicate ids, priority outside 1 to 999, and more than 200 rules. It fails the worker start, not a poll.

6.3 Default rules shipped (generic, no real address)

File	Outcome	Signals (all case-insensitive, substring)
10-bounce.yaml	bounce	report_type: delivery-status with dsn_action: failed; From local part mailer-daemon/postmaster with subject undeliver, delivery status notification, returned mail, mail delivery failed
20-resolved.yaml	resolved	as in §6.2, plus customer has been notified and/service was terminated/content was removed
30-rejected.yaml	rejected	not our customer, not within our network, does not belong to, we do not host, not hosted by, please contact the, outside our jurisdiction, insufficient evidence, unable to act, please use our web form, submit via our portal (rule id form-required)
40-ticket.yaml	ticket_opened	a ticket id was extracted and (auto_submitted starts with auto- or the body contains ticket has been created, case has been opened, we have received your report)
50-auto-ack.yaml	auto_ack	auto_submitted auto-replied/auto-generated; x_autoreply exists; precedence in auto_reply, bulk, junk; subject automatic reply, auto-reply, autoreply, out of office; body this is an automated, do not reply to this; DE automatische antwort, FR réponse automatique, ES respuesta automática, NL automatisch antwoord, PT resposta automática
fallback (code)	human_reply	no rule matched, no auto header, from_domain set, body text ≥ 20 chars
fallback (code)	unknown	anything else

Ordering is bounce, resolved, rejected, ticket, auto-ack. An auto-acknowledgement that carries a ticket number is ticket_opened. A human answer that says “terminated” is resolved.

7. Matching a reply to a report (ingest/ingest/replies/match.py)

7.1 Order (first hit wins)

#	Method (match_method)	How
1	message_id	Any id in In-Reply-To / References (≤ 20 ids, each ≤ 255 chars) equals an abuse_send_requests.message_id with status = 'sent'
2	token	A 12-hex word found right after ref / ref: / ref# / (ref in the Subject, or in the body (first 8 KiB), equals an abuse_reports.reply_token
3	dsn	For a DSN, steps 1 and 2 run on the returned original headers (Message-ID, Subject)
4	ticket	The extracted ticket id equals the ticket_id of an earlier reply already linked to a report, from the same from_domain
5	sender_domain	from_domain equals the registrable domain of the recipient mailbox, override mailbox or a reply_domains entry of exactly one report sent in the last ABUSE_REPLY_MATCH_DAYS (120). More than one → unmatched
—	none	Stored only if it is a DSN or from a known recipient domain (shown under “Replies to check”); otherwise not stored at all (O4)

The token is looked up among reports with status sent or closed. A token that matches a report in another status is treated as none.

7.2 Recipient reply domains

abuse_recipients.reply_domains text[] (≤ 5). Ticket systems often answer from another domain (tickets.example.org for [email protected]). The owner adds one when confirming a reply (“Trust this domain for this provider”).

7.3 Verification (O7)

verified = true when:

• match_method = 'message_id' (or dsn through a Message-ID); or
• auth = 'pass' and from_domain (registrable) equals the recipient’s mailbox domain, override domain or one of its reply_domains; or
• the owner confirmed it (PATCH, §11).

Unverified replies are stored and shown, but do not change auto_outcome. The report’s outcome row gets needs_review = true. This covers the case where an attacker who received a forwarded report writes “resolved” to the reply address.

8. Outcomes (abuse_report_outcomes)

One row per report with status sent or closed. Ingest creates it (auto_outcome = 'none') on its first cycle after the report is sent.

Column group	Writer	Meaning
auto_outcome, reply_count, first_reply_at, first_human_at, ticket_at, resolved_at, rejected_at, bounced_at, ticket_id, needs_review, auto_updated_at	ingest	Derived from verified replies
manual_outcome, manual_at, manual_by, note	API	Owner override; manual_at is when it happened (default now, may be earlier, not before sent_at)

Effective outcome is COALESCE(manual_outcome, auto_outcome). Derivation (pure function outcome.derive(replies) -> AutoOutcome, verified replies only) uses this precedence: resolved > rejected > ticket_opened > human_reply > auto_ack > bounce > unknown > none. Delays are measured from abuse_reports.sent_at.

• No answer is not stored. The API reports it for outcome = 'none' and sent_at < now() - ABUSE_NO_ANSWER_DAYS (14).
• A bounce also sets needs_review, and the report page suggests an override mailbox for the recipient.
• Reports are not closed automatically: closing stays a decision of the owner.

9. Migration 0026_report_outcomes

-- Reply token (§2) and reference style.
ALTER TABLE abuse_reports ADD COLUMN reply_token text NOT NULL
  DEFAULT left(replace(gen_random_uuid()::text, '-', ''), 12)
  CONSTRAINT ck_abuse_reports_reply_token CHECK (reply_token ~ '^[0-9a-f]{12}$');
CREATE UNIQUE INDEX ux_abuse_reports_reply_token ON abuse_reports (reply_token);
ALTER TABLE report_settings ADD COLUMN reference_style text NOT NULL DEFAULT 'subject'
  CHECK (reference_style IN ('subject', 'subject_and_body', 'off'));
ALTER TABLE abuse_recipients ADD COLUMN reply_domains text[] NOT NULL DEFAULT '{}'
  CHECK (cardinality(reply_domains) <= 5);

-- Message-ID of SMTP sends (§2.2), written by ingest once, when it claims the request.
ALTER TABLE abuse_send_requests ADD COLUMN message_id text
  CHECK (message_id IS NULL OR char_length(message_id) <= 255);
CREATE INDEX ix_abuse_send_requests_message_id ON abuse_send_requests (message_id)
  WHERE message_id IS NOT NULL;

CREATE TABLE report_imap (
  id smallint PRIMARY KEY DEFAULT 1 CHECK (id = 1),
  enabled boolean NOT NULL DEFAULT false,
  host text NOT NULL CHECK (char_length(host) BETWEEN 4 AND 253),
  port integer NOT NULL CHECK (port BETWEEN 1 AND 65535),
  security text NOT NULL CHECK (security IN ('ssl', 'starttls')),
  username text NOT NULL CHECK (char_length(username) BETWEEN 1 AND 254),
  password_ciphertext bytea, password_nonce bytea,
  password_set boolean GENERATED ALWAYS AS (password_ciphertext IS NOT NULL) STORED,
  folder text NOT NULL DEFAULT 'INBOX' CHECK (char_length(folder) BETWEEN 1 AND 100),
  processed_action text NOT NULL DEFAULT 'mark_read' CHECK (processed_action IN ('mark_read', 'move')),
  move_folder text CHECK (move_folder IS NULL OR char_length(move_folder) BETWEEN 1 AND 100),
  poll_minutes integer NOT NULL DEFAULT 15 CHECK (poll_minutes BETWEEN 5 AND 1440),
  since date NOT NULL DEFAULT ((now() AT TIME ZONE 'UTC')::date),
  updated_at timestamptz NOT NULL DEFAULT now(),
  updated_by text CHECK (char_length(updated_by) <= 120),
  CHECK ((password_ciphertext IS NULL) = (password_nonce IS NULL)),
  CHECK (processed_action <> 'move' OR (move_folder IS NOT NULL AND move_folder <> folder)));

CREATE TABLE report_imap_state (
  id smallint PRIMARY KEY DEFAULT 1 CHECK (id = 1),
  profile_fingerprint text CHECK (profile_fingerprint ~ '^[0-9a-f]{64}$'),
  uidvalidity bigint, last_uid bigint NOT NULL DEFAULT 0,
  move_supported boolean,
  last_poll_at timestamptz, last_ok_at timestamptz,
  last_error text CHECK (char_length(last_error) <= 64),
  counters jsonb NOT NULL DEFAULT '{}' CHECK (octet_length(counters::text) <= 1024));

CREATE TABLE report_imap_requests (
  id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
  kind text NOT NULL CHECK (kind IN ('test', 'poll')),
  requested_by text NOT NULL CHECK (char_length(requested_by) <= 120),
  requested_at timestamptz NOT NULL DEFAULT now(),
  status text NOT NULL DEFAULT 'pending' CHECK (status IN ('pending', 'done', 'failed')),
  error_code text CHECK (char_length(error_code) <= 64),
  result jsonb NOT NULL DEFAULT '{}' CHECK (octet_length(result::text) <= 8192),
  done_at timestamptz,
  CHECK ((status = 'pending') = (done_at IS NULL)));

CREATE TABLE abuse_replies (
  id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
  source text NOT NULL CHECK (source IN ('imap', 'paste')),
  uidvalidity bigint, uid bigint,
  message_id_sha256 text CHECK (message_id_sha256 ~ '^[0-9a-f]{64}$'),
  received_at timestamptz NOT NULL,                 -- INTERNALDATE, or the paste time
  date_header timestamptz,
  from_address text CHECK (char_length(from_address) <= 254),
  from_domain text CHECK (char_length(from_domain) <= 253),
  subject text CHECK (char_length(subject) <= 300),
  auth text NOT NULL DEFAULT 'none' CHECK (auth IN ('pass', 'fail', 'none')),
  report_id bigint REFERENCES abuse_reports (id) ON DELETE CASCADE,
  match_method text NOT NULL CHECK (match_method IN
    ('message_id', 'token', 'dsn', 'ticket', 'sender_domain', 'manual', 'none')),
  verified boolean NOT NULL DEFAULT false,
  classification text NOT NULL CHECK (classification IN
    ('auto_ack', 'ticket_opened', 'resolved', 'rejected', 'bounce', 'human_reply', 'unknown')),
  classification_source text NOT NULL DEFAULT 'rule' CHECK (classification_source IN ('rule', 'manual')),
  rule_id text CHECK (char_length(rule_id) <= 64),
  ticket_id text CHECK (char_length(ticket_id) <= 64),
  dsn_status text CHECK (dsn_status ~ '^[245]\.[0-9]{1,3}\.[0-9]{1,3}$'),
  size_bytes integer NOT NULL CHECK (size_bytes >= 0),
  truncated boolean NOT NULL DEFAULT false,
  attachments jsonb NOT NULL DEFAULT '{}' CHECK (octet_length(attachments::text) <= 2048),
  excerpt text CHECK (excerpt IS NULL OR char_length(excerpt) <= 2000),
  excerpt_purged_at timestamptz,
  processed_at timestamptz NOT NULL DEFAULT now(),
  reviewed_at timestamptz, reviewed_by text CHECK (char_length(reviewed_by) <= 120),
  UNIQUE (source, uidvalidity, uid),
  UNIQUE (message_id_sha256));
CREATE INDEX ix_abuse_replies_report ON abuse_replies (report_id, received_at);
CREATE INDEX ix_abuse_replies_unmatched ON abuse_replies (received_at DESC) WHERE report_id IS NULL;
CREATE INDEX ix_abuse_replies_ticket ON abuse_replies (from_domain, ticket_id) WHERE ticket_id IS NOT NULL;

CREATE TABLE abuse_reply_imports (
  id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
  report_id bigint REFERENCES abuse_reports (id) ON DELETE CASCADE,  -- optional hint from the owner
  raw bytea NOT NULL CHECK (octet_length(raw) <= 262144),
  requested_by text NOT NULL CHECK (char_length(requested_by) <= 120),
  requested_at timestamptz NOT NULL DEFAULT now(),
  status text NOT NULL DEFAULT 'pending' CHECK (status IN ('pending', 'done', 'failed')),
  reply_id bigint REFERENCES abuse_replies (id) ON DELETE SET NULL,
  error_code text CHECK (char_length(error_code) <= 64),
  done_at timestamptz);

CREATE TABLE abuse_report_outcomes (
  report_id bigint PRIMARY KEY REFERENCES abuse_reports (id) ON DELETE CASCADE,
  auto_outcome text NOT NULL DEFAULT 'none' CHECK (auto_outcome IN
    ('none', 'auto_ack', 'ticket_opened', 'resolved', 'rejected', 'bounce', 'human_reply', 'unknown')),
  reply_count integer NOT NULL DEFAULT 0,
  first_reply_at timestamptz, first_human_at timestamptz, ticket_at timestamptz,
  resolved_at timestamptz, rejected_at timestamptz, bounced_at timestamptz,
  ticket_id text CHECK (char_length(ticket_id) <= 64),
  needs_review boolean NOT NULL DEFAULT false,
  auto_updated_at timestamptz,
  manual_outcome text CHECK (manual_outcome IN
    ('none', 'auto_ack', 'ticket_opened', 'resolved', 'rejected', 'bounce', 'human_reply', 'unknown')),
  manual_at timestamptz, manual_by text CHECK (char_length(manual_by) <= 120),
  note text CHECK (char_length(note) <= 1000));

Triggers and functions:

• abuse_reports_transition: copy of the 0022 function with OR NEW.reply_token <> OLD.reply_token added to the immutable identity check. Downgrade restores the 0022 body.

• abuse_send_requests_guard: copy of 0022 plus two rules. The API may not insert a message_id. message_id changes only from NULL, by ingest, in the update that sets status = 'sending', and is immutable afterwards.

• abuse_report_outcomes_guard (BEFORE INSERT OR UPDATE): the report must be sent or closed. manual_at may not be before abuse_reports.sent_at.

• suckerpot_abuse_prune (replace):

  • delete abuse_send_requests older than 30 days only when kind = 'test' or status <> 'sent' (sent send rows go with their report);
  • set excerpt = NULL, excerpt_purged_at = now() on replies older than excerpt_days (config key, default 180);
  • delete unmatched replies older than 180 days;
  • delete abuse_reply_imports that are done and older than 7 days;
  • delete report_imap_requests that are done and older than 30 days.

  suckerpot_db.retention passes excerpt_days from ABUSE_REPLY_EXCERPT_DAYS.

Models in db/suckerpot_db/models.py: ReportImapRow, ReportImapStateRow, ReportImapRequestRow, AbuseReplyRow, AbuseReplyImportRow, AbuseReportOutcomeRow, plus new columns on existing models. Advisory lock ABUSE_IMAP_PROFILE in suckerpot_db.locks: ingest holds it shared during a poll; the API PUT/DELETE takes it exclusively with a 150 s lock timeout.

10. Roles (db/suckerpot_db/roles.py)

Table	suckerpot_ingest	suckerpot_api
report_imap	SELECT (all columns incl. ciphertext)	SELECT (all but password_ciphertext, password_nonce), INSERT, UPDATE, DELETE
report_imap_state	SELECT, INSERT, UPDATE	SELECT
report_imap_requests	SELECT, UPDATE (status, error_code, result, done_at)	SELECT, INSERT
abuse_replies	SELECT, INSERT, UPDATE (report_id, match_method, verified, excerpt, excerpt_purged_at)	SELECT, UPDATE (report_id, match_method, verified, classification, classification_source, ticket_id, reviewed_at, reviewed_by)
abuse_reply_imports	SELECT, UPDATE (raw, status, reply_id, error_code, done_at)	INSERT, SELECT (all but raw)
abuse_report_outcomes	SELECT, INSERT, UPDATE (auto columns)	SELECT, INSERT (report_id, manual columns), UPDATE (manual columns, note)
abuse_recipients	unchanged	+ UPDATE (reply_domains)
abuse_send_requests	+ UPDATE (message_id)	unchanged

11. API (api/api/abuse_imap.py, api/api/abuse_replies.py, api/api/abuse_outcomes.py)

Method	Path	Who	Notes
GET	/api/v1/abuse/imap	admin	Profile without password, password_set, state (last_poll_at, last_error, counters, move_supported), last test
PUT	/api/v1/abuse/imap	admin, re-auth	Body like SmtpBody: enabled, host, port, security, username, password?, folder, processed_action, move_folder?, poll_minutes, since?, current_password. Validation §3. Audit abuse.imap.update with changed: [field names]
DELETE	/api/v1/abuse/imap	admin, re-auth	Audit abuse.imap.delete. Stored replies stay
POST	/api/v1/abuse/imap/test	admin	Inserts a test request (6 per hour). Result {ok, messages, move_supported, folders[]} or an error code. Audit abuse.imap.test
POST	/api/v1/abuse/imap/poll	admin	“Check now” (1 per minute)
GET	/api/v1/abuse/imap/requests/{id}	admin	Poll the state of a test or poll request
GET	/api/v1/abuse/replies	any role	Cursor pagination (limit ≤ 100). Filters report (public id), classification, matched (bool), needs_review, since. from_address and excerpt only for admins; viewers get from_domain
GET	/api/v1/abuse/replies/{id}	any role	Same field rule
PATCH	/api/v1/abuse/replies/{id}	admin	`{report: public_id
POST	/api/v1/abuse/replies/import	admin	multipart/form-data (file, .eml) or JSON {raw, report?}; ≤ 256 KiB; 20 per hour. Returns the import id; the UI polls GET /api/v1/abuse/replies/imports/{id}. Audit abuse.reply.import
PUT	/api/v1/abuse/reports/{public_id}/outcome	admin	{outcome, at?, note?}; 409 not_sent unless the report is sent/closed. Audit abuse.outcome.set
DELETE	/api/v1/abuse/reports/{public_id}/outcome	admin	Clears the manual columns (back to automatic). Audit abuse.outcome.clear
GET	/api/v1/abuse/outcomes	any role	Query: since, until (≤ 730 days), group = kind / asn / country / recipient. Answer: totals {sent, answered, no_answer, by_outcome{...}, first_reply_p50_s, first_reply_p90_s, resolved_p50_s}, groups[{key, label, sent, answered, by_outcome, first_reply_p50_s}] (≤ 200, sorted by sent desc)
GET	/api/v1/abuse/outcomes/timeline	any role	bucket = week / month; series sent per kind, answered, resolved

GET /api/v1/abuse/reports/{public_id} gains outcome {effective, auto, manual, needs_review, first_reply_at, ...}, replies (latest 20, same field rule), and reply_token (admins only). GET /api/v1/abuse/reports gains outcome per row. GET /api/v1/abuse/status gains replies {enabled, configured, last_poll_at, last_error, unmatched, needs_review}.

Grouping rules: a report counts once per ASN/country of its subjects (abuse_report_subjects.network_id → abuse_networks.asn, country); recipient groups by abuse_recipients.id with the recipient name. “Answered” means effective outcome not in (none, bounce) among reports sent at least 3 days before until. Demo reports are never counted.

api/api/setup_status.py gains an optional item replies (“Replies mailbox: optional, not set up”), which never blocks the checklist.

12. UI (follow web/DESIGN.md; no new hue)

• Navigation: “Abuse reports” becomes a hub (web/src/app/nav.ts) with tabs Reports (/defend/abuse) and Results (/defend/results). M27 adds Submissions.
• Settings > Reporting > Replies card (web/src/components/abuse/imap-card.tsx), admin only.
  • Opening sentence: “Replies are read from your mailbox through the VPN, every 15 minutes.” or “No mailbox: paste replies on the report page, or set one up here.”
  • Fields from §3 and a “Use a dedicated mailbox or alias” hint.
  • “Test connection” shows the folder list for the move target. “Check now”.
  • Status line: last check (relative time), last error as a sentence.
  • Reference style control (Subject / Subject and body / Off) with the fingerprint note.
• Report page (abuse-detail.tsx): an Outcome card under the stepper for sent and closed reports.
  • Opening sentence, e.g. “The provider opened ticket 48213 2 h after you sent this report.” or “No answer after 16 days.”
  • Outcome badge (icon + word); “Set outcome” (admin: select + date + note); “Back to automatic”.
  • A Replies list: date, sender domain, classification badge, verified mark (icon + word “Verified” or “Check it”), the rule description, excerpt (admin) in a monospace text block. Text only: never HTML, links not clickable.
  • “Add a reply” (admin): paste or .eml upload.
• Reports list: Sent and Closed tabs gain an Outcome column. A Replies to check view (unmatched or unverified) is reached from a banner on the Reports tab: “3 replies need your eye.” Each row offers “Link to report”, “Confirm” and “Trust this domain”.
• Results page (web/src/pages/results.tsx). Every card opens with a sentence built by web/src/lib/results-sentences.ts:
  1. Stat tiles: Reports sent, Answered (%), Median first reply, Resolved.
  2. Reports over time, e.g. “You sent 14 reports in October, 5 more than in September.” Stacked weekly bars by kind.
  3. Did anyone answer?, e.g. “9 of 23 providers answered; 4 opened a ticket and 2 said they acted.” One horizontal bar per outcome (neutral tones; “Rejected” and “Bounce” use the warning tone, never red).
  4. How fast they answer, e.g. “Half of the first answers came within 3 hours.” Delay histogram (buckets < 1 h, < 6 h, < 24 h, < 7 d, later).
  5. Providers table: provider (ASN chip or recipient name), country flag, sent, answered %, median first reply, last outcome. Sortable, filters in the URL (?group=asn&since=…). Rows open the Reports tab filtered by recipient.
  6. Empty state when nothing was sent: “No report sent yet. Results appear here once you send one.”
• Mocks: web/src/mocks/replies.ts, web/src/mocks/results.ts (documentation ranges, example.net mailboxes, documentation ASNs).

13. Configuration

Variable	Default	Used by
ABUSE_REPLIES_ENABLED	false	ingest (replies worker; also processes pasted replies)
ABUSE_REPLY_MAX_BYTES	262144 (65,536 to 1,048,576)	ingest
ABUSE_REPLY_MAX_PER_POLL, ABUSE_REPLY_DAILY_CAP	50, 500	ingest
ABUSE_REPLY_POLL_DEADLINE_S	120	ingest
ABUSE_REPLY_EXCERPT_CHARS	2000 (0 stores no excerpt)	ingest
ABUSE_REPLY_MATCH_DAYS	120	ingest
ABUSE_REPLY_EXCERPT_DAYS	180	migrate (retention)
ABUSE_NO_ANSWER_DAYS	14	api

The mailbox itself is not an environment variable: it lives in report_imap. docker-compose.yml passes the variables above (ingest, api, migrate); .env.example documents them.

14. Retention

Reply metadata lives as long as its report: sent reports are kept for ABUSE_REPORT_RETENTION_DAYS (730), and their replies and outcome rows are deleted with them (cascade). Excerpts are cleared after 180 days. Unmatched replies are deleted after 180 days. Pasted raw mail is cleared right after processing. M27’s monthly roll-up keeps the aggregates after reports expire.

15. Tests

• reporting/:
  • new_reply_token() format and uniqueness (10,000 draws).
  • Subject with {ref}, without it (suffix appended), and with off; body line only with subject_and_body.
  • The token is absent from X-ARF YAML, CSV, STIX, MISP and file names (golden tests updated).
  • The guard passes a rendered message that carries a token (clean corpus).
  • The guard fails with internal_id when a planted subject carries id_seed.hex or a session id instead of a token.
  • The token is not in legacy.py wording; test_static.py still passes.
• egress/: TunnelIMAP4 and TunnelIMAP4_SSL against the fake CONNECT proxy and a fake IMAP server in egress/tests/ (gate closed → EgressClosed before any socket; TLS verified; _MAXLINE lowered).
• ingest/ (ingest/tests/replies/):
  • parser fixtures fixtures/replies/*.eml (synthetic, example.org): plain, HTML-only, multipart with PDF attachment (never decoded), DSN with message/rfc822-headers, quoted-printable, iso-2022-jp, broken charset, 300 headers, 60 parts, nesting depth 10, a 16 MiB line (truncated input), base64 garbage, bidi and zero-width characters in Subject;
  • each fixture asserts limits and excerpt cleaning;
  • classifier: one fixture per default rule, plus precedence (auto-ack with ticket → ticket_opened; human text with “terminated” → resolved);
  • rules loader refuses a bad file;
  • matcher: each method, ambiguous sender domain → none, token of a draft report → none;
  • verification: forged reply with token from a foreign domain → stored, verified=false, outcome unchanged, needs_review=true; DMARC pass from the recipient domain → verified;
  • outcome derivation precedence;
  • worker with a fake SafeMailbox: gate closed → no connect; unrelated mail → nothing stored and no flag; relevant mail → stored then \Seen or moved; no MOVE capability → \Seen and move_supported=false; UIDVALIDITY change; daily cap; fingerprint reset;
  • paste import path with no profile.
• Static tests:
  • ingest/tests/test_egress_static.py additions (§4.1).
  • New ingest/tests/test_replies_static.py. The replies package never:
    • calls expunge, append, delete, rename, setacl, store outside SafeMailbox.mark_seen;
    • holds a string constant containing \Deleted or EXPUNGE;
    • imports smtplib, ingest.reportmail, httpx, imaplib, webbrowser;
    • calls get_payload(decode=True) outside parse.py.
• db/: upgrade and downgrade 0025 ↔ 0026; existing reports get distinct tokens; the token is immutable; the API cannot insert or write message_id; ingest cannot write manual_*; the API cannot write auto_*; no outcome row for a draft; prune keeps sent send rows and clears excerpts.
• api/: authz walk (viewer 403 on every write; GET imap admin-only); re-auth on PUT and DELETE; password never in any answer or log; viewers never see excerpt or from_address; outcome 409 on a draft; summary grouping SQL on a synthetic fixture (integration).
• web/: vitest for the sentences module (every branch, singular and plural), the imap card (write-only password, test result), outcome control, replies list read-only for viewers; screenshots 1440/390 dark/light of Results, the report Outcome card, Settings > Replies.

16. Task split

Every prompt repeats: no value from ENVIRONMENT.local.md or .env in any file; documentation ranges, documentation ASNs and example.* names only; error codes are constants and never carry server text or mail content.

Wave 1 (parallel)

A. Migration and roles (sonnet, backend-engineer)

• Owns: db/migrations/versions/0026_report_outcomes.py, db/suckerpot_db/models.py, db/suckerpot_db/roles.py, db/suckerpot_db/retention.py, db/suckerpot_db/locks.py, db/tests/**.
• Contracts: §9, §10, §14.
• Verify: just lint typecheck test, just test-integration.

B. Token, guard and IMAP adapter (sonnet, backend-engineer)

• Owns: reporting/suckerpot_reporting/{ids,message,guard}.py, reporting/tests/**, egress/suckerpot_egress/{imap,__init__}.py, egress/tests/**.
• Contracts: §2, §4.1. render_* signatures gain keyword-only reply_token=None, reference_style="subject", with defaults that keep today’s output.
• Verify: just lint typecheck test (packages reporting, egress).

C. Web (sonnet, frontend-engineer, on mocks)

• Owns:
  • web/src/app/nav.ts, web/src/app/nav.test.tsx, web/src/app/router.tsx;
  • web/src/pages/results.tsx, web/src/pages/abuse.tsx, web/src/pages/abuse-detail.tsx;
  • web/src/components/abuse/{imap-card,outcome-card,replies-list,reply-import-dialog,reporting-card}.tsx;
  • web/src/components/results/**, web/src/lib/results-sentences.ts, web/src/lib/api*;
  • web/src/mocks/{replies,results,handlers}.ts, and the tests.
• Contracts: §11 shapes, §12.
• Verify: just web-test, just web-build, just screenshots; ui-designer review.

Wave 2 (after A and B are merged)

D. Ingest replies worker (sonnet, backend-engineer)

• Owns:
  • ingest/ingest/replies/** (new), ingest/rules/replies/*.yaml (new);
  • ingest/ingest/tagging/engine.py (the fields= parameter and evaluate only);
  • ingest/ingest/reportmail/{outgoing,sender}.py (store message_id, pass the token and id_seed);
  • ingest/ingest/abuse/build.py (guard context id_seed);
  • ingest/ingest/features.py, ingest/ingest/main.py (worker wiring);
  • ingest/tests/**;
  • docker-compose.yml (ingest, api and migrate environment only), .env.example.
• Contracts: §4 to §8, O1 to O8.
• Verify: just lint typecheck test, just test-integration.

E. API (sonnet, backend-engineer)

• Owns:
  • api/api/abuse_imap.py, api/api/abuse_replies.py, api/api/abuse_outcomes.py (new);
  • api/api/abuse.py (detail, list, status fields), api/api/abuse_core.py (guard context id_seed);
  • api/api/routes.py, api/api/setup_status.py, api/tests/**.
• Contracts: §11.
• Verify: just lint typecheck test, just test-integration, just abuse-guard-check on the live stack.

Wave 3

F. Docs (haiku, docs-writer)

• docs/abuse-reports.md (replies, outcomes, the token, a dedicated mailbox).
• docs/api.md, docs/ui.md (hub and Results page), docs/configuration.md, docs/retention.md.
• docs/ARCHITECTURE.md (worker, tables, IMAP through the tunnel), docs/SECURITY.md (“Reading replies: hostile mail”).
• docs/index.md (toctree design/m26).
• The orchestrator adds the replies worker to CLAUDE.md §3’s worker list.

Reviews: security-reviewer (opus) on B, D and E together. The review hunts for:

• a way to make the worker delete, write or answer mail;
• a parser crash or blow-up from a crafted .eml;
• an unverified reply changing an outcome;
• the token leaking into an export, or derived from an internal id;
• an IMAP connection outside the tunnel.

ui-designer on C.

17. What M29 (community network) reuses

• Outcome and classification codes (auto_ack … unknown, none) and match_method values are stable constants. A node can share “provider AS64500: 12 reports, 7 answered, 3 resolved, median first reply 4 h” without any text.
• abuse_report_outcomes and the grouping keys (ASN, country, recipient mailbox domain) are the inputs of M27’s outcome_rollup_monthly, which is what a node shares.
• Never shared: excerpts, subjects, sender addresses, reply tokens, Message-IDs, ticket ids. The hub needs none of them.
• abuse_recipients.reply_domains (a provider’s ticket-system domains) is the one piece of reference data a hub could usefully pool (domain names of public abuse desks, no personal data). That would be an opt-in, M29 decision.

18. Risks and open questions

1. Proton: IMAP needs Proton Bridge. A Bridge container would be a new service with its own egress question. Out of scope; the paste path covers Proton users.
2. Mailbox not dedicated: the header-first relevance check keeps unrelated mail out of the database. The owner should still use an alias that only receives replies.
3. Authentication-Results trust: only the topmost header is read. If the owner’s provider does not add one, nothing is auto-verified except Message-ID matches. The owner confirms by hand.
4. Token shape fingerprint: public format, random value. Accepted; can be moved or turned off.
5. Auto-close: proposed no. A resolved outcome does not close the report; the owner closes it.

Review hardening (integration)

Decisions taken after the security review of the merged M26/M27 branch:

• A bad mail never stops the mailbox. A mail whose own data breaks a constraint is counted (counters.failed) and skipped; the position (last_uid) always moves past it. Only a broken connection or database stops a poll without moving it. Every stored field is clamped to its column in parse.py (sender address 254, domain 253, attachment types ASCII, at most 64 characters, attachments JSON at most 2048 bytes).
• Authentication-Results is only read from the owner’s own server. report_imap.trusted_authserv_id (Settings, optional) names the authserv-id the mailbox provider writes. Only the first header with that id counts; without it auth is always none. Clauses are split outside quoted strings and comments.
• Quoting our Message-ID verifies nothing by itself. A Message-ID match verifies only from a party we wrote to; a delivery status notification verifies with auth = pass or from such a party; every other match needs both.
• Trust is by exact domain. A subdomain is another party. Free and shared mail providers (suckerpot_db.shared_mail) are never trusted by domain, never make a mail relevant and never match by sender domain: only the exact address of the recipient’s mailbox (or an address the owner trusted) counts. The API exposes shared_mailbox on the IMAP profile (the owner’s own mailbox is on such a provider: use a dedicated mailbox) and on each abuse recipient.
• Viewers get renders and downloads without the reply token, and subjects with ref ************.
• A new IMAP host, port, user name or security needs the mailbox password again.
• Outcomes are recomputed only for reports whose replies changed since the last computation.