Shadowbroker/docs/OUTBOUND_DATA.md

Outbound data and third-party exposure

Shadowbroker is self-hosted: each install uses its own backend egress IP. This document is the operator-facing record for GitHub audit issues #348–#366 (tg12): what contacts third parties, why, and how to opt out without losing unrelated features.

Architecture

Path	Who calls third parties
Map UI → /api/* → fetchers	This install’s backend
Basemap tiles / fonts	Operator’s browser (CARTO, demotiles.maplibre.org)
CCTV still/video proxy	Backend (Referer/Origin set per agency — see #349)

---

Issue disposition summary

Issue	Status	Approach
#351	Fixed	Region dossier via backend proxy
#352	Fixed	Geocode via /api/geocode only
#360	Fixed	Wikipedia/Wikidata via backend
#362	Fixed	DEEPSTATE_MIRROR_COMMIT optional pin
#363	Fixed	Madrid KML HTTPS-first
#364	Fixed	KiwiSDR HTTPS-first + validation
#348	Accepted + gated	Windows UI opt-in; env override; stealth documented
#349	Accepted + documented	Agency-required Referer on backend proxy only
#350	Mitigated	Callsign in UA off by default; opt-in MESHTASTIC_SEND_CALLSIGN_HEADER=true
#354	Accepted + documented	Default basemap CDN; optional self-hosted tiles
#361	Mitigated	UA is install handle only (operator-…), not shared Shadowbroker/ token
#366	Accepted + documented	Honest per-install scrape; feature degrades if blocked

---

Per-install User-Agent (#361)

• Code: backend/services/network_utils.py — outbound_user_agent(), OPERATOR_HANDLE
• Sent: operator-7f3a92 or your-handle (purpose: nominatim) — no shared app product name
• Why: Upstreams can rate-limit one install; a block on operator-abc123 does not require blocking every Shadowbroker user
• Override: SHADOWBROKER_USER_AGENT replaces the entire string
• Note: The same handle across Wikipedia, Broadcastify, etc. still correlates your traffic across those sites — that is intentional per-install attribution, not anonymity

---

LiveUAMap scraper (#348)

• Layer: global_incidents (LiveUAMap map pins; GDELT text still loads without LiveUAMap)
• Code: backend/services/liveuamap_scraper.py (Playwright + stealth for Turnstile)
• Windows: Scraper off until you enable Global Incidents and confirm the UI dialog → backend/data/liveuamap_scraper_opt_in.json
• Linux/macOS: Scraper runs when the layer is on (unless env forces off)
• API: GET /api/liveuamap/scraper-status, POST /api/liveuamap/scraper-opt-in
• Env: SHADOWBROKER_ENABLE_LIVEUAMAP_SCRAPER=true|false overrides UI on all platforms
• Honesty: Backend-only; no browser-direct LiveUAMap from end users. Stealth remains a functional tradeoff for Turnstile; disable layer or env if unacceptable

UAP sightings (NUFORC map layer)

• Window: last 60 days (NUFORC_RECENT_DAYS, ~2 months) from live nuforc.org
• Cadence: Weekly (Monday 12:00 UTC) per install; typical yield ~400–500 geocoded pins
• Between weeks: backend/data/nuforc_recent_sightings.json (7-day TTL) so restarts do not wipe the layer
• Immediate pull: admin GET /api/refresh on that install
• Not used for map pins: stale Hugging Face mirror (frozen ~2023) unless live is down and mirror happens to have in-window rows

---

CCTV proxy Referer / Origin (#349)

• Code: backend/routers/cctv.py, backend/main.py
• Behavior: Backend proxies streams and sets Referer / Origin each agency expects (e.g. https://511ga.org/cctv, https://informo.madrid.es/)
• Exposure: Agency sees backend IP, not each viewer’s browser
• Not removed: Without these headers, most public DOT/city feeds return 403 — this is not end-user browser impersonation, it is the same headers a normal browser session would send to play the feed

---

Meshtastic map callsign (#350)

• Layer: sigint_meshtastic must be active for fetch_meshtastic_nodes()
• Default: MESHTASTIC_SEND_CALLSIGN_HEADER=false — callsign not sent to meshtastic.liamcottle.net unless you set true
• Optional: MESHTASTIC_OPERATOR_CALLSIGN for local display; header only when explicitly enabled

---

Basemap CDN (#354)

• Code: frontend/src/components/map/styles/mapStyles.ts, frontend/public/map-style.json
• Hosts: *.basemaps.cartocdn.com, demotiles.maplibre.org
• Exposure: Browser loads tiles (client IP + pan/zoom), not the backend
• Mitigation: Self-host raster tiles and point MapLibre sources at your tile server (operator choice; not required for core features)

---

Broadcastify top feeds (#366)

• Code: backend/services/radio_intercept.py
• Behavior: Backend fetches https://www.broadcastify.com/listen/top with per-install handle UA; parses public HTML for feed metadata and CDN stream URLs
• Exposure: Your backend IP; 5-minute cache
• If blocked: Panel shows empty list — feature not removed from the app
• Not: Fake Chrome UA or cloudscraper bypass (removed in Round 7a)

---

Ukraine frontline mirror (#362)

• Layer: ukraine_frontline / frontlines
• Pin: DEEPSTATE_MIRROR_COMMIT, optional DEEPSTATE_MIRROR_REPO

Madrid CCTV (#363) / KiwiSDR (#364)

• Madrid: HTTPS-first KML catalog; image URLs unchanged
• KiwiSDR: HTTPS-first directory fetch; shape validation + bundled fallback

---

Operator checklist

1. Set OPERATOR_HANDLE if you want a recognizable name on upstream logs.
2. Pin DEEPSTATE_MIRROR_COMMIT for reproducible frontlines (optional).
3. Windows: enable Global Incidents in UI only if you accept LiveUAMap server contact.
4. Set SHADOWBROKER_ENABLE_LIVEUAMAP_SCRAPER=false to forbid LiveUAMap entirely.
5. Set MESHTASTIC_SEND_CALLSIGN_HEADER=true only if you want callsign sent upstream.
6. Self-host map tiles if basemap CDN exposure matters.