# Pixel Data-Loss Baseline

This baseline captures queue and insert behavior before hardening changes.

## Snapshot (2026-05-22 12:54 ET)

### 24-hour channel summary

| channel | runs | total | inserted | delta | errors |
| --- | ---: | ---: | ---: | ---: | ---: |
| articlesource | 386 | 26029 | 26029 | 0 | 14 |
| articlestats | 400 | 191127 | 191127 | 0 | 0 |
| boardstats | 400 | 55769 | 55769 | 0 | 0 |
| noidinteractions | 398 | 186702 | 177150 | 9552 | 2 |
| peoplestats | 400 | 42064 | 42064 | 0 | 0 |
| prodstats | 338 | 1322 | 1322 | 0 | 0 |
| regionalstats | 399 | 178099 | 178099 | 0 | 1 |
| sitepixel | 400 | 158430 | 158430 | 0 | 0 |
| uuidinteractions | 385 | 2858 | 2858 | 0 | 0 |

### Latest confirmed truncation event

- `articlesource` produced: `SQLSTATE[22001]: String data, right truncated: 1406 Data too long for column 'sourcevar'`.
- Seen in `process-stats.log` at `2026-05-22 12:51:02`.

### Live queue observation

- Current `articlesource.queue` contains long referrers.
- Max sampled `sourcevar` length at capture time: `305`.
- `SOURCEVAR_MAX_LENGTH` is now clamped at ingest to prevent chunk rollback from oversized values.

## Runbook checks

Use these checks during rollout and after deploys.

1. `rg "ERROR:" /home/banners/public_html/bway/pixel/logs/process-stats.log`
2. `rg "WARNING: skipped" /home/banners/public_html/bway/pixel/logs/process-stats.log`
3. `rg "deadletter=" /home/banners/public_html/bway/pixel/logs/process-stats.log`
4. `ls /home/banners/public_html/bway/pixel/queue/*.processing`
5. `ls /home/banners/public_html/bway/pixel/queue/deadletter`

## Baseline command snippets

### 24h channel totals

```bash
python - <<'PY'
from collections import defaultdict
import re
from datetime import datetime, timedelta
log='/home/banners/public_html/bway/pixel/logs/process-stats.log'
cutoff=datetime.utcnow()-timedelta(hours=24)
pat=re.compile(r'^(?P<ts>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}).*\[(?P<ch>[^\]]+)\] Inserted (?P<ins>\d+) of (?P<tot>\d+) events')
err_pat=re.compile(r'^(?P<ts>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}).*\[(?P<ch>[^\]]+)\] ERROR:')
ins=defaultdict(int);tot=defaultdict(int);runs=defaultdict(int);errs=defaultdict(int)
with open(log,'r',errors='ignore') as f:
    for line in f:
        m=pat.search(line)
        if m:
            ts=datetime.strptime(m.group('ts'),'%Y-%m-%d %H:%M:%S')
            if ts>=cutoff:
                ch=m.group('ch'); i=int(m.group('ins')); t=int(m.group('tot'))
                ins[ch]+=i; tot[ch]+=t; runs[ch]+=1
            continue
        e=err_pat.search(line)
        if e:
            ts=datetime.strptime(e.group('ts'),'%Y-%m-%d %H:%M:%S')
            if ts>=cutoff:
                errs[e.group('ch')]+=1
print('24h channel\\truns\\ttotal\\tinserted\\tdelta\\terrors')
for ch in sorted(set(list(tot)+list(errs))):
    print(f"{ch}\\t{runs[ch]}\\t{tot[ch]}\\t{ins[ch]}\\t{tot[ch]-ins[ch]}\\t{errs[ch]}")
PY
```