Engineering technical report: gghstats ecosystem v1.5.1

gghstats v1.5.1 — Honest Reporting

1. Context and design philosophy

GitHub traffic data (views and clones) is retained only in a rolling 14-day window. Without your own capture, that telemetry is gone. gghstats persists traffic history in SQLite, with a dashboard, API, CLI, and Prometheus metrics, so the organization keeps series beyond the API limit.

GitHub API 14 days vs gghstats SQLite

The design prioritizes data sovereignty and operational minimalism: a single Go binary + SQLite, no mandatory external database, self-hosted rather than a SaaS that requires handing a PAT to a third party.

Separation of responsibilities

Repository	Role
gghstats (core)	Application: GitHub sync → SQLite, `serve`, API/UI, CLI, Prometheus, opt-in alerts, GHCR image / binaries. App env vars such as `GGHSTATS_DB`, `GGHSTATS_TRUSTED_PROXIES`, `GGHSTATS_GITHUB_TOKEN`, etc.
gghstats-selfhosted (deploy)	Operator IaC: Compose (minimal / Traefik TLS / observability), Helm, `GGHSTATS_HOST_DATA` layout, image pin, VPS docs. Not the engine source.

gghstats ecosystem v1.5.1 diagram

Traefik, Helm, Grafana/Prometheus/Loki stacks, and the GGHSTATS_HOST_DATA convention belong to selfhosted. The core only ships the binary/image and the semantics of application variables (e.g. trusting X-Forwarded-* via GGHSTATS_TRUSTED_PROXIES).

2. Install and deploy

Quick evaluation (core)

Method	Command / tool	Use
Script	`curl -fsSL https://get.gghstats.com/install.sh \| sh`	Fast install on Linux/macOS (pin: `VERSION=v1.5.1 …`)
Homebrew	`brew install hrodrig/gghstats/gghstats`	Workstations
Go	`go install github.com/hrodrig/gghstats/cmd/gghstats@latest`	Source / bleeding-edge

Demo mode: gghstats serve --demo (or GGHSTATS_DEMO=true) — UI with synthetic data, no PAT and no real sync.

Production: what lives where

Core (application):

SQLite persistence: GGHSTATS_DB / --db (in container often /data/gghstats.db with a volume on /data).
Behind a reverse proxy: GGHSTATS_TRUSTED_PROXIES (CIDR/IP of the peer talking to gghstats) so rate-limit / whitelist / access log use the real client via X-Forwarded-For / X-Real-IP. Empty list = ignore those headers (safe default).
Image: ghcr.io/hrodrig/gghstats on gcr.io/distroless/static-debian13:nonroot.

Selfhosted (manifests): repo version ~0.1.58, app pin GGHSTATS_VERSION=v1.5.1.

GGHSTATS_HOST_DATA: host directory (e.g. /home/gghstats/gghstats-data) with .env, SQLite mounted at /data, and optionally .env.observability / theme CSS. Compose is invoked with --env-file "${GGHSTATS_HOST_DATA}/.env".
Compose minimal: one service, host port (e.g. 8080).
Compose Traefik: HTTPS (80/443), Let’s Encrypt (GGHSTATS_HOSTNAME, ACME_EMAIL). Edge + app rate-limit; /metrics excluded from the public router (internal scrape). Optional Authelia SSO on the gghstats_edge network.
Observability (optional, needs Traefik / gghstats_edge): Prometheus, Grafana, Loki; Compose project -p gghstats-obs; Traefik overlay for Grafana on its own FQDN.
Helm: chart gghstats from https://hrodrig.github.io/gghstats-selfhosted (or clone ./run/kubernetes/helm/gghstats); token in a Secret; PVC on /data.
Image upgrade: change GGHSTATS_VERSION → pull + up -d (restart is not enough). Helper: ./run/scripts/compose-stack.sh.

Alerts (Slack / webhook / Loki / SMTP) belong to the core (GGHSTATS_ALERTS_*); selfhosted only wires those vars in Compose/Helm and documents gghstats alert test.

3. Trajectory: v1.1.0 → v1.5.1

v1.1.0 — Pins + Featured (migration v6)

CLI gghstats repo / gghstats featured: pins union into the traffic set (FILTER ∪ pins); Featured is an editorial showcase without traffic on /.
Tables pins and featured (migration v6).
Page /featured (neo-brutalist grid); nav link only when the catalog is non-empty.

v1.2.0 — Featured UX + numbers

Count formatting (separators / GGHSTATS_COMPACT_NUMBERS).
/featured: pagination, search, and sort (page, per_page, q, sort, dir) via FilterFeatured.

v1.3.0 — Index statistics + JSONL

Daily clone statistics panel, rank/# - %, Unique line on the index chart.
JSONL export from the index (summary + history + referrers/paths/stars per filtered repo). Born here; traffic API/SQLite schema unchanged.

v1.4.0 — Uniques UX + Featured JSON + UTC stamp

U1: uniques as the primary figure; clone/view events secondary.
GET /api/v1/featured + /featured in the sitemap (when entries exist).
JSONL: Content-Disposition with UTC stamp gghstats-export-YYYYMMDD-HHMM.jsonl (#23). Carlok fixes #22–#25 (locale charts, tooltips, Rank).

v1.5.0 — Five reporting slices

Slice	Content
F-fresh	Per-metric freshness / coverage (`views` / `clones`). Statuses: `fresh` \| `delayed` \| `missing` \| `failed` \| `never`. Charts use `null` gaps (explicit zero ≠ omitted day). Independent views/clones fetch. Migration v7 (`traffic_metric_state`, `traffic_metric_coverage`).
V-vis	`github_visibility` + `report_policy` (`exclude > include > inherit`); `GGHSTATS_REPORT_PRIVATE`; CLI `repo report ls/set`. Report surfaces fail-closed (indistinguishable 404). Collection (filter/pins/`INCLUDE_PRIVATE`) separate from reporting. Migration v8.
V-json	`gghstats repo report ls --json` (+ `--visibility` / `--policy` filters) for post-upgrade scripts.
U-legend	i18n legend on detail charts: gap/`null` = not reported by GitHub; `0` = confirmed zero.
X-chart	Chart-aligned JSON download (`/{owner}/{repo}/traffic.json`); dogfood `…/traffic?dense=1` (default sparse unchanged).

Operators: after 1.5.0, existing rows → unknown + inherit until the next sync (or repo report set … include). SQLite history is not deleted; dashboard/APIs/exports/badges can look empty until then.

v1.5.1 — Featured fix (#47)

In 1.5.0, Featured was incorrectly treated as a report surface: /featured, API, nav, and sitemap required a collected repos row and reportVisible. That breaks the editorial showcase (a Featured entry ≠ a traffic-collected repo).

1.5.1 restores the catalog: FilterFeatured / FeaturedCount again. Report visibility still applies only to traffic/dashboard routes. No sync needed for the vitrine. This is not a relaxation of dashboard fail-closed; the bug was Featured’s wrong scope.

4. Data, security, and supply chain

Relevant SQLite migrations

Version	Content
v6	`pins`, `featured`
v7	Traffic freshness / coverage
v8	`repos.github_visibility`, `repos.report_policy`

Opened with WAL for concurrent reads (UI/API) while sync writes.

Defense in depth (core)

Token-bucket rate limiting per IP; optional IP whitelist; GGHSTATS_API_ONLY.
GGHSTATS_TRUSTED_PROXIES (see §2).
Cosign + SBOM on releases; test coverage ≥ 80% in release-check.
Distroless nonroot image.

Alerts

Opt-in (GGHSTATS_ALERTS_ENABLED): Slack / webhook / Loki / SMTP sinks; rules after sync; validate with gghstats alert test. Grafana/Loki stacks in observability Compose are selfhosted deploy, distinct from the app’s Loki sink.

5. Closing

v1.5.1 consolidates the 1.5 line: honest freshness (five statuses + null gaps), fail-closed report visibility with migrations v7/v8, and the fix that Featured is an editorial catalog, not report scope. The engine stays in gghstats; Traefik, Helm, GGHSTATS_HOST_DATA, and observability stacks live in gghstats-selfhosted, with image pin v1.5.1.

References: CHANGELOG · SPEC · core README · gghstats-selfhosted · install https://get.gghstats.com/install.sh