datarena.app

datarena is the side project I keep coming back to. It’s where Gavin He and I get to own a data product end-to-end - ingestion, modeling, ML, and the app. It’s public NHL analytics, but really it’s a sandbox: a place to slow down and build things carefully, the way it’s hard to find time for under a deadline.

How good can public hockey analytics get? This is me trying to find out.

Table of Contents

• The short version
• The 2026 rebuild
• The stack, end to end
• How it gets built and shipped
• Under the hood
• Try it

The short version

The app surfaces 0–100 player ratings (à la JFresh) - even-strength offense and defense, powerplay, penalty-kill, finishing, discipline - plus per-player expected-goals heatmaps and season trends. Pick a player, see where they create and suppress goals, and how that’s trended.

Underneath the friendly dials is the part I actually care about: a calibrated expected-goals (xG) model scoring every unblocked shot since 2013-14, fed by a clean dbt warehouse, fed by reproducible ingestion pipelines. Four small repos, each doing one job well, connected by explicit data contracts.

The 2026 rebuild

The original version (the 2022 data-engineering write-up) ran on Meltano/Singer and BigQuery, hosted on Heroku. It worked, but it was expensive and brittle for a hobby project, and the modeling never got past a few notebooks.

So in 2026 we tore it down and rebuilt it as four focused repos, with two goals: make it nearly free to run, and build it the way I’d build it at work - versioned, tested, reproducible, and honest about what it does and doesn’t do.

What changed:

The whole platform now runs on my laptop for free and deploys to the cloud for the price of a coffee per month. That constraint - make it cheap and reproducible - drove most of the architectural decisions.

The stack, end to end

Four repos, one direction of data flow. Each owns a single responsibility and hands off through a documented contract:

datarena-dlt  →  datarena-dbt  →  datarena-ml  →  datarena-app
 ingest NHL       DuckDB marts     calibrated      FastAPI + Next.js:
 API → DuckDB     (dims, facts,    per-season xG   typed API + web,
                  fct_shot, xG)    on every shot   served on Cloud Run

datarena-dlt - ingestion. Pulls the NHL Stats API into a local DuckDB warehouse with dlt. Seven pipelines, each owning exactly one grain - teams, games, play-by-play, shifts, rosters, and skater/goalie season stats - with schema validation and separate dev/prod datasets. No duplicated raw sources: everything downstream derives from these. Zero credentials, zero cloud cost.

datarena-dbt - modeling. Turns the raw warehouse into clean dimensions and facts with dbt-duckdb. The centerpiece is fct_shot, the shot-level training table for xG (one row per unblocked - Fenwick - shot, with net-normalized geometry, sequence context, and leak-safe pre-shot game state). Every grain key gets unique + not_null tests, enums get accepted_values, facts get relationships back to dims, SQL is linted with SQLFluff, and changes follow SemVer with a changelog. I wanted it to read like a small production warehouse, not weekend SQL.

datarena-ml - the modeling. A calibrated XGBoost expected-goals model, packaged properly and tracked in MLflow. This is the heart of the project, so it gets its own deep-dive - calibration-first design, temporal validation, and walk-forward scoring across 13 seasons.

datarena-app - the product. Two stateless services on Cloud Run: a typed FastAPI backend (api.datarena.app) and a mobile-first Next.js frontend (datarena.app). Neither touches the warehouse at runtime - a refresh step bakes a tiny (~5 MB) DuckDB snapshot into the API image, which answers everything as JSON; the heavier shot-plot images sit in a private cloud bucket and stream through the API, keeping the container small and cold-starts fast. Each service ships on its own path-filtered, keyless CI/CD pipeline (GitHub Actions + Workload Identity Federation - no stored secrets).

How it gets built and shipped

The workflow is deliberately the same one I’d want on a real team - just scaled down to two people and a free budget.

1. Build locally, for free. The entire platform runs on a laptop. dlt pulls the NHL API into a local DuckDB file, dbt builds the marts into that same file, the ML package trains and scores against it, and the app’s API runs from a baked snapshot of it. No cloud account needed to develop - DuckDB is the great equalizer here. Each repo has its own conda/uv (or npm) environment and a one-liner you can run end-to-end - dbt build, train, or uvicorn + npm run dev - before touching git.

2. Branch → PR → main. Nothing lands directly on main. Work happens on a feature branch (feature/xg_foundation, revival/modernize-xg, …), with Conventional Commits and SemVer so the changelog more or less writes itself. A PR opens against main, where the guardrails run: pre-commit hooks, SQLFluff on the dbt models, the ML/data tests, and a human review (Gavin or me). After merge, releases get tagged on main.

3. Merge deploys itself. For the app, a push to main triggers GitHub Actions, which builds and deploys to Cloud Run - and the API and web services deploy independently via path-filtered workflows, so a frontend tweak never rebuilds the backend. It’s keyless, too: Workload Identity Federation means no long-lived cloud credentials sit in a secret anywhere. No manual gcloud step, no clicking around a console.

4. Data and models ship on their own cadence. Code and data deploy separately on purpose. When the model re-scores a season, a refresh step re-bakes the app’s snapshot; when the model itself changes, it’s registered in MLflow and only goes live when I explicitly promote its alias to production. So a retrain never silently changes what users see, and the app image stays tiny and stateless.

The throughline: a change is cheap to make, hard to break, and boring to ship - which is exactly what you want, even (especially) for a side project.

Under the hood

A few decisions I’m proud of - the kind of thing that doesn’t show up in the UI but is the actual work:

• Explicit contracts between repos. Each handoff (dlt→dbt, dbt→ml, ml→app) has a documented data contract, so a change upstream can’t silently break a consumer. This is the single habit that’s saved me the most pain.
• Typed all the way to the browser. The FastAPI backend publishes an OpenAPI schema the Next.js frontend builds against, so types flow across the Python→TypeScript boundary instead of breaking silently at it. Static images route through the API too - which leaves a clean seam to gate them behind access controls later, without changing a single client URL.
• Calibration over vanity metrics. An xG of 0.2 has to mean a ~20% chance of a goal, because xG gets summed across players and games. We optimize for proper scoring and calibration (ECE ≈ 0.005), not just ROC-AUC. More in the model write-up.
• Leakage-safe by construction. Temporal train/validation/test splits (never random), features audited for label leakage, and per-season walk-forward scoring so an old season is never scored with future information.
• Cost as a design constraint. Local-first DuckDB, baked snapshots, scale-to-zero. The platform is essentially free, and that forced cleaner architecture than a cloud budget ever would have.

Is it perfect? No - and I’d rather be honest about it. xG coverage starts at 2013-14 (the NHL didn’t emit the strength data we need before then), and contract/cap data is on hold since CapFriendly shut down in 2024 - it sits behind a seam, ready for a live source to drop in. Real constraints, all documented.

Try it

The app is live at datarena.app - and it’s backed by a public, typed API you can poke at api.datarena.app/docs:

If any of this is your kind of thing - sports analytics, calibrated probability models, lean data stacks - I’d genuinely love to connect. And if you want the gory ML details, head to the xG model deep-dive.

Developers: @dmf95 · @gavh3