Sportpark Visitor Analytics

A hobby project exploring how far you can get building a full-stack application from a single specification document — with AI assistance and a few manual tweaks.

The app scrapes live visitor counts from a fitness club, stores historical data, and provides an analytics dashboard with occupancy trends, personalized recommendations, and ML-based visitor predictions.

What's in here

Component	Tech	Purpose
Scraper	Python + APScheduler	Polls visitor count + weather data on a schedule
Backend	FastAPI	REST API for analytics, recommendations, ML training & prediction
Frontend	React + Recharts	Dashboard with live status, charts, heatmap, predictions
ML Pipeline	scikit-learn, XGBoost, LightGBM, Optuna	Visitor count forecasting + realtime ghost correction
Database	PostgreSQL	Time-series storage for visitor counts and weather
Databricks Sync	Python + APScheduler	Nightly incremental export of visitor counts to Databricks
Deployment	Docker Compose	Single-command local deploy

How it was built

The entire application — scraper, backend API, ML pipeline, frontend dashboard, database schema, and Docker setup — was generated from SPECIFICATION.md using AI coding tools. Manual additions were limited to tweaks and iterative refinements.

Quick start

cp .env.example .env
# Edit .env with your API URL and a strong database password
docker compose up -d

The dashboard is available at http://localhost:18080 and the API at http://localhost:18432.

Databricks sync

The databricks-sync service pushes newly scraped visitor counts to a Databricks Free Edition SQL warehouse once a night (03:15 Europe/Berlin by default), so the data can be analysed with Databricks SQL and notebooks. Only visitor_counts is exported — weather data stays in Postgres.

Each run stamps its rows with an incrementing snapshot_id, so you can always tell which load brought a row in:

SELECT snapshot_id, COUNT(*), MIN(`timestamp`), MAX(`timestamp`), MAX(loaded_at)
FROM workspace.sportpark.visitor_counts
GROUP BY snapshot_id ORDER BY snapshot_id;

The watermark lives in Databricks itself (MAX(id) on the target table), so there is no local state to drift: dropping the table on the Databricks side simply re-triggers a full backfill, and a run that fails halfway resumes exactly where it stopped.

Setup: create a Free Edition workspace, start its SQL warehouse and copy the warehouse ID, generate a personal access token under Settings → Developer → Access tokens, then fill in DATABRICKS_HOST, DATABRICKS_TOKEN, and DATABRICKS_WAREHOUSE_ID in .env.

The container syncs once on startup before scheduling the nightly run, so the initial backfill happens as soon as you bring it up — no manual step. To run a one-off sync without starting the scheduler:

docker compose run --rm databricks-sync python main.py --once

Configuration

See .env.example for all available environment variables.

Project specification

The full specification that this project was built from is in SPECIFICATION.md.