first_commit

docs/architecture-principles.md

@@ -0,0 +1,71 @@
+# Architecture Principles
+
+## Standalone Subsystems
+
+Each class should behave like a small standalone subsystem. It should own one clear responsibility, expose a narrow public interface, and avoid hidden dependencies on the internals of other classes.
+
+Good subsystem boundaries:
+- accept explicit inputs
+- return explicit outputs
+- keep internal state private
+- avoid reaching into global state
+- avoid performing unrelated work
+- can be tested with recorded or fixture data
+
+Examples:
+- a weather client fetches forecast data
+- a weather parser converts API payloads into forecast points
+- a weather builder normalizes external forecast records for storage
+- a storage class persists records
+- Gibil makes decisions from snapshots
+
+## Data Models Between Subsystems
+
+Subsystems should communicate through shared data models rather than through source-specific payloads.
+
+For example:
+- Open-Meteo JSON should become `WeatherForecastRun`
+- Modbus register reads should become `Observation`
+- HASS entity state should become `Observation`
+- Gibil should reason from `Snapshot`
+
+This keeps the edges messy and the core calm.
+
+## Side Effects At The Edges
+
+Network calls, database writes, MQTT publishes, and filesystem writes should live at clear boundaries.
+
+Core reasoning classes should generally be pure or nearly pure:
+- input data in
+- answer out
+- no surprise I/O
+
+Stateful classes are allowed, but their state should be deliberate and inspectable.
+
+## Grow By Composition
+
+Astrape should grow by connecting small subsystems together, not by building one large object that knows everything.
+
+The desired shape is:
+
+```text
+source client -> parser -> model -> storage -> query/snapshot -> Gibil -> publisher
+```
+
+Each part should be replaceable without rewriting the others.
+
+## Prefer Working Slices
+
+Build one thin working path at a time. A thin slice may start with empty storage or recorded source data, but it should still follow the real subsystem boundaries.
+
+For example, the weather slice can start with:
+
+```text
+Open-Meteo forecast run -> WeatherBuilder -> clean forecast records
+```
+
+Then grow into:
+
+```text
+Open-Meteo -> parser -> WeatherBuilder -> TimescaleDB -> weather_predictor.py
+```

docs/ingestion-and-storage.md

@@ -0,0 +1,256 @@
+# Ingestion & Storage
+
+## Purpose
+
+Astrape needs a reliable way to collect energy-related data, normalize it, store it, and give Gibil a clean view of the current system state. The first version should favor boring, inspectable data flows over cleverness.
+
+Gibil should not need to know whether a value came from Modbus, Home Assistant, a weather API, a price API, or a manual override. It should receive timestamped observations and snapshots with enough metadata to decide whether the data is fresh and trustworthy.
+
+## Initial Sources
+
+### Sigen Inverter
+
+- Protocol: Modbus TCP
+- Polling target: every 5-10 seconds for fast-changing electrical state
+- Initial metrics:
+  - `solar_power_w`
+  - `battery_soc_pct`
+  - `battery_charge_w`
+  - `battery_discharge_w`
+  - `grid_import_w`
+  - `grid_export_w`
+  - `daily_yield_kwh`
+- Risk: register map must be confirmed before this can be real
+
+### Home Assistant / Ganymede
+
+- Preferred integration: MQTT
+- Direction: HASS/Ganymede should publish selected state to Astrape where possible
+- Initial metrics:
+  - `home_power_w`
+  - `indoor_temp_c`
+  - selected device states
+  - selected sensor values needed for water/heating logic
+- Reasoning: MQTT keeps Astrape loosely coupled and avoids making HASS a synchronous dependency for every decision tick
+
+### Weather
+
+- Preferred first source: OpenMeteo
+- Polling target: hourly forecast refresh
+- Initial metrics:
+  - `outdoor_temp_c`
+  - `cloud_cover_pct`
+  - `ghi_w_m2`
+  - `wind_speed_m_s`
+- Use: external forecast history for generation and heating models
+
+### Grid Pricing
+
+- First implementation: static time-of-use config
+- Later implementation: spot pricing API if needed
+- Initial metrics:
+  - `grid_price_per_kwh`
+  - `price_stage`
+  - `cheap_window_active`
+- Reasoning: static config lets Gibil produce useful behavior before price API work is settled
+
+### Manual Inputs
+
+- Purpose: allow operator-supplied values when a real integration is not available yet
+- Inputs may come from local config or a small authenticated admin path
+- Manual data should be marked clearly with `source = manual`
+
+## Observation Shape
+
+Every collector should produce normalized observations.
+
+```text
+observed_at: timestamp when the measurement was true
+received_at: timestamp when Astrape received it
+source: sigen | hass | weather | price | manual
+metric: stable metric name
+value: number, string, or boolean
+unit: W | kWh | pct | C | SEK/kWh | state | none
+quality: ok | stale | estimated | missing | error
+metadata: source-specific context
+```
+
+Guidelines:
+- `observed_at` and `received_at` are both needed because pushed data may arrive late
+- metric names should be stable and boring
+- raw source names/registers/entities belong in metadata, not in the metric name
+- Gibil should be able to ignore stale or low-quality observations
+
+## Derived Snapshots
+
+Gibil should reason from snapshots, not directly from loose individual observations.
+
+A snapshot is the best-known whole-system state at a decision tick. It can include:
+
+- current solar generation
+- current home consumption
+- battery SoC
+- battery charge/discharge power
+- grid import/export
+- current price stage
+- active forecast window
+- stale/missing input flags
+
+Snapshots should be persisted because they explain what Gibil knew when it made a decision.
+
+## Storage Choice
+
+Use TimescaleDB as the first primary store.
+
+Reasons:
+- It is Postgres, so querying and joining data stays straightforward
+- It handles time-series retention and aggregation well
+- It works for raw observations, derived snapshots, decisions, forecasts, and events
+- It leaves room for later model training without needing a second historical store immediately
+
+InfluxDB remains a reasonable alternative, but TimescaleDB is the better default if we want relational joins, auditability, and forecast training queries.
+
+The runtime expects `ASTRAPE_DATABASE_URL` to point at TimescaleDB. Weather ingest also expects `ASTRAPE_LATITUDE` and `ASTRAPE_LONGITUDE`.
+
+## Initial Tables
+
+### `observations`
+
+Raw normalized metric samples from all collectors.
+
+Core fields:
+- `id`
+- `observed_at`
+- `received_at`
+- `source`
+- `metric`
+- `value_num`
+- `value_text`
+- `value_bool`
+- `unit`
+- `quality`
+- `metadata`
+
+Notes:
+- use one value column based on the metric type
+- keep metadata as JSON for source-specific details
+- make this a hypertable on `observed_at`
+
+### `snapshots`
+
+Periodic whole-system state used by Gibil.
+
+Core fields:
+- `id`
+- `created_at`
+- `snapshot`
+- `input_quality`
+
+Notes:
+- store the snapshot as JSON initially
+- this can be normalized later if query patterns demand it
+
+### `decisions`
+
+Gibil outputs and reasoning.
+
+Core fields:
+- `id`
+- `created_at`
+- `snapshot_id`
+- `stage`
+- `recommendations`
+- `reasons`
+- `confidence`
+
+Notes:
+- decisions should be explainable enough to debug after the fact
+- this table becomes the audit trail for HASS-facing behavior
+
+### `weather_forecast_points`
+
+Clean external weather forecast points from weather sources.
+
+Core fields:
+- `id`
+- `issued_at`
+- `target_at`
+- `horizon_hours`
+- `source`
+- `temperature_c`
+- `shortwave_radiation_w_m2`
+- `cloud_cover_pct`
+
+Notes:
+- this stores external forecasts, not internal predictions
+- make this a hypertable on `target_at`
+
+### `weather_resolved_truth`
+
+Observed weather for target hours that have already happened.
+
+Core fields:
+- `id`
+- `resolved_at`
+- `source`
+- `temperature_c`
+- `shortwave_radiation_w_m2`
+
+Notes:
+- future prediction modules can join this to `weather_forecast_points`
+- make this a hypertable on `resolved_at`
+
+### `system_events`
+
+Operational events from collectors, storage, Gibil, and publishers.
+
+Core fields:
+- `id`
+- `created_at`
+- `component`
+- `severity`
+- `event_type`
+- `message`
+- `metadata`
+
+Notes:
+- this should capture stale data, auth failures, bad Modbus reads, publish failures, and degraded-mode decisions
+
+## Retention
+
+Initial retention targets:
+- raw 5-10 second observations: 7-30 days
+- 1-minute aggregates: 6-12 months
+- 15-minute/hourly aggregates: keep indefinitely unless storage becomes a problem
+- decisions: keep indefinitely
+- system events: keep indefinitely or archive after a year
+
+Retention should be revisited after real sample rates and database size are known.
+
+## First Slice
+
+The first implementation slice should prove the shape before touching real hardware.
+
+1. Define the observation and snapshot models.
+2. Add a manual collector only if needed for operator-supplied values.
+3. Store observations in TimescaleDB or a local development substitute.
+4. Build one snapshot from the latest observations.
+5. Let Gibil make a simple stage decision from that snapshot.
+6. Persist the decision with reasons.
+
+This gives us the whole loop:
+
+```text
+collector -> observations -> snapshot -> Gibil decision -> stored audit trail
+```
+
+MQTT publishing can come immediately after this loop exists.
+
+## Open Questions
+
+- Should development use real TimescaleDB from day one, or SQLite/Postgres first?
+- What is the exact MQTT topic namespace for HASS/Ganymede integration?
+- Which HASS entities should be included in the first read-only state feed?
+- How should the `gibil` IPA identity authenticate to MQTT and HASS?
+- What high-resolution retention target is acceptable on the Astrape VM?
+- Should snapshots be created on a fixed schedule, on new data, or both?

docs/operations.md

@@ -0,0 +1,105 @@
+# Operations
+
+## Web UI
+
+Start the web UI daemon:
+
+```bash
+python3 -m gibil.scripts.web_daemon
+```
+
+The daemon listens on:
+
+```text
+http://0.0.0.0:8080
+```
+
+By default the server binds to all network interfaces so it can be reached from another machine. Override the bind address or port if needed:
+
+```bash
+export ASTRAPE_WEB_HOST='0.0.0.0'
+export ASTRAPE_WEB_PORT='8080'
+```
+
+The host process reloads `webui.py` and display modules on each request. The browser polls `/api/ui-version` and refreshes when those files change.
+
+## Systemd Services
+
+Install service units:
+
+```bash
+sudo cp deploy/systemd/astrape-web.service /etc/systemd/system/
+sudo cp deploy/systemd/astrape-db.service /etc/systemd/system/
+sudo systemctl daemon-reload
+sudo systemctl enable --now astrape-web.service astrape-db.service
+```
+
+Check status:
+
+```bash
+systemctl status astrape-web.service
+systemctl status astrape-db.service
+journalctl -u astrape-web.service -f
+journalctl -u astrape-db.service -f
+```
+
+Both services run as the IPA-managed `gibil` user from `/mnt/astrape`.
+
+## Database Daemon
+
+Install runtime dependencies:
+
+```bash
+python3 -m pip install -r requirements.txt
+```
+
+Create a local env file:
+
+```bash
+cp env/astrape.env.example env/astrape.env
+nano env/astrape.env
+```
+
+Required values:
+
+```text
+ASTRAPE_DATABASE_URL=postgresql://USER:PASSWORD@HOST:PORT/DBNAME
+ASTRAPE_LATITUDE=59.0000
+ASTRAPE_LONGITUDE=18.0000
+```
+
+Optional values:
+
+```text
+ASTRAPE_WEATHER_FORECAST_HOURS=48
+ASTRAPE_WEATHER_POLL_SECONDS=3600
+ASTRAPE_WEATHER_TRUTH_LOOKBACK_DAYS=14
+ASTRAPE_WEATHER_TRUTH_END_DELAY_DAYS=5
+```
+
+The daemons load `env/*.env` automatically. Existing process environment variables win over file values.
+
+For temporary frontend tuning, enable display-only sample weather data:
+
+```text
+ASTRAPE_WEB_SAMPLE_DATA=1
+```
+
+This does not write artificial data to TimescaleDB. It only changes the web UI weather API response.
+
+Start the database ingest daemon:
+
+```bash
+python3 -m gibil.scripts.db_daemon
+```
+
+Current behavior:
+- initializes TimescaleDB weather tables
+- fetches real Open-Meteo hourly forecasts
+- normalizes them through `WeatherBuilder`
+- stores rows in `weather_forecast_points`
+- fetches Open-Meteo archive data for resolved truth
+- stores rows in `weather_resolved_truth`
+- repeats every `ASTRAPE_WEATHER_POLL_SECONDS`
+
+No internal weather predictions are generated here. This daemon only stores external forecast and resolved-truth data for later modules.

docs/weather-source-data.md

@@ -0,0 +1,117 @@
+# Weather Source Data
+
+## Goal
+
+This subsystem aggregates external weather forecasts and stores them in a clean database-ready shape.
+
+Terminology:
+- **forecast**: data from an external weather source, such as Open-Meteo
+- **resolved truth**: observed weather for a time that has already happened
+- **prediction**: an internal estimate produced by a future Astrape/Gibil model
+
+This module should not produce predictions or confidence scores. A later `weather_predictor.py` subsystem can use this clean forecast database to produce predictions and confidence.
+
+## Subsystem Boundary
+
+Initial classes should stay narrowly scoped:
+
+- `OpenMeteoClient`: fetch raw hourly forecast payloads
+- `OpenMeteoParser`: convert API payloads into external forecast runs and points
+- `WeatherBuilder`: normalize and select clean forecast records for database use
+- `WeatherStore`: persist forecast points and resolved truth
+
+These classes communicate through data models like `WeatherForecastRun`, `WeatherForecastPoint`, and `WeatherResolvedTruth`.
+
+## Core Data Shape
+
+Every weather API pull is a forecast run.
+
+```text
+issued_at = when the external forecast was fetched
+target_at = the hour being forecast
+horizon_hours = target_at - issued_at
+forecast_value = external forecast value for that target hour
+```
+
+Later, when `target_at` is in the past, Astrape can attach resolved truth:
+
+```text
+resolved_at = the hour that actually happened
+truth = observed temperature / observed solar radiation
+```
+
+That creates rows future modules can use:
+
+```text
+target_at | resolved_truth | forecast_1h | forecast_2h | ... | forecast_48h
+```
+
+The future predictor can learn from those rows without needing to know anything about Open-Meteo payloads.
+
+## First Variables
+
+Use Open-Meteo hourly forecast fields:
+
+- `temperature_2m`
+- `shortwave_radiation`
+- `cloud_cover`
+
+Open-Meteo documents `shortwave_radiation` as average incoming solar radiation over the preceding hour at the surface, equivalent to GHI, measured in W/m2. That is the right starting solar forecast variable for Astrape.
+
+## Storage Shape
+
+Forecast points should be stored as individual rows.
+
+Core fields:
+- `issued_at`
+- `target_at`
+- `horizon_hours`
+- `source`
+- `temperature_c`
+- `shortwave_radiation_w_m2`
+- `cloud_cover_pct`
+
+Resolved truth should be stored separately. For now, resolved truth comes from the Open-Meteo historical archive API.
+
+Until archive data is available, Astrape can also store the current 0-hour Open-Meteo forecast as provisional truth with `source = open_meteo_zero_hour`. This gives the UI and future joins a near-real-time truth line. Archive truth remains separate with `source = open_meteo_archive`, so later modules can choose whether to prefer archive actuals over provisional 0-hour values.
+
+Core fields:
+- `resolved_at`
+- `source`
+- `temperature_c`
+- `shortwave_radiation_w_m2`
+
+The future predictor can join forecast points to truth by `target_at = resolved_at`.
+
+Open-Meteo archive data can lag behind current time depending on model availability, so the database daemon backfills a configurable historical window instead of assuming the last completed hour is immediately available.
+
+## Visual Explorer
+
+We should build a small web output for inspecting forecast history.
+
+Useful first view:
+- select a weather variable, such as temperature or shortwave radiation
+- select forecast horizons, such as 2h and 4h
+- overlay those horizon-specific external forecasts against resolved truth
+- plot by `target_at`
+
+Example:
+
+```text
+target_at on x-axis
+temperature_c on y-axis
+line 1: Open-Meteo forecast made 2 hours before target_at
+line 2: Open-Meteo forecast made 4 hours before target_at
+line 3: resolved truth
+```
+
+This visual layer should read from the cleaned weather database. It should not be part of the Open-Meteo client or parser.
+
+## First Implementation Slice
+
+1. Fetch one Open-Meteo-style hourly forecast run.
+2. Parse it into forecast points.
+3. Normalize the run through `WeatherBuilder`.
+4. Store forecast points through `WeatherStore`.
+5. Add resolved truth rows when we have a source for observed weather.
+6. Build the visual explorer after forecast/truth storage exists.