Astrape/docs/ingestion-and-storage.md

Ingestion & Storage

Purpose

Astrape needs a reliable way to collect energy-related data, normalize it, store it, and give Gibil a clean view of the current system state. The first version should favor boring, inspectable data flows over cleverness.

Gibil should not need to know whether a value came from Modbus, Home Assistant, a weather API, a price API, or a manual override. It should receive timestamped observations and snapshots with enough metadata to decide whether the data is fresh and trustworthy.

Initial Sources

Sigen Inverter

• Protocol: Modbus TCP
• Polling target: every 5-10 seconds for fast-changing electrical state
• Initial metrics:
  • solar_power_w
  • battery_soc_pct
  • battery_charge_w
  • battery_discharge_w
  • grid_import_w
  • grid_export_w
  • daily_yield_kwh
• Risk: register map must be confirmed before this can be real

Home Assistant / Ganymede

• Preferred integration: MQTT
• Direction: HASS/Ganymede should publish selected state to Astrape where possible
• Initial metrics:
  • home_power_w
  • indoor_temp_c
  • selected device states
  • selected sensor values needed for water/heating logic
• Reasoning: MQTT keeps Astrape loosely coupled and avoids making HASS a synchronous dependency for every decision tick

Weather

• Preferred first source: OpenMeteo
• Polling target: hourly forecast refresh
• Initial metrics:
  • outdoor_temp_c
  • cloud_cover_pct
  • ghi_w_m2
  • wind_speed_m_s
• Use: external forecast history for generation and heating models

Grid Pricing

• First implementation: static time-of-use config
• Later implementation: spot pricing API if needed
• Initial metrics:
  • grid_price_per_kwh
  • price_stage
  • cheap_window_active
• Reasoning: static config lets Gibil produce useful behavior before price API work is settled

Manual Inputs

• Purpose: allow operator-supplied values when a real integration is not available yet
• Inputs may come from local config or a small authenticated admin path
• Manual data should be marked clearly with source = manual

Observation Shape

Every collector should produce normalized observations.

observed_at: timestamp when the measurement was true
received_at: timestamp when Astrape received it
source: sigen | hass | weather | price | manual
metric: stable metric name
value: number, string, or boolean
unit: W | kWh | pct | C | SEK/kWh | state | none
quality: ok | stale | estimated | missing | error
metadata: source-specific context

Guidelines:

• observed_at and received_at are both needed because pushed data may arrive late
• metric names should be stable and boring
• raw source names/registers/entities belong in metadata, not in the metric name
• Gibil should be able to ignore stale or low-quality observations

Derived Snapshots

Gibil should reason from snapshots, not directly from loose individual observations.

A snapshot is the best-known whole-system state at a decision tick. It can include:

• current solar generation
• current home consumption
• battery SoC
• battery charge/discharge power
• grid import/export
• current price stage
• active forecast window
• stale/missing input flags

Snapshots should be persisted because they explain what Gibil knew when it made a decision.

Storage Choice

Use TimescaleDB as the first primary store.

Reasons:

• It is Postgres, so querying and joining data stays straightforward
• It handles time-series retention and aggregation well
• It works for raw observations, derived snapshots, decisions, forecasts, and events
• It leaves room for later model training without needing a second historical store immediately

InfluxDB remains a reasonable alternative, but TimescaleDB is the better default if we want relational joins, auditability, and forecast training queries.

The runtime expects ASTRAPE_DATABASE_URL to point at TimescaleDB. Weather ingest also expects ASTRAPE_LATITUDE and ASTRAPE_LONGITUDE.

Initial Tables

observations

Raw normalized metric samples from all collectors.

Core fields:

• id
• observed_at
• received_at
• source
• metric
• value_num
• value_text
• value_bool
• unit
• quality
• metadata

Notes:

• use one value column based on the metric type
• keep metadata as JSON for source-specific details
• make this a hypertable on observed_at

snapshots

Periodic whole-system state used by Gibil.

Core fields:

• id
• created_at
• snapshot
• input_quality

Notes:

• store the snapshot as JSON initially
• this can be normalized later if query patterns demand it

decisions

Gibil outputs and reasoning.

Core fields:

• id
• created_at
• snapshot_id
• stage
• recommendations
• reasons
• confidence

Notes:

• decisions should be explainable enough to debug after the fact
• this table becomes the audit trail for HASS-facing behavior

weather_forecast_points

Clean external weather forecast points from weather sources.

Core fields:

• id
• issued_at
• target_at
• horizon_hours
• source
• temperature_c
• shortwave_radiation_w_m2
• cloud_cover_pct

Notes:

• this stores external forecasts, not internal predictions
• make this a hypertable on target_at

weather_resolved_truth

Observed weather for target hours that have already happened.

Core fields:

• id
• resolved_at
• source
• temperature_c
• shortwave_radiation_w_m2

Notes:

• future prediction modules can join this to weather_forecast_points
• make this a hypertable on resolved_at

system_events

Operational events from collectors, storage, Gibil, and publishers.

Core fields:

• id
• created_at
• component
• severity
• event_type
• message
• metadata

Notes:

• this should capture stale data, auth failures, bad Modbus reads, publish failures, and degraded-mode decisions

Retention

Initial retention targets:

• raw 5-10 second observations: 7-30 days
• 1-minute aggregates: 6-12 months
• 15-minute/hourly aggregates: keep indefinitely unless storage becomes a problem
• decisions: keep indefinitely
• system events: keep indefinitely or archive after a year

Retention should be revisited after real sample rates and database size are known.

First Slice

The first implementation slice should prove the shape before touching real hardware.

1. Define the observation and snapshot models.
2. Add a manual collector only if needed for operator-supplied values.
3. Store observations in TimescaleDB or a local development substitute.
4. Build one snapshot from the latest observations.
5. Let Gibil make a simple stage decision from that snapshot.
6. Persist the decision with reasons.

This gives us the whole loop:

collector -> observations -> snapshot -> Gibil decision -> stored audit trail

MQTT publishing can come immediately after this loop exists.

Open Questions

• Should development use real TimescaleDB from day one, or SQLite/Postgres first?
• What is the exact MQTT topic namespace for HASS/Ganymede integration?
• Which HASS entities should be included in the first read-only state feed?
• How should the gibil IPA identity authenticate to MQTT and HASS?
• What high-resolution retention target is acceptable on the Astrape VM?
• Should snapshots be created on a fixed schedule, on new data, or both?