ryan/Astrape
Public Access
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

first_commit

Browse Source
This commit is contained in:
rpotter6298
2026-04-25 20:35:25 +02:00
commit 9d15860f0b
22 changed files with 2254 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.gitignore
+6
View File
@@ -0,0 +1,6 @@
__pycache__/
*.py[cod]
logs/
*.log
env/*.env
!env/*.example
deploy/systemd/astrape-db.service
+18
View File
@@ -0,0 +1,18 @@
[Unit]
Description=Astrape Database Ingest
After=network-online.target postgresql.service
Wants=network-online.target
[Service]
Type=simple
User=gibil
Group=gibil
WorkingDirectory=/mnt/astrape
Environment=PYTHONUNBUFFERED=1
Environment=PYTHONDONTWRITEBYTECODE=1
ExecStart=/usr/bin/python3 -m gibil.scripts.db_daemon
Restart=always
RestartSec=10
[Install]
WantedBy=multi-user.target
deploy/systemd/astrape-web.service
+18
View File
@@ -0,0 +1,18 @@
[Unit]
Description=Astrape Web UI
After=network-online.target
Wants=network-online.target
[Service]
Type=simple
User=gibil
Group=gibil
WorkingDirectory=/mnt/astrape
Environment=PYTHONUNBUFFERED=1
Environment=PYTHONDONTWRITEBYTECODE=1
ExecStart=/usr/bin/python3 -m gibil.scripts.web_daemon
Restart=always
RestartSec=5
[Install]
WantedBy=multi-user.target
docs/architecture-principles.md
+71
View File
@@ -0,0 +1,71 @@
# Architecture Principles
## Standalone Subsystems
Each class should behave like a small standalone subsystem. It should own one clear responsibility, expose a narrow public interface, and avoid hidden dependencies on the internals of other classes.
Good subsystem boundaries:
- accept explicit inputs
- return explicit outputs
- keep internal state private
- avoid reaching into global state
- avoid performing unrelated work
- can be tested with recorded or fixture data
Examples:
- a weather client fetches forecast data
- a weather parser converts API payloads into forecast points
- a weather builder normalizes external forecast records for storage
- a storage class persists records
- Gibil makes decisions from snapshots
## Data Models Between Subsystems
Subsystems should communicate through shared data models rather than through source-specific payloads.
For example:
- Open-Meteo JSON should become `WeatherForecastRun`
- Modbus register reads should become `Observation`
- HASS entity state should become `Observation`
- Gibil should reason from `Snapshot`
This keeps the edges messy and the core calm.
## Side Effects At The Edges
Network calls, database writes, MQTT publishes, and filesystem writes should live at clear boundaries.
Core reasoning classes should generally be pure or nearly pure:
- input data in
- answer out
- no surprise I/O
Stateful classes are allowed, but their state should be deliberate and inspectable.
## Grow By Composition
Astrape should grow by connecting small subsystems together, not by building one large object that knows everything.
The desired shape is:
```text
source client -> parser -> model -> storage -> query/snapshot -> Gibil -> publisher
```
Each part should be replaceable without rewriting the others.
## Prefer Working Slices
Build one thin working path at a time. A thin slice may start with empty storage or recorded source data, but it should still follow the real subsystem boundaries.
For example, the weather slice can start with:
```text
Open-Meteo forecast run -> WeatherBuilder -> clean forecast records
```
Then grow into:
```text
Open-Meteo -> parser -> WeatherBuilder -> TimescaleDB -> weather_predictor.py
```
docs/ingestion-and-storage.md
+256
View File
@@ -0,0 +1,256 @@
# Ingestion & Storage
## Purpose
Astrape needs a reliable way to collect energy-related data, normalize it, store it, and give Gibil a clean view of the current system state. The first version should favor boring, inspectable data flows over cleverness.
Gibil should not need to know whether a value came from Modbus, Home Assistant, a weather API, a price API, or a manual override. It should receive timestamped observations and snapshots with enough metadata to decide whether the data is fresh and trustworthy.
## Initial Sources
### Sigen Inverter
- Protocol: Modbus TCP
- Polling target: every 5-10 seconds for fast-changing electrical state
- Initial metrics:
- `solar_power_w`
- `battery_soc_pct`
- `battery_charge_w`
- `battery_discharge_w`
- `grid_import_w`
- `grid_export_w`
- `daily_yield_kwh`
- Risk: register map must be confirmed before this can be real
### Home Assistant / Ganymede
- Preferred integration: MQTT
- Direction: HASS/Ganymede should publish selected state to Astrape where possible
- Initial metrics:
- `home_power_w`
- `indoor_temp_c`
- selected device states
- selected sensor values needed for water/heating logic
- Reasoning: MQTT keeps Astrape loosely coupled and avoids making HASS a synchronous dependency for every decision tick
### Weather
- Preferred first source: OpenMeteo
- Polling target: hourly forecast refresh
- Initial metrics:
- `outdoor_temp_c`
- `cloud_cover_pct`
- `ghi_w_m2`
- `wind_speed_m_s`
- Use: external forecast history for generation and heating models
### Grid Pricing
- First implementation: static time-of-use config
- Later implementation: spot pricing API if needed
- Initial metrics:
- `grid_price_per_kwh`
- `price_stage`
- `cheap_window_active`
- Reasoning: static config lets Gibil produce useful behavior before price API work is settled
### Manual Inputs
- Purpose: allow operator-supplied values when a real integration is not available yet
- Inputs may come from local config or a small authenticated admin path
- Manual data should be marked clearly with `source = manual`
## Observation Shape
Every collector should produce normalized observations.
```text
observed_at: timestamp when the measurement was true
received_at: timestamp when Astrape received it
source: sigen | hass | weather | price | manual
metric: stable metric name
value: number, string, or boolean
unit: W | kWh | pct | C | SEK/kWh | state | none
quality: ok | stale | estimated | missing | error
metadata: source-specific context
```
Guidelines:
- `observed_at` and `received_at` are both needed because pushed data may arrive late
- metric names should be stable and boring
- raw source names/registers/entities belong in metadata, not in the metric name
- Gibil should be able to ignore stale or low-quality observations
## Derived Snapshots
Gibil should reason from snapshots, not directly from loose individual observations.
A snapshot is the best-known whole-system state at a decision tick. It can include:
- current solar generation
- current home consumption
- battery SoC
- battery charge/discharge power
- grid import/export
- current price stage
- active forecast window
- stale/missing input flags
Snapshots should be persisted because they explain what Gibil knew when it made a decision.
## Storage Choice
Use TimescaleDB as the first primary store.
Reasons:
- It is Postgres, so querying and joining data stays straightforward
- It handles time-series retention and aggregation well
- It works for raw observations, derived snapshots, decisions, forecasts, and events
- It leaves room for later model training without needing a second historical store immediately
InfluxDB remains a reasonable alternative, but TimescaleDB is the better default if we want relational joins, auditability, and forecast training queries.
The runtime expects `ASTRAPE_DATABASE_URL` to point at TimescaleDB. Weather ingest also expects `ASTRAPE_LATITUDE` and `ASTRAPE_LONGITUDE`.
## Initial Tables
### `observations`
Raw normalized metric samples from all collectors.
Core fields:
- `id`
- `observed_at`
- `received_at`
- `source`
- `metric`
- `value_num`
- `value_text`
- `value_bool`
- `unit`
- `quality`
- `metadata`
Notes:
- use one value column based on the metric type
- keep metadata as JSON for source-specific details
- make this a hypertable on `observed_at`
### `snapshots`
Periodic whole-system state used by Gibil.
Core fields:
- `id`
- `created_at`
- `snapshot`
- `input_quality`
Notes:
- store the snapshot as JSON initially
- this can be normalized later if query patterns demand it
### `decisions`
Gibil outputs and reasoning.
Core fields:
- `id`
- `created_at`
- `snapshot_id`
- `stage`
- `recommendations`
- `reasons`
- `confidence`
Notes:
- decisions should be explainable enough to debug after the fact
- this table becomes the audit trail for HASS-facing behavior
### `weather_forecast_points`
Clean external weather forecast points from weather sources.
Core fields:
- `id`
- `issued_at`
- `target_at`
- `horizon_hours`
- `source`
- `temperature_c`
- `shortwave_radiation_w_m2`
- `cloud_cover_pct`
Notes:
- this stores external forecasts, not internal predictions
- make this a hypertable on `target_at`
### `weather_resolved_truth`
Observed weather for target hours that have already happened.
Core fields:
- `id`
- `resolved_at`
- `source`
- `temperature_c`
- `shortwave_radiation_w_m2`
Notes:
- future prediction modules can join this to `weather_forecast_points`
- make this a hypertable on `resolved_at`
### `system_events`
Operational events from collectors, storage, Gibil, and publishers.
Core fields:
- `id`
- `created_at`
- `component`
- `severity`
- `event_type`
- `message`
- `metadata`
Notes:
- this should capture stale data, auth failures, bad Modbus reads, publish failures, and degraded-mode decisions
## Retention
Initial retention targets:
- raw 5-10 second observations: 7-30 days
- 1-minute aggregates: 6-12 months
- 15-minute/hourly aggregates: keep indefinitely unless storage becomes a problem
- decisions: keep indefinitely
- system events: keep indefinitely or archive after a year
Retention should be revisited after real sample rates and database size are known.
## First Slice
The first implementation slice should prove the shape before touching real hardware.
1. Define the observation and snapshot models.
2. Add a manual collector only if needed for operator-supplied values.
3. Store observations in TimescaleDB or a local development substitute.
4. Build one snapshot from the latest observations.
5. Let Gibil make a simple stage decision from that snapshot.
6. Persist the decision with reasons.
This gives us the whole loop:
```text
collector -> observations -> snapshot -> Gibil decision -> stored audit trail
```
MQTT publishing can come immediately after this loop exists.
## Open Questions
- Should development use real TimescaleDB from day one, or SQLite/Postgres first?
- What is the exact MQTT topic namespace for HASS/Ganymede integration?
- Which HASS entities should be included in the first read-only state feed?
- How should the `gibil` IPA identity authenticate to MQTT and HASS?
- What high-resolution retention target is acceptable on the Astrape VM?
- Should snapshots be created on a fixed schedule, on new data, or both?
docs/operations.md
+105
View File
@@ -0,0 +1,105 @@
# Operations
## Web UI
Start the web UI daemon:
```bash
python3 -m gibil.scripts.web_daemon
```
The daemon listens on:
```text
http://0.0.0.0:8080
```
By default the server binds to all network interfaces so it can be reached from another machine. Override the bind address or port if needed:
```bash
export ASTRAPE_WEB_HOST='0.0.0.0'
export ASTRAPE_WEB_PORT='8080'
```
The host process reloads `webui.py` and display modules on each request. The browser polls `/api/ui-version` and refreshes when those files change.
## Systemd Services
Install service units:
```bash
sudo cp deploy/systemd/astrape-web.service /etc/systemd/system/
sudo cp deploy/systemd/astrape-db.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now astrape-web.service astrape-db.service
```
Check status:
```bash
systemctl status astrape-web.service
systemctl status astrape-db.service
journalctl -u astrape-web.service -f
journalctl -u astrape-db.service -f
```
Both services run as the IPA-managed `gibil` user from `/mnt/astrape`.
## Database Daemon
Install runtime dependencies:
```bash
python3 -m pip install -r requirements.txt
```
Create a local env file:
```bash
cp env/astrape.env.example env/astrape.env
nano env/astrape.env
```
Required values:
```text
ASTRAPE_DATABASE_URL=postgresql://USER:PASSWORD@HOST:PORT/DBNAME
ASTRAPE_LATITUDE=59.0000
ASTRAPE_LONGITUDE=18.0000
```
Optional values:
```text
ASTRAPE_WEATHER_FORECAST_HOURS=48
ASTRAPE_WEATHER_POLL_SECONDS=3600
ASTRAPE_WEATHER_TRUTH_LOOKBACK_DAYS=14
ASTRAPE_WEATHER_TRUTH_END_DELAY_DAYS=5
```
The daemons load `env/*.env` automatically. Existing process environment variables win over file values.
For temporary frontend tuning, enable display-only sample weather data:
```text
ASTRAPE_WEB_SAMPLE_DATA=1
```
This does not write artificial data to TimescaleDB. It only changes the web UI weather API response.
Start the database ingest daemon:
```bash
python3 -m gibil.scripts.db_daemon
```
Current behavior:
- initializes TimescaleDB weather tables
- fetches real Open-Meteo hourly forecasts
- normalizes them through `WeatherBuilder`
- stores rows in `weather_forecast_points`
- fetches Open-Meteo archive data for resolved truth
- stores rows in `weather_resolved_truth`
- repeats every `ASTRAPE_WEATHER_POLL_SECONDS`
No internal weather predictions are generated here. This daemon only stores external forecast and resolved-truth data for later modules.
docs/weather-source-data.md
+117
View File
@@ -0,0 +1,117 @@
# Weather Source Data
## Goal
This subsystem aggregates external weather forecasts and stores them in a clean database-ready shape.
Terminology:
- **forecast**: data from an external weather source, such as Open-Meteo
- **resolved truth**: observed weather for a time that has already happened
- **prediction**: an internal estimate produced by a future Astrape/Gibil model
This module should not produce predictions or confidence scores. A later `weather_predictor.py` subsystem can use this clean forecast database to produce predictions and confidence.
## Subsystem Boundary
Initial classes should stay narrowly scoped:
- `OpenMeteoClient`: fetch raw hourly forecast payloads
- `OpenMeteoParser`: convert API payloads into external forecast runs and points
- `WeatherBuilder`: normalize and select clean forecast records for database use
- `WeatherStore`: persist forecast points and resolved truth
These classes communicate through data models like `WeatherForecastRun`, `WeatherForecastPoint`, and `WeatherResolvedTruth`.
## Core Data Shape
Every weather API pull is a forecast run.
```text
issued_at = when the external forecast was fetched
target_at = the hour being forecast
horizon_hours = target_at - issued_at
forecast_value = external forecast value for that target hour
```
Later, when `target_at` is in the past, Astrape can attach resolved truth:
```text
resolved_at = the hour that actually happened
truth = observed temperature / observed solar radiation
```
That creates rows future modules can use:
```text
target_at | resolved_truth | forecast_1h | forecast_2h | ... | forecast_48h
```
The future predictor can learn from those rows without needing to know anything about Open-Meteo payloads.
## First Variables
Use Open-Meteo hourly forecast fields:
- `temperature_2m`
- `shortwave_radiation`
- `cloud_cover`
Open-Meteo documents `shortwave_radiation` as average incoming solar radiation over the preceding hour at the surface, equivalent to GHI, measured in W/m2. That is the right starting solar forecast variable for Astrape.
## Storage Shape
Forecast points should be stored as individual rows.
Core fields:
- `issued_at`
- `target_at`
- `horizon_hours`
- `source`
- `temperature_c`
- `shortwave_radiation_w_m2`
- `cloud_cover_pct`
Resolved truth should be stored separately. For now, resolved truth comes from the Open-Meteo historical archive API.
Until archive data is available, Astrape can also store the current 0-hour Open-Meteo forecast as provisional truth with `source = open_meteo_zero_hour`. This gives the UI and future joins a near-real-time truth line. Archive truth remains separate with `source = open_meteo_archive`, so later modules can choose whether to prefer archive actuals over provisional 0-hour values.
Core fields:
- `resolved_at`
- `source`
- `temperature_c`
- `shortwave_radiation_w_m2`
The future predictor can join forecast points to truth by `target_at = resolved_at`.
Open-Meteo archive data can lag behind current time depending on model availability, so the database daemon backfills a configurable historical window instead of assuming the last completed hour is immediately available.
## Visual Explorer
We should build a small web output for inspecting forecast history.
Useful first view:
- select a weather variable, such as temperature or shortwave radiation
- select forecast horizons, such as 2h and 4h
- overlay those horizon-specific external forecasts against resolved truth
- plot by `target_at`
Example:
```text
target_at on x-axis
temperature_c on y-axis
line 1: Open-Meteo forecast made 2 hours before target_at
line 2: Open-Meteo forecast made 4 hours before target_at
line 3: resolved truth
```
This visual layer should read from the cleaned weather database. It should not be part of the Open-Meteo client or parser.
## First Implementation Slice
1. Fetch one Open-Meteo-style hourly forecast run.
2. Parse it into forecast points.
3. Normalize the run through `WeatherBuilder`.
4. Store forecast points through `WeatherStore`.
5. Add resolved truth rows when we have a source for observed weather.
6. Build the visual explorer after forecast/truth storage exists.
gibil/__init__.py
+2
View File
@@ -0,0 +1,2 @@
"""Gibil intelligence package for Astrape."""
gibil/classes/__init__.py
+2
View File
@@ -0,0 +1,2 @@
"""Core classes used by Gibil."""
gibil/classes/agent.py
+81
View File
@@ -0,0 +1,81 @@
from __future__ import annotations
from datetime import datetime, timezone
from gibil.classes.models import Decision, PowerStage, Snapshot
class GibilAgent:
"""Stateful decision engine for Astrape."""
def __init__(self) -> None:
self.surplus_latch_set = False
self.previous_stage: PowerStage | None = None
def decide(self, snapshot: Snapshot) -> Decision:
reasons: list[str] = []
self._update_surplus_latch(snapshot, reasons)
if self.surplus_latch_set:
stage = PowerStage.SURPLUS
reasons.append("surplus latch is set")
elif snapshot.cheap_window_active:
stage = PowerStage.CHEAP_GRID
reasons.append("cheap grid window is active")
elif self._should_conserve(snapshot):
stage = PowerStage.CONSERVE
reasons.append("battery is low and there is no useful solar surplus")
else:
stage = PowerStage.STANDARD
reasons.append("no surplus, cheap window, or conserve condition is active")
if self.previous_stage != stage:
previous = self.previous_stage.value if self.previous_stage else "none"
reasons.append(f"stage changed from {previous} to {stage.value}")
self.previous_stage = stage
return Decision(
created_at=datetime.now(timezone.utc),
stage=stage,
reasons=reasons,
confidence=self._confidence(snapshot),
)
def _update_surplus_latch(
self, snapshot: Snapshot, reasons: list[str]
) -> None:
if snapshot.battery_soc_pct is None or snapshot.solar_power_w is None:
return
home_power_w = snapshot.home_power_w or 0
has_surplus = snapshot.solar_power_w > home_power_w
if not self.surplus_latch_set:
if snapshot.battery_soc_pct >= 95 and has_surplus:
self.surplus_latch_set = True
reasons.append("surplus latch set: battery >= 95% and solar exceeds load")
return
if snapshot.battery_soc_pct < 80 or not has_surplus:
self.surplus_latch_set = False
reasons.append("surplus latch cleared: battery < 80% or surplus ended")
def _should_conserve(self, snapshot: Snapshot) -> bool:
if snapshot.battery_soc_pct is None:
return False
solar_power_w = snapshot.solar_power_w or 0
home_power_w = snapshot.home_power_w or 0
return snapshot.battery_soc_pct < 25 and solar_power_w < home_power_w
def _confidence(self, snapshot: Snapshot) -> float:
expected_inputs = [
snapshot.solar_power_w,
snapshot.home_power_w,
snapshot.battery_soc_pct,
]
present = sum(value is not None for value in expected_inputs)
return present / len(expected_inputs)
gibil/classes/env_loader.py
+37
View File
@@ -0,0 +1,37 @@
from __future__ import annotations
from os import environ
from pathlib import Path
class EnvLoader:
"""Loads Astrape env files without overriding process environment."""
def __init__(self, env_dir: Path | None = None) -> None:
if env_dir is None:
env_dir = Path(__file__).resolve().parents[2] / "env"
self.env_dir = env_dir
def load(self) -> None:
if not self.env_dir.exists():
return
for path in sorted(self.env_dir.glob("*.env")):
self._load_file(path)
def _load_file(self, path: Path) -> None:
for raw_line in path.read_text(encoding="utf-8").splitlines():
line = raw_line.strip()
if not line or line.startswith("#") or "=" not in line:
continue
key, value = line.split("=", 1)
key = key.strip()
value = self._clean_value(value.strip())
if key and key not in environ:
environ[key] = value
def _clean_value(self, value: str) -> str:
if len(value) >= 2 and value[0] == value[-1] and value[0] in {"'", '"'}:
return value[1:-1]
return value
gibil/classes/models.py
+82
View File
@@ -0,0 +1,82 @@
from __future__ import annotations
from dataclasses import dataclass, field
from datetime import datetime, timezone
from enum import Enum
from typing import Any
class ObservationQuality(str, Enum):
OK = "ok"
STALE = "stale"
ESTIMATED = "estimated"
MISSING = "missing"
ERROR = "error"
class PowerStage(str, Enum):
ALWAYS = "always"
SURPLUS = "surplus"
CHEAP_GRID = "cheap_grid"
STANDARD = "standard"
CONSERVE = "conserve"
@dataclass(frozen=True)
class Observation:
source: str
metric: str
value: int | float | str | bool | None
unit: str = "none"
quality: ObservationQuality = ObservationQuality.OK
observed_at: datetime = field(default_factory=lambda: datetime.now(timezone.utc))
received_at: datetime = field(default_factory=lambda: datetime.now(timezone.utc))
metadata: dict[str, Any] = field(default_factory=dict)
@dataclass(frozen=True)
class Snapshot:
created_at: datetime
solar_power_w: float | None = None
home_power_w: float | None = None
battery_soc_pct: float | None = None
grid_import_w: float | None = None
grid_export_w: float | None = None
cheap_window_active: bool = False
input_quality: dict[str, ObservationQuality] = field(default_factory=dict)
@dataclass(frozen=True)
class Decision:
created_at: datetime
stage: PowerStage
reasons: list[str]
confidence: float
@dataclass(frozen=True)
class WeatherForecastPoint:
issued_at: datetime
target_at: datetime
horizon_hours: int
temperature_c: float | None
shortwave_radiation_w_m2: float | None
cloud_cover_pct: float | None = None
source: str = "open_meteo"
@dataclass(frozen=True)
class WeatherForecastRun:
issued_at: datetime
source: str
latitude: float
longitude: float
points: list[WeatherForecastPoint]
@dataclass(frozen=True)
class WeatherResolvedTruth:
resolved_at: datetime
temperature_c: float | None
shortwave_radiation_w_m2: float | None
source: str
gibil/classes/snapshot_builder.py
+63
View File
@@ -0,0 +1,63 @@
from __future__ import annotations
from datetime import datetime, timezone
from gibil.classes.models import Observation, ObservationQuality, Snapshot
class SnapshotBuilder:
"""Builds Gibil's decision input from the latest observations."""
def build(self, observations: list[Observation]) -> Snapshot:
latest = self._latest_by_metric(observations)
return Snapshot(
created_at=datetime.now(timezone.utc),
solar_power_w=self._number(latest, "solar_power_w"),
home_power_w=self._number(latest, "home_power_w"),
battery_soc_pct=self._number(latest, "battery_soc_pct"),
grid_import_w=self._number(latest, "grid_import_w"),
grid_export_w=self._number(latest, "grid_export_w"),
cheap_window_active=self._boolean(latest, "cheap_window_active"),
input_quality={
metric: observation.quality for metric, observation in latest.items()
},
)
def _latest_by_metric(
self, observations: list[Observation]
) -> dict[str, Observation]:
latest: dict[str, Observation] = {}
for observation in observations:
existing = latest.get(observation.metric)
if existing is None or observation.observed_at > existing.observed_at:
latest[observation.metric] = observation
return latest
def _number(
self, observations: dict[str, Observation], metric: str
) -> float | None:
observation = observations.get(metric)
if observation is None or observation.quality != ObservationQuality.OK:
return None
if isinstance(observation.value, bool):
return None
if isinstance(observation.value, int | float):
return float(observation.value)
return None
def _boolean(self, observations: dict[str, Observation], metric: str) -> bool:
observation = observations.get(metric)
if observation is None or observation.quality != ObservationQuality.OK:
return False
if isinstance(observation.value, bool):
return observation.value
return False
gibil/classes/weather_builder.py
+251
View File
@@ -0,0 +1,251 @@
from __future__ import annotations
from datetime import date, datetime, timezone
from typing import Any
from urllib.parse import urlencode
from urllib.request import urlopen
import json
from gibil.classes.models import (
WeatherForecastPoint,
WeatherForecastRun,
WeatherResolvedTruth,
)
class OpenMeteoClient:
"""Fetches external weather forecasts from Open-Meteo."""
base_url = "https://api.open-meteo.com/v1/forecast"
def build_url(
self,
latitude: float,
longitude: float,
forecast_hours: int = 48,
timezone_name: str = "UTC",
) -> str:
params = {
"latitude": latitude,
"longitude": longitude,
"hourly": ",".join(
[
"temperature_2m",
"shortwave_radiation",
"cloud_cover",
]
),
"forecast_hours": forecast_hours,
"timezone": timezone_name,
}
return f"{self.base_url}?{urlencode(params)}"
def fetch_forecast(
self,
latitude: float,
longitude: float,
forecast_hours: int = 48,
) -> WeatherForecastRun:
url = self.build_url(latitude, longitude, forecast_hours)
with urlopen(url, timeout=10) as response:
payload = json.loads(response.read().decode("utf-8"))
return OpenMeteoParser().parse_forecast(
payload=payload,
latitude=latitude,
longitude=longitude,
issued_at=datetime.now(timezone.utc),
)
class OpenMeteoArchiveClient:
"""Fetches historical weather data from Open-Meteo archive."""
base_url = "https://archive-api.open-meteo.com/v1/archive"
def build_url(
self,
latitude: float,
longitude: float,
start_date: date,
end_date: date,
timezone_name: str = "UTC",
) -> str:
params = {
"latitude": latitude,
"longitude": longitude,
"start_date": start_date.isoformat(),
"end_date": end_date.isoformat(),
"hourly": ",".join(
[
"temperature_2m",
"shortwave_radiation",
]
),
"timezone": timezone_name,
}
return f"{self.base_url}?{urlencode(params)}"
def fetch_resolved_truth(
self,
latitude: float,
longitude: float,
start_date: date,
end_date: date,
) -> list[WeatherResolvedTruth]:
url = self.build_url(latitude, longitude, start_date, end_date)
with urlopen(url, timeout=20) as response:
payload = json.loads(response.read().decode("utf-8"))
return OpenMeteoArchiveParser().parse_resolved_truth(payload)
class OpenMeteoParser:
"""Converts Open-Meteo JSON into clean external forecast records."""
def parse_forecast(
self,
payload: dict[str, Any],
latitude: float,
longitude: float,
issued_at: datetime,
) -> WeatherForecastRun:
hourly = payload.get("hourly", {})
times = hourly.get("time", [])
temperatures = hourly.get("temperature_2m", [])
radiation = hourly.get("shortwave_radiation", [])
cloud_cover = hourly.get("cloud_cover", [])
points: list[WeatherForecastPoint] = []
for index, raw_time in enumerate(times):
target_at = self._parse_time(raw_time)
horizon_hours = max(
0, round((target_at - issued_at).total_seconds() / 3600)
)
points.append(
WeatherForecastPoint(
issued_at=issued_at,
target_at=target_at,
horizon_hours=horizon_hours,
temperature_c=self._at(temperatures, index),
shortwave_radiation_w_m2=self._at(radiation, index),
cloud_cover_pct=self._at(cloud_cover, index),
)
)
return WeatherForecastRun(
issued_at=issued_at,
source="open_meteo",
latitude=latitude,
longitude=longitude,
points=points,
)
def _parse_time(self, raw_time: str) -> datetime:
parsed = datetime.fromisoformat(raw_time)
if parsed.tzinfo is None:
return parsed.replace(tzinfo=timezone.utc)
return parsed.astimezone(timezone.utc)
def _at(self, values: list[Any], index: int) -> float | None:
if index >= len(values):
return None
value = values[index]
if value is None:
return None
return float(value)
class OpenMeteoArchiveParser:
"""Converts Open-Meteo archive JSON into resolved truth records."""
def parse_resolved_truth(self, payload: dict[str, Any]) -> list[WeatherResolvedTruth]:
hourly = payload.get("hourly", {})
times = hourly.get("time", [])
temperatures = hourly.get("temperature_2m", [])
radiation = hourly.get("shortwave_radiation", [])
truth: list[WeatherResolvedTruth] = []
for index, raw_time in enumerate(times):
truth.append(
WeatherResolvedTruth(
resolved_at=self._parse_time(raw_time),
temperature_c=self._at(temperatures, index),
shortwave_radiation_w_m2=self._at(radiation, index),
source="open_meteo_archive",
)
)
return truth
def _parse_time(self, raw_time: str) -> datetime:
parsed = datetime.fromisoformat(raw_time)
if parsed.tzinfo is None:
return parsed.replace(tzinfo=timezone.utc)
return parsed.astimezone(timezone.utc)
def _at(self, values: list[Any], index: int) -> float | None:
if index >= len(values):
return None
value = values[index]
if value is None:
return None
return float(value)
class WeatherBuilder:
"""Builds a clean database-ready set of external weather forecast records."""
def build_forecast_run(
self,
source: str,
latitude: float,
longitude: float,
points: list[WeatherForecastPoint],
issued_at: datetime | None = None,
) -> WeatherForecastRun:
if issued_at is None:
issued_at = datetime.now(timezone.utc)
clean_points = [
WeatherForecastPoint(
issued_at=issued_at,
target_at=point.target_at,
horizon_hours=max(
0, round((point.target_at - issued_at).total_seconds() / 3600)
),
temperature_c=point.temperature_c,
shortwave_radiation_w_m2=point.shortwave_radiation_w_m2,
cloud_cover_pct=point.cloud_cover_pct,
source=source,
)
for point in sorted(points, key=lambda item: item.target_at)
]
return WeatherForecastRun(
issued_at=issued_at,
source=source,
latitude=latitude,
longitude=longitude,
points=clean_points,
)
def points_for_horizon(
self,
forecast_runs: list[WeatherForecastRun],
horizon_hours: int,
) -> list[WeatherForecastPoint]:
points: list[WeatherForecastPoint] = []
for run in forecast_runs:
points.extend(
point
for point in run.points
if point.horizon_hours == horizon_hours
)
return sorted(points, key=lambda point: point.target_at)
gibil/classes/weather_display.py
+310
View File
@@ -0,0 +1,310 @@
from __future__ import annotations
import json
from dataclasses import dataclass
from datetime import datetime
from gibil.classes.models import WeatherForecastPoint, WeatherResolvedTruth
@dataclass(frozen=True)
class WeatherDisplayDataset:
forecast_points: list[WeatherForecastPoint]
resolved_truth: list[WeatherResolvedTruth]
class WeatherDisplay:
"""Renders weather source data for the Astrape web UI."""
def render(self) -> str:
return """
<section class="panel weather-panel" data-module="weather-display">
<div class="panel-heading">
<div>
<h2>Weather</h2>
<p>External forecast history</p>
</div>
<div class="control-row">
<label>
Variable
<select id="weather-variable">
<option value="temperature_c">Temperature</option>
<option value="shortwave_radiation_w_m2">Solar radiation</option>
</select>
</label>
<div class="legend-control">
<div class="legend-title">Horizons</div>
<div id="weather-horizons" class="horizon-options"></div>
<div class="horizon-option">
<span class="legend-swatch truth-swatch"></span>
<span>Resolved truth</span>
</div>
</div>
</div>
</div>
<div class="chart-shell">
<canvas id="weather-chart" width="1100" height="420"></canvas>
</div>
</section>
<script>
window.astrapeModules = window.astrapeModules || {};
window.astrapeModules.weatherDisplay = (() => {
const palette = ["#60a5fa", "#34d399", "#fbbf24", "#a78bfa", "#fb7185", "#22d3ee"];
const defaultSlots = [
{ enabled: true, horizon: 4 },
{ enabled: true, horizon: 8 },
{ enabled: true, horizon: 12 },
{ enabled: false, horizon: 16 },
{ enabled: false, horizon: 24 },
{ enabled: false, horizon: 36 },
];
function init() {
const variable = document.getElementById("weather-variable");
variable.addEventListener("change", render);
buildHorizonControls();
refresh();
setInterval(refresh, 5000);
}
async function refresh() {
const response = await fetch("/api/weather", { cache: "no-store" });
const payload = await response.json();
window.astrapeWeatherData = payload;
render();
}
function buildHorizonControls() {
const horizons = document.getElementById("weather-horizons");
horizons.innerHTML = "";
const slots = loadSlots();
slots.forEach((slot, index) => {
const option = document.createElement("div");
option.className = "horizon-option";
option.innerHTML = `
<input class="horizon-enabled" type="checkbox" ${slot.enabled ? "checked" : ""}>
<span class="legend-swatch" style="background: ${palette[index]}"></span>
<input class="horizon-value" type="number" min="1" max="47" step="1" value="${slot.horizon}">
<span>h</span>
`;
const checkbox = option.querySelector(".horizon-enabled");
const value = option.querySelector(".horizon-value");
checkbox.addEventListener("change", render);
value.addEventListener("input", render);
horizons.appendChild(option);
});
}
function render() {
const payload = window.astrapeWeatherData || { forecast_points: [], resolved_truth: [] };
const variable = document.getElementById("weather-variable").value;
const selectedHorizons = selectedSlots();
saveSlots();
drawChart(payload, variable, selectedHorizons);
}
function drawChart(payload, variable, selectedHorizons) {
const canvas = document.getElementById("weather-chart");
const ctx = canvas.getContext("2d");
const series = buildSeries(payload, variable, selectedHorizons);
ctx.clearRect(0, 0, canvas.width, canvas.height);
const allPoints = series.flatMap((item) => item.points);
const now = Date.now();
const xs = allPoints.map((point) => new Date(point.target_at).getTime());
xs.push(now);
const ys = allPoints.map((point) => point.value).filter((value) => value !== null);
if (!xs.length || !ys.length) return;
const bounds = {
minX: Math.min(...xs),
maxX: Math.max(...xs),
minY: Math.min(...ys),
maxY: Math.max(...ys),
};
if (bounds.minY === bounds.maxY) {
bounds.minY -= 1;
bounds.maxY += 1;
}
drawAxes(ctx, canvas, bounds);
drawNowMarker(ctx, canvas, bounds);
series.forEach((item) => {
drawSeries(ctx, canvas, bounds, item.points, item.color, item.width);
});
}
function buildSeries(payload, variable, selectedHorizons) {
const series = [];
for (const slot of selectedHorizons) {
const points = (payload.forecast_points || [])
.filter((point) => point.horizon_hours === slot.horizon)
.map((point) => ({ target_at: point.target_at, value: point[variable] }))
.filter((point) => point.value !== null);
series.push({ label: `${slot.horizon}h forecast`, points, color: slot.color, width: 2 });
}
const truth = (payload.resolved_truth || [])
.map((point) => ({ target_at: point.resolved_at, value: point[variable] }))
.filter((point) => point.value !== null);
series.push({ label: "resolved truth", points: truth, color: "#f8fafc", width: 3 });
return series;
}
function drawAxes(ctx, canvas, bounds) {
const margin = chartMargin();
ctx.strokeStyle = "#94a3b8";
ctx.lineWidth = 1;
ctx.beginPath();
ctx.moveTo(margin.left, margin.top);
ctx.lineTo(margin.left, canvas.height - margin.bottom);
ctx.lineTo(canvas.width - margin.right, canvas.height - margin.bottom);
ctx.stroke();
ctx.fillStyle = "#475569";
ctx.font = "12px system-ui";
ctx.fillText(bounds.maxY.toFixed(1), 10, margin.top + 4);
ctx.fillText(bounds.minY.toFixed(1), 10, canvas.height - margin.bottom);
}
function drawSeries(ctx, canvas, bounds, points, color, width) {
if (!points.length) return;
const margin = chartMargin();
if (points.length === 1) {
const point = points[0];
const x = scale(new Date(point.target_at).getTime(), bounds.minX, bounds.maxX, margin.left, canvas.width - margin.right);
const y = scale(point.value, bounds.minY, bounds.maxY, canvas.height - margin.bottom, margin.top);
ctx.fillStyle = color;
ctx.beginPath();
ctx.arc(x, y, width + 3, 0, Math.PI * 2);
ctx.fill();
return;
}
ctx.strokeStyle = color;
ctx.lineWidth = width;
ctx.beginPath();
points.forEach((point, index) => {
const x = scale(new Date(point.target_at).getTime(), bounds.minX, bounds.maxX, margin.left, canvas.width - margin.right);
const y = scale(point.value, bounds.minY, bounds.maxY, canvas.height - margin.bottom, margin.top);
if (index === 0) ctx.moveTo(x, y);
else ctx.lineTo(x, y);
});
ctx.stroke();
}
function drawNowMarker(ctx, canvas, bounds) {
const now = Date.now();
if (now < bounds.minX || now > bounds.maxX) return;
const margin = chartMargin();
const x = scale(now, bounds.minX, bounds.maxX, margin.left, canvas.width - margin.right);
ctx.save();
ctx.strokeStyle = "#f8fafc";
ctx.lineWidth = 1;
ctx.setLineDash([5, 5]);
ctx.beginPath();
ctx.moveTo(x, margin.top);
ctx.lineTo(x, canvas.height - margin.bottom);
ctx.stroke();
ctx.setLineDash([]);
ctx.fillStyle = "#f8fafc";
ctx.font = "12px system-ui";
ctx.fillText("now", Math.min(x + 8, canvas.width - margin.right - 28), margin.top + 14);
ctx.restore();
}
function selectedSlots() {
return [...document.querySelectorAll("#weather-horizons .horizon-option")]
.map((item, index) => {
const enabled = item.querySelector(".horizon-enabled").checked;
const input = item.querySelector(".horizon-value");
const horizon = clamp(Number(input.value), 1, 47);
input.value = horizon;
return enabled ? { horizon, color: palette[index] } : null;
})
.filter(Boolean);
}
function saveSlots() {
const slots = [...document.querySelectorAll("#weather-horizons .horizon-option")]
.map((item) => ({
enabled: item.querySelector(".horizon-enabled").checked,
horizon: Number(item.querySelector(".horizon-value").value),
}));
localStorage.setItem("astrapeWeatherHorizonSlots", JSON.stringify(slots));
}
function loadSlots() {
try {
const parsed = JSON.parse(localStorage.getItem("astrapeWeatherHorizonSlots"));
if (Array.isArray(parsed) && parsed.length === 6) return parsed;
} catch (error) {
return defaultSlots;
}
return defaultSlots;
}
function clamp(value, min, max) {
if (!Number.isFinite(value)) return min;
return Math.min(max, Math.max(min, Math.round(value)));
}
function chartMargin() {
return { top: 24, right: 28, bottom: 34, left: 52 };
}
function scale(value, inMin, inMax, outMin, outMax) {
if (inMin === inMax) return (outMin + outMax) / 2;
return outMin + ((value - inMin) / (inMax - inMin)) * (outMax - outMin);
}
return { init };
})();
window.astrapeModules.weatherDisplay.init();
</script>
"""
def data_payload(self, dataset: WeatherDisplayDataset | None = None) -> str:
if dataset is None:
dataset = WeatherDisplayDataset(forecast_points=[], resolved_truth=[])
forecast_points = [self._forecast_point(point) for point in dataset.forecast_points]
resolved_truth = [self._truth_point(point) for point in dataset.resolved_truth]
horizons = sorted({point["horizon_hours"] for point in forecast_points})
return json.dumps(
{
"forecast_points": forecast_points,
"resolved_truth": resolved_truth,
"horizons": horizons,
"min_horizon": 1,
"max_horizon": 47,
}
)
def _forecast_point(self, point: WeatherForecastPoint) -> dict[str, object]:
return {
"issued_at": self._iso(point.issued_at),
"target_at": self._iso(point.target_at),
"horizon_hours": point.horizon_hours,
"source": point.source,
"temperature_c": point.temperature_c,
"shortwave_radiation_w_m2": point.shortwave_radiation_w_m2,
"cloud_cover_pct": point.cloud_cover_pct,
}
def _truth_point(self, point: WeatherResolvedTruth) -> dict[str, object]:
return {
"resolved_at": self._iso(point.resolved_at),
"source": point.source,
"temperature_c": point.temperature_c,
"shortwave_radiation_w_m2": point.shortwave_radiation_w_m2,
}
def _iso(self, value: datetime) -> str:
return value.isoformat()
gibil/classes/weather_sample_data.py
+74
View File
@@ -0,0 +1,74 @@
from __future__ import annotations
from datetime import datetime, timedelta, timezone
from math import pi, sin
from gibil.classes.models import WeatherForecastPoint, WeatherResolvedTruth
from gibil.classes.weather_display import WeatherDisplayDataset
class WeatherSampleData:
"""Builds temporary display-only weather data for UI tuning."""
def build(self, hours: int = 72) -> WeatherDisplayDataset:
now = datetime.now(timezone.utc).replace(minute=0, second=0, microsecond=0)
start = now - timedelta(hours=hours)
horizons = [2, 4, 8, 12, 24]
forecast_points: list[WeatherForecastPoint] = []
resolved_truth: list[WeatherResolvedTruth] = []
for offset in range(hours + 1):
target_at = start + timedelta(hours=offset)
truth_temperature = self._temperature(target_at, offset)
truth_solar = self._solar(target_at, offset)
resolved_truth.append(
WeatherResolvedTruth(
resolved_at=target_at,
source="sample",
temperature_c=truth_temperature,
shortwave_radiation_w_m2=truth_solar,
)
)
for horizon in horizons:
forecast_points.append(
WeatherForecastPoint(
issued_at=target_at - timedelta(hours=horizon),
target_at=target_at,
horizon_hours=horizon,
temperature_c=truth_temperature
+ self._temperature_error(offset, horizon),
shortwave_radiation_w_m2=max(
0,
truth_solar + self._solar_error(offset, horizon),
),
cloud_cover_pct=max(
0,
min(100, 45 + 30 * sin((offset + horizon) / 9)),
),
source="sample",
)
)
return WeatherDisplayDataset(
forecast_points=forecast_points,
resolved_truth=resolved_truth,
)
def _temperature(self, target_at: datetime, offset: int) -> float:
daily = sin(((target_at.hour - 7) / 24) * 2 * pi)
slow = sin(offset / 18)
return round(6.5 + daily * 5.5 + slow * 1.3, 1)
def _solar(self, target_at: datetime, offset: int) -> float:
daylight = max(0, sin(((target_at.hour - 5) / 15) * pi))
cloud_effect = 0.75 + 0.25 * sin(offset / 7)
return round(780 * daylight * cloud_effect, 1)
def _temperature_error(self, offset: int, horizon: int) -> float:
return round((horizon / 8) * sin((offset + horizon) / 5), 1)
def _solar_error(self, offset: int, horizon: int) -> float:
return round((horizon * 9) * sin((offset + horizon) / 4), 1)
gibil/classes/weather_store.py
+278
View File
@@ -0,0 +1,278 @@
from __future__ import annotations
from contextlib import contextmanager
from dataclasses import dataclass
from datetime import datetime, timedelta, timezone
from os import environ
from typing import Iterator
from gibil.classes.models import WeatherForecastPoint, WeatherForecastRun, WeatherResolvedTruth
from gibil.classes.weather_display import WeatherDisplayDataset
class WeatherStoreConfigurationError(RuntimeError):
pass
@dataclass(frozen=True)
class WeatherStoreConfig:
database_url: str
@classmethod
def from_env(cls) -> "WeatherStoreConfig":
database_url = environ.get("ASTRAPE_DATABASE_URL")
if not database_url:
raise WeatherStoreConfigurationError(
"ASTRAPE_DATABASE_URL is required for weather storage"
)
return cls(database_url=database_url)
class WeatherStore:
"""Persists external weather forecasts and resolved truth in TimescaleDB."""
def __init__(self, config: WeatherStoreConfig) -> None:
self.config = config
@classmethod
def from_env(cls) -> "WeatherStore":
return cls(WeatherStoreConfig.from_env())
def initialize(self) -> None:
with self._connection() as connection:
with connection.cursor() as cursor:
cursor.execute("CREATE EXTENSION IF NOT EXISTS timescaledb")
cursor.execute(
"""
CREATE TABLE IF NOT EXISTS weather_forecast_points (
issued_at TIMESTAMPTZ NOT NULL,
target_at TIMESTAMPTZ NOT NULL,
horizon_hours INTEGER NOT NULL,
source TEXT NOT NULL,
latitude DOUBLE PRECISION NOT NULL,
longitude DOUBLE PRECISION NOT NULL,
temperature_c DOUBLE PRECISION,
shortwave_radiation_w_m2 DOUBLE PRECISION,
cloud_cover_pct DOUBLE PRECISION,
inserted_at TIMESTAMPTZ NOT NULL DEFAULT now(),
PRIMARY KEY (issued_at, target_at, source)
)
"""
)
cursor.execute(
"""
SELECT create_hypertable(
'weather_forecast_points',
'target_at',
if_not_exists => TRUE
)
"""
)
cursor.execute(
"""
CREATE TABLE IF NOT EXISTS weather_resolved_truth (
resolved_at TIMESTAMPTZ NOT NULL,
source TEXT NOT NULL,
temperature_c DOUBLE PRECISION,
shortwave_radiation_w_m2 DOUBLE PRECISION,
inserted_at TIMESTAMPTZ NOT NULL DEFAULT now(),
PRIMARY KEY (resolved_at, source)
)
"""
)
cursor.execute(
"""
SELECT create_hypertable(
'weather_resolved_truth',
'resolved_at',
if_not_exists => TRUE
)
"""
)
connection.commit()
def save_forecast_run(self, forecast_run: WeatherForecastRun) -> int:
rows = [
(
point.issued_at,
point.target_at,
point.horizon_hours,
forecast_run.source,
forecast_run.latitude,
forecast_run.longitude,
point.temperature_c,
point.shortwave_radiation_w_m2,
point.cloud_cover_pct,
)
for point in forecast_run.points
]
if not rows:
return 0
with self._connection() as connection:
with connection.cursor() as cursor:
cursor.executemany(
"""
INSERT INTO weather_forecast_points (
issued_at,
target_at,
horizon_hours,
source,
latitude,
longitude,
temperature_c,
shortwave_radiation_w_m2,
cloud_cover_pct
)
VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s)
ON CONFLICT (issued_at, target_at, source)
DO UPDATE SET
horizon_hours = EXCLUDED.horizon_hours,
latitude = EXCLUDED.latitude,
longitude = EXCLUDED.longitude,
temperature_c = EXCLUDED.temperature_c,
shortwave_radiation_w_m2 = EXCLUDED.shortwave_radiation_w_m2,
cloud_cover_pct = EXCLUDED.cloud_cover_pct,
inserted_at = now()
""",
rows,
)
connection.commit()
return len(rows)
def save_resolved_truth(self, truth_points: list[WeatherResolvedTruth]) -> int:
rows = [
(
point.resolved_at,
point.source,
point.temperature_c,
point.shortwave_radiation_w_m2,
)
for point in truth_points
]
if not rows:
return 0
with self._connection() as connection:
with connection.cursor() as cursor:
cursor.executemany(
"""
INSERT INTO weather_resolved_truth (
resolved_at,
source,
temperature_c,
shortwave_radiation_w_m2
)
VALUES (%s, %s, %s, %s)
ON CONFLICT (resolved_at, source)
DO UPDATE SET
temperature_c = EXCLUDED.temperature_c,
shortwave_radiation_w_m2 = EXCLUDED.shortwave_radiation_w_m2,
inserted_at = now()
""",
rows,
)
connection.commit()
return len(rows)
def save_zero_hour_forecast_as_truth(
self, forecast_run: WeatherForecastRun
) -> int:
truth_points = [
WeatherResolvedTruth(
resolved_at=point.target_at,
source="open_meteo_zero_hour",
temperature_c=point.temperature_c,
shortwave_radiation_w_m2=point.shortwave_radiation_w_m2,
)
for point in forecast_run.points
if point.horizon_hours == 0
]
return self.save_resolved_truth(truth_points)
def load_display_dataset(
self,
start_at: datetime | None = None,
end_at: datetime | None = None,
) -> WeatherDisplayDataset:
now = datetime.now(timezone.utc)
if start_at is None:
start_at = now - timedelta(hours=24)
if end_at is None:
end_at = now + timedelta(hours=48)
with self._connection() as connection:
with connection.cursor() as cursor:
cursor.execute(
"""
SELECT
issued_at,
target_at,
horizon_hours,
source,
temperature_c,
shortwave_radiation_w_m2,
cloud_cover_pct
FROM weather_forecast_points
WHERE target_at >= %s AND target_at <= %s
ORDER BY target_at, horizon_hours
LIMIT 5000
""",
(start_at, end_at),
)
forecast_rows = cursor.fetchall()
cursor.execute(
"""
SELECT
resolved_at,
source,
temperature_c,
shortwave_radiation_w_m2
FROM weather_resolved_truth
WHERE resolved_at >= %s AND resolved_at <= %s
ORDER BY resolved_at
LIMIT 5000
""",
(start_at, end_at),
)
truth_rows = cursor.fetchall()
return WeatherDisplayDataset(
forecast_points=[
WeatherForecastPoint(
issued_at=row[0],
target_at=row[1],
horizon_hours=row[2],
source=row[3],
temperature_c=row[4],
shortwave_radiation_w_m2=row[5],
cloud_cover_pct=row[6],
)
for row in forecast_rows
],
resolved_truth=[
WeatherResolvedTruth(
resolved_at=row[0],
source=row[1],
temperature_c=row[2],
shortwave_radiation_w_m2=row[3],
)
for row in truth_rows
],
)
@contextmanager
def _connection(self) -> Iterator[object]:
try:
import psycopg
except ImportError as error:
raise WeatherStoreConfigurationError(
"Install dependencies with `python3 -m pip install -r requirements.txt`"
) from error
with psycopg.connect(self.config.database_url) as connection:
yield connection
gibil/classes/webui.py
+254
View File
@@ -0,0 +1,254 @@
from __future__ import annotations
from os import environ
from gibil.classes.env_loader import EnvLoader
from gibil.classes.weather_sample_data import WeatherSampleData
from gibil.classes.weather_store import WeatherStore, WeatherStoreConfigurationError
from gibil.classes.weather_display import WeatherDisplay
class WebUI:
"""Composes Astrape web modules into one page."""
def __init__(self) -> None:
self.weather_display = WeatherDisplay()
def render_page(self) -> str:
return f"""<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>Astrape</title>
<style>
:root {{
color-scheme: light;
--bg: #11151c;
--surface: #181e27;
--panel: #1f2733;
--ink: #edf2f7;
--muted: #9aa8ba;
--line: #344052;
--field: #121821;
}}
* {{
box-sizing: border-box;
}}
body {{
margin: 0;
background: var(--bg);
color: var(--ink);
font-family: Inter, ui-sans-serif, system-ui, -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif;
line-height: 1.4;
}}
header {{
display: flex;
align-items: center;
justify-content: space-between;
gap: 24px;
padding: 22px 28px;
border-bottom: 1px solid var(--line);
background: var(--surface);
}}
h1, h2, p {{
margin: 0;
}}
h1 {{
font-size: 22px;
font-weight: 700;
}}
h2 {{
font-size: 18px;
font-weight: 700;
}}
p {{
color: var(--muted);
font-size: 13px;
}}
main {{
width: min(1280px, 100%);
margin: 0 auto;
padding: 24px;
}}
.panel {{
background: var(--surface);
border: 1px solid var(--line);
border-radius: 8px;
padding: 18px;
}}
.panel-heading {{
display: grid;
grid-template-columns: minmax(180px, auto) 1fr;
gap: 24px;
margin-bottom: 18px;
}}
.control-row {{
display: flex;
align-items: end;
justify-content: flex-end;
flex-wrap: wrap;
gap: 14px;
}}
label {{
display: grid;
gap: 6px;
color: var(--muted);
font-size: 12px;
font-weight: 600;
}}
select {{
min-width: 150px;
border: 1px solid var(--line);
border-radius: 6px;
background: var(--field);
color: var(--ink);
padding: 8px;
font: inherit;
}}
.legend-control {{
display: flex;
align-items: center;
justify-content: flex-end;
flex-wrap: wrap;
gap: 10px 12px;
color: var(--muted);
font-size: 12px;
font-weight: 600;
}}
.legend-title {{
line-height: 1;
}}
.horizon-options {{
display: flex;
align-items: center;
flex-wrap: wrap;
gap: 8px 12px;
}}
.horizon-option {{
display: flex;
align-items: center;
gap: 8px;
color: var(--ink);
font-size: 13px;
font-weight: 500;
line-height: 1.2;
}}
.horizon-option input {{
width: 14px;
height: 14px;
margin: 0;
}}
.horizon-option .horizon-value {{
width: 54px;
height: 28px;
border: 1px solid var(--line);
border-radius: 6px;
background: var(--field);
color: var(--ink);
padding: 4px 6px;
font: inherit;
}}
.legend-swatch {{
width: 22px;
height: 3px;
border-radius: 999px;
flex: 0 0 auto;
}}
.truth-swatch {{
background: #f8fafc;
}}
.chart-shell {{
position: relative;
min-height: 420px;
border: 1px solid var(--line);
border-radius: 6px;
overflow: hidden;
background: var(--panel);
}}
canvas {{
display: block;
width: 100%;
height: 420px;
}}
@media (max-width: 760px) {{
header, .panel-heading, .control-row {{
display: grid;
}}
.legend-control, .horizon-options {{
justify-content: flex-start;
}}
main {{
padding: 14px;
}}
select {{
width: 100%;
}}
}}
</style>
</head>
<body>
<header>
<h1>Astrape</h1>
<p>Gibil web UI</p>
</header>
<main>
{self.weather_display.render()}
</main>
<script>
let astrapeUiVersion = null;
async function watchUiVersion() {{
const response = await fetch("/api/ui-version", {{ cache: "no-store" }});
const payload = await response.json();
if (astrapeUiVersion === null) {{
astrapeUiVersion = payload.version;
return;
}}
if (astrapeUiVersion !== payload.version) {{
window.location.reload();
}}
}}
setInterval(watchUiVersion, 1000);
watchUiVersion();
</script>
</body>
</html>"""
def weather_payload(self) -> str:
EnvLoader().load()
if environ.get("ASTRAPE_WEB_SAMPLE_DATA") == "1":
return self.weather_display.data_payload(WeatherSampleData().build())
try:
dataset = WeatherStore.from_env().load_display_dataset()
except WeatherStoreConfigurationError:
dataset = None
return self.weather_display.data_payload(dataset)
gibil/scripts/db_daemon.py
+137
View File
@@ -0,0 +1,137 @@
from __future__ import annotations
from dataclasses import dataclass
from datetime import datetime, timedelta, timezone
from os import environ
from sys import stderr
from time import sleep
from gibil.classes.env_loader import EnvLoader
from gibil.classes.weather_builder import (
OpenMeteoArchiveClient,
OpenMeteoClient,
WeatherBuilder,
)
from gibil.classes.weather_store import WeatherStore
@dataclass(frozen=True)
class DbDaemonConfig:
latitude: float
longitude: float
forecast_hours: int
truth_lookback_days: int
truth_end_delay_days: int
poll_seconds: int
@classmethod
def from_env(cls) -> "DbDaemonConfig":
return cls(
latitude=float(_required_env("ASTRAPE_LATITUDE")),
longitude=float(_required_env("ASTRAPE_LONGITUDE")),
forecast_hours=int(environ.get("ASTRAPE_WEATHER_FORECAST_HOURS", "48")),
truth_lookback_days=int(
environ.get("ASTRAPE_WEATHER_TRUTH_LOOKBACK_DAYS", "14")
),
truth_end_delay_days=int(
environ.get("ASTRAPE_WEATHER_TRUTH_END_DELAY_DAYS", "5")
),
poll_seconds=int(environ.get("ASTRAPE_WEATHER_POLL_SECONDS", "3600")),
)
class DbDaemon:
"""Runs builder components that populate Astrape's database."""
def __init__(
self,
config: DbDaemonConfig,
weather_client: OpenMeteoClient,
archive_client: OpenMeteoArchiveClient,
weather_builder: WeatherBuilder,
weather_store: WeatherStore,
) -> None:
self.config = config
self.weather_client = weather_client
self.archive_client = archive_client
self.weather_builder = weather_builder
self.weather_store = weather_store
@classmethod
def from_env(cls) -> "DbDaemon":
return cls(
config=DbDaemonConfig.from_env(),
weather_client=OpenMeteoClient(),
archive_client=OpenMeteoArchiveClient(),
weather_builder=WeatherBuilder(),
weather_store=WeatherStore.from_env(),
)
def initialize(self) -> None:
self.weather_store.initialize()
def run_once(self) -> tuple[int, int]:
raw_run = self.weather_client.fetch_forecast(
latitude=self.config.latitude,
longitude=self.config.longitude,
forecast_hours=self.config.forecast_hours,
)
forecast_run = self.weather_builder.build_forecast_run(
source=raw_run.source,
latitude=raw_run.latitude,
longitude=raw_run.longitude,
points=raw_run.points,
issued_at=raw_run.issued_at,
)
forecast_count = self.weather_store.save_forecast_run(forecast_run)
zero_hour_truth_count = self.weather_store.save_zero_hour_forecast_as_truth(
forecast_run
)
today = datetime.now(timezone.utc).date()
truth_end = today - timedelta(days=self.config.truth_end_delay_days)
truth_start = truth_end - timedelta(days=self.config.truth_lookback_days)
truth_points = self.archive_client.fetch_resolved_truth(
latitude=self.config.latitude,
longitude=self.config.longitude,
start_date=truth_start,
end_date=truth_end,
)
archive_truth_count = self.weather_store.save_resolved_truth(truth_points)
return forecast_count, zero_hour_truth_count + archive_truth_count
def run_forever(self) -> None:
self.initialize()
while True:
forecast_count, truth_count = self.run_once()
print(
f"stored_weather_forecast_points={forecast_count} "
f"stored_weather_resolved_truth={truth_count}",
flush=True,
)
sleep(self.config.poll_seconds)
def main() -> None:
try:
EnvLoader().load()
daemon = DbDaemon.from_env()
daemon.run_forever()
except Exception as error:
print(f"db_daemon_startup_error={error}", file=stderr)
raise SystemExit(1) from error
def _required_env(name: str) -> str:
value = environ.get(name)
if not value:
raise RuntimeError(
f"{name} is required. Set ASTRAPE_DATABASE_URL, "
"ASTRAPE_LATITUDE, and ASTRAPE_LONGITUDE before starting db_daemon."
)
return value
if __name__ == "__main__":
main()
gibil/scripts/web_daemon.py
+84
View File
@@ -0,0 +1,84 @@
from __future__ import annotations
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from importlib import import_module, reload
from os import environ
from pathlib import Path
import json
from gibil.classes.env_loader import EnvLoader
EnvLoader().load()
HOST = environ.get("ASTRAPE_WEB_HOST", "0.0.0.0")
PORT = int(environ.get("ASTRAPE_WEB_PORT", "8080"))
PROJECT_ROOT = Path(__file__).resolve().parents[2]
WATCHED_PATHS = [
PROJECT_ROOT / "gibil" / "classes" / "webui.py",
PROJECT_ROOT / "gibil" / "classes" / "weather_display.py",
PROJECT_ROOT / "gibil" / "classes" / "weather_store.py",
]
class AstrapeWebHandler(BaseHTTPRequestHandler):
def do_GET(self) -> None:
if self.path == "/":
self._send_html(self._webui().render_page())
return
if self.path == "/api/weather":
self._send_json_text(self._webui().weather_payload())
return
if self.path == "/api/ui-version":
self._send_json_text(json.dumps({"version": self._ui_version()}))
return
self.send_error(404)
def log_message(self, format: str, *args: object) -> None:
print(f"{self.address_string()} - {format % args}")
def _webui(self):
weather_store_module = import_module("gibil.classes.weather_store")
weather_display_module = import_module("gibil.classes.weather_display")
webui_module = import_module("gibil.classes.webui")
reload(weather_store_module)
reload(weather_display_module)
reload(webui_module)
return webui_module.WebUI()
def _ui_version(self) -> str:
mtimes = [
str(path.stat().st_mtime_ns)
for path in WATCHED_PATHS
if path.exists()
]
return ".".join(mtimes)
def _send_html(self, body: str) -> None:
encoded = body.encode("utf-8")
self.send_response(200)
self.send_header("Content-Type", "text/html; charset=utf-8")
self.send_header("Content-Length", str(len(encoded)))
self.end_headers()
self.wfile.write(encoded)
def _send_json_text(self, body: str) -> None:
encoded = body.encode("utf-8")
self.send_response(200)
self.send_header("Content-Type", "application/json; charset=utf-8")
self.send_header("Cache-Control", "no-store")
self.send_header("Content-Length", str(len(encoded)))
self.end_headers()
self.wfile.write(encoded)
def main() -> None:
server = ThreadingHTTPServer((HOST, PORT), AstrapeWebHandler)
print(f"Astrape web UI listening on http://{HOST}:{PORT}")
server.serve_forever()
if __name__ == "__main__":
main()
main.py
+7
View File
@@ -0,0 +1,7 @@
def main() -> None:
print("Run `python3 -m gibil.scripts.web_daemon` to start the Astrape web UI.")
print("Run `python3 -m gibil.scripts.db_daemon` to start database ingest.")
if __name__ == "__main__":
main()
requirements.txt
+1
View File
@@ -0,0 +1 @@
psycopg[binary]>=3.2,<4