Files

T

rpotter6298 512ebd13b2 Add distributed server implementation and protocol definitions

- Introduced `protocol.py` for shared data models used in server/client communication, including request and response schemas for registration, job submission, and status updates.
- Implemented `server.py` to manage a SQLite job queue and client registry, handling job polling, status updates, and job completion.
- Created a cheat sheet for server usage, detailing commands for starting the server, submitting jobs, and monitoring clients.
- Added several experiment configuration files for various training setups, including geometry vector injections and baseline ensembles.

2026-04-28 08:24:25 +02:00

3.8 KiB

Raw Blame History

Distributed Server Cheat Sheet

All commands assume server is running on hades at port 8765.

Start Server

python -m v4.distributed.server --token hypertower

Run inside tmux so it survives disconnects:

tmux new -s htserver
python -m v4.distributed.server --token hypertower
# Ctrl-B D to detach
tmux attach -t htserver   # reattach later

Local Workflow (hades as server + client)

# Terminal 1 — server
python -m v4.distributed.server --token hypertower

# Terminal 2 — client (skip rsync, results already local)
python -m v4.distributed.client \
    --server http://hades:8765 --token hypertower \
    --server-ssh ignored --server-path ignored \
    --local-path /home/rpotter/hypertower \
    --no-sync

# Terminal 3 — dispatch (exits after queuing; client picks up jobs)
python -m v4.distributed.batch_dispatch \
    --server http://hades:8765 --token hypertower \
    --config v4/configs/ensemble_fused.json \
    --batch v4/scripts/experiments/my_batch.json

Start Clients

Hades (server-local, no sync):

python -m v4.distributed.client \
    --server http://hades:8765 --token hypertower \
    --server-ssh ignored --server-path ignored \
    --local-path /home/rpotter/hypertower \
    --no-sync

Apollo (remote client):

python -m v4.distributed.client \
    --server http://hades:8765 --token hypertower \
    --server-ssh rpotter@hades \
    --server-path /home/rpotter/hypertower \
    --local-path /home/odin/hypertower

Monitoring

Live client monitor (refreshes every 5s):

python -m v4.distributed.cli --server http://hades:8765 --token hypertower clients --watch

Faster refresh:

python -m v4.distributed.cli --server http://hades:8765 --token hypertower clients --watch --interval 2

Inspect a single client:

python -m v4.distributed.cli --server http://hades:8765 --token hypertower clients <client_id>

View job queue:

python -m v4.distributed.cli --server http://hades:8765 --token hypertower jobs

Filter by state:

python -m v4.distributed.cli --server http://hades:8765 --token hypertower jobs --state pending
python -m v4.distributed.cli --server http://hades:8765 --token hypertower jobs --state running
python -m v4.distributed.cli --server http://hades:8765 --token hypertower jobs --state failed

Submitting Jobs

Batch dispatch (dry run first):

python -m v4.distributed.batch_dispatch \
    --server http://hades:8765 --token hypertower \
    --config v4/configs/ensemble_fused.json \
    --batch v4/scripts/experiments/fusion_dim_sweep.json \
    --dry-run

Live submit:

python -m v4.distributed.batch_dispatch \
    --server http://hades:8765 --token hypertower \
    --config v4/configs/ensemble_fused.json \
    --batch v4/scripts/experiments/fusion_dim_sweep.json

Fewer reps (e.g. quick test):

python -m v4.distributed.batch_dispatch \
    --server http://hades:8765 --token hypertower \
    --config v4/configs/ensemble_fused.json \
    --batch v4/scripts/experiments/fusion_dim_sweep.json \
    --reps 3

Queue Management

Clear failed jobs:

python -m v4.distributed.cli --server http://hades:8765 --token hypertower clear --states failed

Clear running jobs (orphan cleanup):

python -m v4.distributed.cli --server http://hades:8765 --token hypertower clear --states running

Clear all jobs:

python -m v4.distributed.cli --server http://hades:8765 --token hypertower clear --all

Clear by run name:

python -m v4.distributed.cli --server http://hades:8765 --token hypertower clear --run-name experiments/fusion_dim_sweep/dim256

Cancel a specific job:

python -m v4.distributed.cli --server http://hades:8765 --token hypertower cancel <job_id>

3.8 KiB Raw Blame History