Add distributed server implementation and protocol definitions
- Introduced `protocol.py` for shared data models used in server/client communication, including request and response schemas for registration, job submission, and status updates. - Implemented `server.py` to manage a SQLite job queue and client registry, handling job polling, status updates, and job completion. - Created a cheat sheet for server usage, detailing commands for starting the server, submitting jobs, and monitoring clients. - Added several experiment configuration files for various training setups, including geometry vector injections and baseline ensembles.
This commit is contained in:
@@ -0,0 +1,147 @@
|
||||
# Distributed Server Cheat Sheet
|
||||
|
||||
All commands assume server is running on hades at port 8765.
|
||||
|
||||
## Start Server
|
||||
```bash
|
||||
python -m v4.distributed.server --token hypertower
|
||||
```
|
||||
Run inside tmux so it survives disconnects:
|
||||
```bash
|
||||
tmux new -s htserver
|
||||
python -m v4.distributed.server --token hypertower
|
||||
# Ctrl-B D to detach
|
||||
tmux attach -t htserver # reattach later
|
||||
```
|
||||
|
||||
## Local Workflow (hades as server + client)
|
||||
|
||||
```bash
|
||||
# Terminal 1 — server
|
||||
python -m v4.distributed.server --token hypertower
|
||||
|
||||
# Terminal 2 — client (skip rsync, results already local)
|
||||
python -m v4.distributed.client \
|
||||
--server http://hades:8765 --token hypertower \
|
||||
--server-ssh ignored --server-path ignored \
|
||||
--local-path /home/rpotter/hypertower \
|
||||
--no-sync
|
||||
|
||||
# Terminal 3 — dispatch (exits after queuing; client picks up jobs)
|
||||
python -m v4.distributed.batch_dispatch \
|
||||
--server http://hades:8765 --token hypertower \
|
||||
--config v4/configs/ensemble_fused.json \
|
||||
--batch v4/scripts/experiments/my_batch.json
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Start Clients
|
||||
|
||||
**Hades (server-local, no sync):**
|
||||
```bash
|
||||
python -m v4.distributed.client \
|
||||
--server http://hades:8765 --token hypertower \
|
||||
--server-ssh ignored --server-path ignored \
|
||||
--local-path /home/rpotter/hypertower \
|
||||
--no-sync
|
||||
```
|
||||
|
||||
**Apollo (remote client):**
|
||||
```bash
|
||||
python -m v4.distributed.client \
|
||||
--server http://hades:8765 --token hypertower \
|
||||
--server-ssh rpotter@hades \
|
||||
--server-path /home/rpotter/hypertower \
|
||||
--local-path /home/odin/hypertower
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Monitoring
|
||||
|
||||
**Live client monitor (refreshes every 5s):**
|
||||
```bash
|
||||
python -m v4.distributed.cli --server http://hades:8765 --token hypertower clients --watch
|
||||
```
|
||||
|
||||
**Faster refresh:**
|
||||
```bash
|
||||
python -m v4.distributed.cli --server http://hades:8765 --token hypertower clients --watch --interval 2
|
||||
```
|
||||
|
||||
**Inspect a single client:**
|
||||
```bash
|
||||
python -m v4.distributed.cli --server http://hades:8765 --token hypertower clients <client_id>
|
||||
```
|
||||
|
||||
**View job queue:**
|
||||
```bash
|
||||
python -m v4.distributed.cli --server http://hades:8765 --token hypertower jobs
|
||||
```
|
||||
|
||||
**Filter by state:**
|
||||
```bash
|
||||
python -m v4.distributed.cli --server http://hades:8765 --token hypertower jobs --state pending
|
||||
python -m v4.distributed.cli --server http://hades:8765 --token hypertower jobs --state running
|
||||
python -m v4.distributed.cli --server http://hades:8765 --token hypertower jobs --state failed
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Submitting Jobs
|
||||
|
||||
**Batch dispatch (dry run first):**
|
||||
```bash
|
||||
python -m v4.distributed.batch_dispatch \
|
||||
--server http://hades:8765 --token hypertower \
|
||||
--config v4/configs/ensemble_fused.json \
|
||||
--batch v4/scripts/experiments/fusion_dim_sweep.json \
|
||||
--dry-run
|
||||
```
|
||||
|
||||
**Live submit:**
|
||||
```bash
|
||||
python -m v4.distributed.batch_dispatch \
|
||||
--server http://hades:8765 --token hypertower \
|
||||
--config v4/configs/ensemble_fused.json \
|
||||
--batch v4/scripts/experiments/fusion_dim_sweep.json
|
||||
```
|
||||
|
||||
**Fewer reps (e.g. quick test):**
|
||||
```bash
|
||||
python -m v4.distributed.batch_dispatch \
|
||||
--server http://hades:8765 --token hypertower \
|
||||
--config v4/configs/ensemble_fused.json \
|
||||
--batch v4/scripts/experiments/fusion_dim_sweep.json \
|
||||
--reps 3
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Queue Management
|
||||
|
||||
**Clear failed jobs:**
|
||||
```bash
|
||||
python -m v4.distributed.cli --server http://hades:8765 --token hypertower clear --states failed
|
||||
```
|
||||
|
||||
**Clear running jobs (orphan cleanup):**
|
||||
```bash
|
||||
python -m v4.distributed.cli --server http://hades:8765 --token hypertower clear --states running
|
||||
```
|
||||
|
||||
**Clear all jobs:**
|
||||
```bash
|
||||
python -m v4.distributed.cli --server http://hades:8765 --token hypertower clear --all
|
||||
```
|
||||
|
||||
**Clear by run name:**
|
||||
```bash
|
||||
python -m v4.distributed.cli --server http://hades:8765 --token hypertower clear --run-name experiments/fusion_dim_sweep/dim256
|
||||
```
|
||||
|
||||
**Cancel a specific job:**
|
||||
```bash
|
||||
python -m v4.distributed.cli --server http://hades:8765 --token hypertower cancel <job_id>
|
||||
```
|
||||
Reference in New Issue
Block a user