rpotter6298
512ebd13b2
- Introduced `protocol.py` for shared data models used in server/client communication, including request and response schemas for registration, job submission, and status updates. - Implemented `server.py` to manage a SQLite job queue and client registry, handling job polling, status updates, and job completion. - Created a cheat sheet for server usage, detailing commands for starting the server, submitting jobs, and monitoring clients. - Added several experiment configuration files for various training setups, including geometry vector injections and baseline ensembles.
148 lines
3.8 KiB
Markdown
148 lines
3.8 KiB
Markdown
# Distributed Server Cheat Sheet
|
|
|
|
All commands assume server is running on hades at port 8765.
|
|
|
|
## Start Server
|
|
```bash
|
|
python -m v4.distributed.server --token hypertower
|
|
```
|
|
Run inside tmux so it survives disconnects:
|
|
```bash
|
|
tmux new -s htserver
|
|
python -m v4.distributed.server --token hypertower
|
|
# Ctrl-B D to detach
|
|
tmux attach -t htserver # reattach later
|
|
```
|
|
|
|
## Local Workflow (hades as server + client)
|
|
|
|
```bash
|
|
# Terminal 1 — server
|
|
python -m v4.distributed.server --token hypertower
|
|
|
|
# Terminal 2 — client (skip rsync, results already local)
|
|
python -m v4.distributed.client \
|
|
--server http://hades:8765 --token hypertower \
|
|
--server-ssh ignored --server-path ignored \
|
|
--local-path /home/rpotter/hypertower \
|
|
--no-sync
|
|
|
|
# Terminal 3 — dispatch (exits after queuing; client picks up jobs)
|
|
python -m v4.distributed.batch_dispatch \
|
|
--server http://hades:8765 --token hypertower \
|
|
--config v4/configs/ensemble_fused.json \
|
|
--batch v4/scripts/experiments/my_batch.json
|
|
```
|
|
|
|
---
|
|
|
|
## Start Clients
|
|
|
|
**Hades (server-local, no sync):**
|
|
```bash
|
|
python -m v4.distributed.client \
|
|
--server http://hades:8765 --token hypertower \
|
|
--server-ssh ignored --server-path ignored \
|
|
--local-path /home/rpotter/hypertower \
|
|
--no-sync
|
|
```
|
|
|
|
**Apollo (remote client):**
|
|
```bash
|
|
python -m v4.distributed.client \
|
|
--server http://hades:8765 --token hypertower \
|
|
--server-ssh rpotter@hades \
|
|
--server-path /home/rpotter/hypertower \
|
|
--local-path /home/odin/hypertower
|
|
```
|
|
|
|
---
|
|
|
|
## Monitoring
|
|
|
|
**Live client monitor (refreshes every 5s):**
|
|
```bash
|
|
python -m v4.distributed.cli --server http://hades:8765 --token hypertower clients --watch
|
|
```
|
|
|
|
**Faster refresh:**
|
|
```bash
|
|
python -m v4.distributed.cli --server http://hades:8765 --token hypertower clients --watch --interval 2
|
|
```
|
|
|
|
**Inspect a single client:**
|
|
```bash
|
|
python -m v4.distributed.cli --server http://hades:8765 --token hypertower clients <client_id>
|
|
```
|
|
|
|
**View job queue:**
|
|
```bash
|
|
python -m v4.distributed.cli --server http://hades:8765 --token hypertower jobs
|
|
```
|
|
|
|
**Filter by state:**
|
|
```bash
|
|
python -m v4.distributed.cli --server http://hades:8765 --token hypertower jobs --state pending
|
|
python -m v4.distributed.cli --server http://hades:8765 --token hypertower jobs --state running
|
|
python -m v4.distributed.cli --server http://hades:8765 --token hypertower jobs --state failed
|
|
```
|
|
|
|
---
|
|
|
|
## Submitting Jobs
|
|
|
|
**Batch dispatch (dry run first):**
|
|
```bash
|
|
python -m v4.distributed.batch_dispatch \
|
|
--server http://hades:8765 --token hypertower \
|
|
--config v4/configs/ensemble_fused.json \
|
|
--batch v4/scripts/experiments/fusion_dim_sweep.json \
|
|
--dry-run
|
|
```
|
|
|
|
**Live submit:**
|
|
```bash
|
|
python -m v4.distributed.batch_dispatch \
|
|
--server http://hades:8765 --token hypertower \
|
|
--config v4/configs/ensemble_fused.json \
|
|
--batch v4/scripts/experiments/fusion_dim_sweep.json
|
|
```
|
|
|
|
**Fewer reps (e.g. quick test):**
|
|
```bash
|
|
python -m v4.distributed.batch_dispatch \
|
|
--server http://hades:8765 --token hypertower \
|
|
--config v4/configs/ensemble_fused.json \
|
|
--batch v4/scripts/experiments/fusion_dim_sweep.json \
|
|
--reps 3
|
|
```
|
|
|
|
---
|
|
|
|
## Queue Management
|
|
|
|
**Clear failed jobs:**
|
|
```bash
|
|
python -m v4.distributed.cli --server http://hades:8765 --token hypertower clear --states failed
|
|
```
|
|
|
|
**Clear running jobs (orphan cleanup):**
|
|
```bash
|
|
python -m v4.distributed.cli --server http://hades:8765 --token hypertower clear --states running
|
|
```
|
|
|
|
**Clear all jobs:**
|
|
```bash
|
|
python -m v4.distributed.cli --server http://hades:8765 --token hypertower clear --all
|
|
```
|
|
|
|
**Clear by run name:**
|
|
```bash
|
|
python -m v4.distributed.cli --server http://hades:8765 --token hypertower clear --run-name experiments/fusion_dim_sweep/dim256
|
|
```
|
|
|
|
**Cancel a specific job:**
|
|
```bash
|
|
python -m v4.distributed.cli --server http://hades:8765 --token hypertower cancel <job_id>
|
|
```
|