rpotter6298
123 lines
3.1 KiB
Markdown
123 lines
3.1 KiB
Markdown
# Distributed Server Cheat Sheet
|
|
|
|
All commands assume server is running on hades at port 8765.
|
|
|
|
## Start Server
|
|
```bash
|
|
python -m v3.distributed.server --token hypertower
|
|
```
|
|
Run inside tmux so it survives disconnects:
|
|
```bash
|
|
tmux new -s htserver
|
|
python -m v3.distributed.server --token hypertower
|
|
# Ctrl-B D to detach
|
|
tmux attach -t htserver # reattach later
|
|
```
|
|
|
|
## Start Clients
|
|
|
|
**Hades (server-local, no sync):**
|
|
```bash
|
|
python -m v3.distributed.client \
|
|
--server http://hades:8765 --token hypertower \
|
|
--server-ssh ignored --server-path ignored \
|
|
--local-path /home/rpotter/hypertower \
|
|
--no-sync
|
|
```
|
|
|
|
**Apollo (remote client):**
|
|
```bash
|
|
python -m v3.distributed.client \
|
|
--server http://hades:8765 --token hypertower \
|
|
--server-ssh rpotter@hades \
|
|
--server-path /home/rpotter/hypertower \
|
|
--local-path /home/odin/hypertower
|
|
```
|
|
|
|
---
|
|
|
|
## Monitoring
|
|
|
|
**Live client monitor (refreshes every 5s):**
|
|
```bash
|
|
python -m v3.distributed.cli --server http://hades:8765 --token hypertower clients --watch
|
|
```
|
|
|
|
**Faster refresh:**
|
|
```bash
|
|
python -m v3.distributed.cli --server http://hades:8765 --token hypertower clients --watch --interval 2
|
|
```
|
|
|
|
**Inspect a single client:**
|
|
```bash
|
|
python -m v3.distributed.cli --server http://hades:8765 --token hypertower clients <client_id>
|
|
```
|
|
|
|
**View job queue:**
|
|
```bash
|
|
python -m v3.distributed.cli --server http://hades:8765 --token hypertower jobs
|
|
```
|
|
|
|
**Filter by state:**
|
|
```bash
|
|
python -m v3.distributed.cli --server http://hades:8765 --token hypertower jobs --state pending
|
|
python -m v3.distributed.cli --server http://hades:8765 --token hypertower jobs --state running
|
|
python -m v3.distributed.cli --server http://hades:8765 --token hypertower jobs --state failed
|
|
```
|
|
|
|
---
|
|
|
|
## Submitting Jobs
|
|
|
|
**Phase 3 main grid:**
|
|
```bash
|
|
python -m v3.scripts.main.phase3.dispatch_phase3 \
|
|
--server http://hades:8765 --token hypertower
|
|
```
|
|
|
|
**Dry run (check what would be submitted):**
|
|
```bash
|
|
python -m v3.scripts.main.phase3.dispatch_phase3 \
|
|
--server http://hades:8765 --token hypertower --dry-run
|
|
```
|
|
|
|
**Other grids:**
|
|
```bash
|
|
python -m v3.scripts.main.phase3.dispatch_phase3 \
|
|
--server http://hades:8765 --token hypertower \
|
|
--grid v3/scripts/main/phase3/epoch_grid.json
|
|
|
|
python -m v3.scripts.main.phase3.dispatch_phase3 \
|
|
--server http://hades:8765 --token hypertower \
|
|
--grid v3/scripts/main/phase3/phase35_grid.json
|
|
```
|
|
|
|
---
|
|
|
|
## Queue Management
|
|
|
|
**Clear failed jobs:**
|
|
```bash
|
|
python -m v3.distributed.cli --server http://hades:8765 --token hypertower clear --states failed
|
|
```
|
|
|
|
**Clear running jobs (orphan cleanup):**
|
|
```bash
|
|
python -m v3.distributed.cli --server http://hades:8765 --token hypertower clear --states running
|
|
```
|
|
|
|
**Clear all jobs:**
|
|
```bash
|
|
python -m v3.distributed.cli --server http://hades:8765 --token hypertower clear --all
|
|
```
|
|
|
|
**Clear by run name:**
|
|
```bash
|
|
python -m v3.distributed.cli --server http://hades:8765 --token hypertower clear --run-name phase3/baseline
|
|
```
|
|
|
|
**Cancel a specific job:**
|
|
```bash
|
|
python -m v3.distributed.cli --server http://hades:8765 --token hypertower cancel <job_id>
|
|
```
|