Minecraft PvP Agent Lab

A remote-first, screen-only Minecraft Java PvP learning platform for private test environments. The hot path runs on the Arch/Hyprland gaming PC at LAN_HOST; the Mac is the authenticated controller, replay browser, and optional local inference client.

The implementation never reads Minecraft process memory, rejects public server destinations, and does not include anti-cheat bypasses or public matchmaking automation.

Deployed state

The worker is installed as the user service pvp-agent-worker.service on karma-pc.

Capability	Live state
Remote API	gRPC v1 on LAN_HOST:50051, bearer-token authenticated
GPU	RTX 4070 SUPER, PyTorch 2.13.0 + CUDA 13.0
Capture	xdg-desktop-portal ScreenCast/PipeWire, grim, and synthetic replay
Physical recording	Seven readable evdev keyboard/mouse interfaces
Input	uinput with per-instance queues and emergency release
Launcher	Verified Prism Launcher 11.0.2 portable build
Profiles	pvp-agent-0 through pvp-agent-3, Minecraft 1.21.8
Private server	Official 1.21.8 server, enabled user service, private-LAN two-player flat arena
Mac live arena	Two Prism clients, independent captures, PID-targeted controls, verified PvP damage
Verification	26 Mac tests pass; 25 pass and one MPS-only test skips on the gaming PC

Current input isolation is hyprland-focus-serialized: queues and held state are independent, but Hyprland still has one global focused seat. The worker reports concurrent_input_safe=false, so two-match self-play must remain gated until real two-window delivery is tested or a nested-compositor backend is installed.

The Mac two-client path is stronger: keyboard events are posted directly to each Java PID and each instance keeps its own virtual pointer state. macOS suppresses background mouse buttons, so the automation profiles bind semantic attack/use-item actions to R/F; recordings and policy outputs still use the normal attack/use-item fields.

Architecture

Mac CLI / local runner
        |
        | token-authenticated gRPC, private LAN only
        v
Arch worker (LAN_HOST)
  +-- Prism instance supervisor
  +-- Hyprland window binding
  +-- Portal/PipeWire or grim capture
  +-- evdev event recorder
  +-- per-instance input router + uinput
  +-- checksummed episode writer/replay
  +-- CUDA behavior-cloning trainer
  +-- checkpoint registry + evaluation league
  +-- PPO intent planner + frozen opponent pool

Raw frames, input events, checkpoints, and training runs stay node-local by default. The Mac receives status, metrics, and explicitly downloaded episode archives.

Mac controller

cd $HOME/code/minecraft-pvp-agent-lab
source .venv/bin/activate

pvp-agent --token-file .secrets/worker.token capabilities
pvp-agent --token-file .secrets/worker.token instances

Visible two-client Mac arena

The live private server runs on the gaming PC at LAN_HOST:25565. Start or replace both Mac clients, run a visible duel, capture both windows, or release every held input with:

pvp-agent-mac-lab up --server LAN_HOST:25565 --replace
pvp-agent-mac-lab demo --rounds 5
pvp-agent-mac-lab snapshot --output data/mac-lab/snapshots
pvp-agent-mac-lab emergency-stop

The two offline-mode identities are AgentZero and AgentOne; they are only for this allowlisted private arena. The service is not exposed through public matchmaking or a public address.

Launch a prepared profile against the loopback-only server after Prism account setup:

pvp-agent --token-file .secrets/worker.token launch \
  instance-0 pvp-agent-0 127.0.0.1:25565 \
  --account ACCOUNT_PROFILE \
  --geometry 0,0,960,540,1

Record and finalize an episode:

pvp-agent --token-file .secrets/worker.token record-start instance-0 --backend portal --fps 30
pvp-agent --token-file .secrets/worker.token record-stop EPISODE_ID

The first portal recording displays Hyprland's source picker. Select Minecraft windows in the same order as the supplied instance IDs; a restore token is stored with mode 0600 for later sessions.

Start CUDA behavior cloning and stream metrics:

pvp-agent --token-file .secrets/worker.token train-start \
  $HOME/.local/share/pvp-agent/episodes \
  --device cuda --epochs 10

pvp-agent --token-file .secrets/worker.token metrics RUN_ID

Resume after a worker restart:

pvp-agent --token-file .secrets/worker.token train-start \
  $HOME/.local/share/pvp-agent/episodes \
  --device cuda --resume $HOME/.local/share/pvp-agent/runs/RUN_ID/checkpoints/epoch-0009.pt

Emergency stop releases held keys/buttons, finalizes active recordings where possible, and stops managed clients:

pvp-agent --token-file .secrets/worker.token emergency-stop

The input router remains latched after an emergency stop. Restart the worker only after checking the desktop:

ssh LAN_HOST systemctl --user restart pvp-agent-worker.service

Episode format

Each finalized episode is atomically installed as:

episode-ID/
  manifest.json
  events/input.jsonl
  events/frames.jsonl
  frames/INSTANCE_ID/00000000.jpg
  labels/actions.jsonl
  metrics.json
  checksums.json

Events retain normalized monotonic time, raw device time, global sequence, source sequence, focus, simultaneous keys, mouse motion/buttons, and scroll. Replay validates every SHA-256 and deterministically reconstructs frame/action windows without Minecraft running.

Learning stack

The mechanics network uses:

• a visual CNN over rendered frames;
• reserved normalized OCR/HUD features;
• recent keyboard/mouse action history;
• frame and action time deltas.

It predicts mouse deltas plus movement, jump, sneak, sprint, attack, and use-item outputs. Splits are episode-disjoint. Checkpoints include parent policy, dataset version, seed, training config, Git revision, hardware backend, and validation metrics.

Self-play trains only six high-level intents: pressure, chase, strafe, reset, disengage, and reposition. Promotion requires at least 100 private held-out matches, unseen seeds, frozen opponents, and an explicit comparison with the incumbent. The previous best remains in the registry for rollback.

Remaining gaming-PC activation

The Mac two-client arena is live. To execute Minecraft clients directly under Hyprland, the four gaming-PC Prism profiles still need licensed account activation and one initial asset launch. Keep concurrent Hyprland self-play disabled while the remote worker reports concurrent_input_safe=false; this does not affect the validated PID-targeted Mac pair.

See operations for exact remote paths and service commands, and acceptance status for what is automated versus what needs live Minecraft windows.

ML 1v1 arena (gaming PC)

A separate ML-only service is implemented as a validated two-instance 1v1 on LAN_HOST, with round-safe live PPO training, a LAN dashboard on port 8787, and a spectator slot for a normal Minecraft client. Configurations above two instances are rejected as not yet validated.

pvp-agent-arena up
pvp-agent-arena status

Open http://LAN_HOST:8787?token=... from the Mac to monitor matches and training. Join LAN_HOST:25565 as GetKarma to watch the bots fight.

The learner samples planner intents during training and selects them deterministically during evaluation, while its opponent keeps a fixed scripted high-level policy. A 15 Hz scripted mechanics controller handles yaw-relative aiming, chasing, and explicit sword attack pulses.

See arena operations for the snapshot contract, local verification, and deferred live acceptance procedure.

Development

python3 -m venv .venv
source .venv/bin/activate
pip install -e '.[dev,train]'
pytest -q
ruff check src tests
python -m pip wheel --no-deps . --wheel-dir /tmp/pvp-agent-wheel

Local macOS inference is available through pvp-agent-local; the two-client harness is pvp-agent-mac-lab. Both use system window capture and Quartz with normal Screen Recording and Accessibility permissions.