CUA (trycua/cua) — Summary

CUA is a multi-component platform for building, benchmarking, and deploying agents that use computers — combining a macOS VM hypervisor (Lume), a cross-platform sandbox SDK (cua), a computer-use MCP driver (cua-driver), a benchmarking harness (cua-bench), and a multi-agent orchestration CLI (cuabot). Its defining primitive is the macOS microVM running on Apple Silicon via Apple's Virtualization.Framework, giving near-native performance without Docker's shared-kernel security posture. The Python cua SDK presents a single unified API (Sandbox.ephemeral(Image.linux()|.macos()|.windows()|.android())) regardless of whether the sandbox runs locally via QEMU or on the cua.ai cloud. The cua-driver component provides background computer-use — clicking, typing, and screenshotting native macOS apps without stealing the cursor or focus, exposed as an MCP server over stdio for Claude Code, Cursor, and custom clients. CUA targets AI agent developers who need to evaluate, train, or deploy agents that interact with full OS GUIs rather than just executing code.

Differs from seeds: No seed in the catalog touches computer-use or GUI automation. All 11 seeds (superpowers, spec-kit, claude-flow, openspec, BMAD-METHOD, taskmaster-ai, agent-os, kiro, ccmemory, claude-conductor, spec-driver) are text-only coding agent frameworks. CUA sits at a fundamentally different layer: it provides the visual I/O sandbox (screenshot, click, type, gesture) that a computer-use agent requires, rather than injecting prompts or skills into a coding session. The cua-driver MCP server is structurally similar to ccmemory's MCP pattern (bundling capabilities as MCP tools), but the capabilities are screen-capture and UI-interaction rather than memory read/write.

CUA — Overview

Origin

TryCua was founded to address the gap between LLM computer-use capabilities and the infrastructure needed to run them reliably. The project grew from the realization that running macOS VMs on Apple Silicon is achievable with near-native performance via Apple's Virtualization.Framework, enabling computer-use agents to operate on actual macOS environments rather than Linux containers with X11 emulation.

Philosophy

The README tagline:

"Build, benchmark, and deploy agents that use computers"

The project philosophy is multi-target, single API: whether you're running a Linux container, a macOS VM, a Windows VM, or an Android emulator, the same Sandbox Python API works. The underlying runtime (QEMU local, cloud VM, Apple Virt.Framework) is abstracted away.

For the CUA Driver specifically:

"Drive any native macOS app in the background — agents click, type, and verify without stealing the cursor, focus, or Space, even on non-AX surfaces like Chromium web content and canvas-based tools (Blender, Figma, DAWs, game engines)."

The emphasis on background operation (not stealing cursor focus) reflects a key insight: computer-use agents deployed in production cannot disrupt the user's active session.

Components philosophy

The project is organized as a mono-repo of distinct but composable components:

• Lume (macOS VMs): infrastructure layer — create/manage macOS and Linux VMs on Apple Silicon
• cua (sandbox SDK): agent layer — unified API for sandbox interaction
• cua-driver (MCP server): integration layer — connects existing agents (Claude Code, Cursor) to CUA sandboxes via MCP
• cuabot (CLI): orchestration layer — run agents and GUI workflows in sandboxes from the command line
• cua-bench (benchmarking): evaluation layer — OSWorld, ScreenSpot, Windows Arena, custom tasks

Trajectory recording

Every CUA Driver session records a replayable trajectory — a design choice explicitly mentioned in the README. This supports RL training data collection.

Key manifesto statements

"One API for any VM or container image — cloud or local."

"Every session records as a replayable trajectory."

"Individual windows appear natively on your desktop with H.265, shared clipboard, and audio." (cuabot)

CUA — Architecture

Sandbox primitive

Two distinct primitives:

1. macOS/Linux microVM via Apple Virtualization.Framework (Lume): near-native performance on Apple Silicon; not QEMU-based for macOS targets.
2. QEMU VM (Linux/macOS/Windows/Android via cua SDK): cross-platform, local or cloud.
3. Linux container (for Image.linux() on cloud): standard container.

Startup time claims: Not explicitly benchmarked in README, but "near-native performance" implied for Lume VMs. The TypeScript adapter README states cold start ~1 second for WASM-backed tasks, but ~10ms after preload.

Distribution

• cua SDK: pip install cua (Python, requires 3.11+)
• lume: /bin/bash -c "$(curl -fsSL .../install.sh)" (Swift binary for Apple Silicon macOS)
• cuabot: npx cuabot (Node.js)
• cua-driver: /bin/bash -c "$(curl -fsSL .../install.sh)" (Swift binary + optional Rust port)
• cua-bench: uv tool install -e . within cua-bench/ subdirectory

Directory tree (repo)

cua/
├── libs/
│   ├── cua-driver/       # Background computer-use driver (Swift + optional Rust port)
│   │   ├── scripts/      # Install scripts
│   │   └── rust/         # Experimental Rust port (cua-driver-rs)
│   ├── cua-driver-fixtures/
│   ├── cuabot/           # Multi-agent computer-use sandbox CLI
│   ├── kasm/             # Kasm-based remote desktop integration
│   ├── lume/             # macOS/Linux VM management (Apple Virt.Framework)
│   │   └── Sources/      # Swift source
│   ├── lumier/           # Docker-compatible interface for Lume VMs
│   ├── python/           # Python SDK packages (cua-core, cua-agent, cua-computer, etc.)
│   │   ├── cua-agent/
│   │   ├── cua-computer/
│   │   ├── cua-computer-server/
│   │   └── cua-mcp-server/
│   ├── qemu-docker/      # QEMU via Docker wrapper
│   ├── typescript/       # TypeScript SDK
│   └── xfce/             # XFCE desktop environment setup
├── libs/cua-bench/       # Benchmarking and RL environments
├── skills/               # Claude Code skill files
├── notebooks/            # Jupyter notebooks for examples
├── pyproject.toml        # Mono-repo Python config
└── package.json          # Node.js config (cuabot)

Required runtime

• macOS on Apple Silicon (for Lume / cua-driver native features)
• Python 3.11+ (for cua SDK)
• Node.js (for cuabot)
• Docker (for QEMU-docker backend)
• QEMU (for local VM backends)

Target AI tools

• Claude Code (via cua-driver MCP server)
• Cursor (via cua-driver MCP server)
• Custom MCP clients
• Any Python-based agent (via cua SDK directly)

Host-OS posture

Significant host access for macOS targets: the computer-use driver must interact with native apps. For Linux containers, standard container isolation applies. Apple Virtualization.Framework provides VM-level isolation for macOS guests.

Isolation mechanism

Config files

• pyproject.toml (workspace Python config)
• package.json (Node.js config)
• Package.swift (Lume Swift package)

CUA — Components

Python packages (pip)

CLI tools

Claude Code skills (libs/skills/)

The repo ships at least one skill file for Claude Code integration (confirmed by skills/ directory in repo root). Specific skill names not exposed in README or top-level listing — checked skills/ directory exists.

MCP server (cua-driver)

The cua-driver exposes computer-use capabilities as MCP tools over stdio:

# Standard MCP registration
claude mcp add --transport stdio cua-driver -- cua-driver mcp

# Claude Code computer-use compat mode (grounding on window screenshots)
claude mcp add --transport stdio cua-computer-use -- cua-driver mcp --claude-code-computer-use-compat

MCP tools provided: screenshot (window-specific in compat mode), click, type, hover, key_press (at minimum — full tool list in libs/cua-driver/README.md)

Sandbox SDK primitives (Python)

sb.shell.run("command")          # Shell command execution
sb.screenshot()                  # Full screenshot
sb.mouse.click(x, y)            # Mouse click
sb.keyboard.type("text")         # Keyboard input
sb.mobile.gesture((x1,y1),(x2,y2))  # Multi-touch gesture

cuabot CLI commands

cuabot                           # Setup onboarding
cuabot claude                    # Run Claude Code in sandbox
cuabot openclaw                  # Run OpenClaw in sandbox
cuabot chromium                  # Run Chromium in sandbox
cuabot --screenshot              # Take screenshot
cuabot --type "text"             # Type text
cuabot --click <x> <y> [button] # Click

cua-bench CLI

cb image create linux-docker     # Create base image
cb run dataset <path> --agent <name> --max-parallel 4  # Run benchmark

Supported OS targets (Sandbox.ephemeral())

• Image.linux() — Linux container or VM
• Image.macos() — macOS VM (via Lume on Apple Silicon)
• Image.windows() — Windows VM
• Image.android() — Android (QEMU)

lumier

Docker-compatible interface for Lume VMs — allows using Docker CLI semantics to manage Lume-backed macOS/Linux VMs.

CUA — Uniqueness & Positioning

Differs from seeds

CUA has no counterpart in the 11 seeds. Every seed is a text-only coding agent framework operating at the LLM prompt/hook layer. CUA operates at the visual I/O layer: it gives agents eyes (screenshots), hands (click/type/gesture), and isolated OS environments to act in. The closest structural parallel within the batch (not seeds) is CubeSandbox or SWE-ReX for the "isolated execution environment" concept, but neither provides computer-use (screen capture + GUI interaction) — they provide only shell command execution. CUA's cua-driver MCP server has the same integration pattern as ccmemory (bundled MCP tools), but the tool semantics are screen/GUI rather than memory/knowledge. No seed addresses RL training data collection via trajectory recording.

Positioning

CUA targets three distinct audiences:

1. Agent developers building computer-use agents that need an SDK to provision and control VMs/containers
2. Researchers evaluating agents on standardized benchmarks (OSWorld, ScreenSpot, Windows Arena)
3. AI tool users wanting to run Claude Code or other coding agents inside sandboxed macOS/Linux VMs without compromising their host system

The cua.ai cloud service is the commercial component; the OSS repo provides the SDK and CLI tooling.

Key architectural bets

1. macOS VM via Apple Virtualization.Framework — near-native performance on M-series chips; the only open-source framework in this batch running full macOS guests.
2. Background operation (no cursor theft) — critical for production deployment where the computer-use agent cannot disrupt the user's session.
3. Single SDK, multiple runtimes — the same Sandbox API works on Linux containers, Linux VMs, macOS VMs, Windows VMs, and Android.
4. Trajectory-first design — recording every session as replayable trajectories enables RL training data collection as a first-class outcome.

Observable failure modes

• Apple Silicon only for native macOS VMs (Lume) — x86 hosts cannot run macOS guests
• Accessibility API limitations — non-AX surfaces (Chromium, Canvas-based apps) require alternative automation paths; the README calls this out explicitly
• QEMU performance on macOS — Linux/Windows VMs on macOS via QEMU are slower than native; users expecting native Mac performance get it only with Lume+macOS images
• cuabot H.265 codec requirement — streaming sandbox screens to the desktop requires H.265 support
• MCP compat mode limitations — --claude-code-computer-use-compat changes screenshot tool signature; tools calling the standard cua-driver screenshot name will break in compat mode

Cross-references

• cuabot supports Claude Code and OpenClaw
• cua-driver integrates with Claude Code and Cursor via MCP
• cua-bench evaluates agents on OSWorld and ScreenSpot benchmarks
• libs/kasm/ integrates with Kasm Workspaces for remote desktop scenarios