Recursive Self-Improvement Infrastructure for Physical Agents

_{English · 中文 · Documentation}

PhyAgentOS is an agent framework for embodied tasks. The Agent plans high-level Tool calls, the Forge Tool API reports what Gateway executed, the observation collector captures before/after evidence, and the task-level verifier decides whether the user-visible goal was actually achieved.

Version	Date	Update
	2026-08-30	Initial stable release of PhyAgentOS.
	2026-08-27	Forge Skills can be installed and managed independently, activated into immutable AgentTask bindings, and used through governed Query, Action, and Session Tool API lifecycles with recovery and version-scoped experience.
	2026-08-21	Unified Forge execution on the Query/Action Tool API and added AgentTask aggregation, a verifiable Skill Runtime, Resource Registry integration, and the move-arm-by-ee Skill while retaining Agent verification and evolution.
	2026-08-14	Added verified task experience, explicit workflow Skill activation, guarded Skill evolution, clustered scope-aware Lessons, and Skill-scoped advisory context for semantic verification.
	2026-08-03	Introduced the Forge execution architecture with Forge Gateway 1.0.0, immutable execution and evidence contracts, system-level semantic verification, Planner-owned recovery, crash-safe SQLite orchestration, and complete removal of the legacy Runtime execution chain.
	2026-07-05	Added benchmarking for policy-loop and target-native builtin paths, plus the Agent verification and failure-recovery service.
	2026-06-27	Added BEHAVIOR-1K support, SessionVerifier, and the explicit session-verification tool.
	2026-06-11	Cleaned protocol files and documentation, moved game scenarios to the general-game-agent branch, and focused the main line on simulation and real-robot work.
	2026-06-05	Improved onboarding, documented communication protocols, refined coding standards, and prepared game-agent benchmarking.
	2026-05-25	Established the strict PolicySkillRuntime / BuiltinSkillRuntime separation and advanced game-agent benchmarking.
	2026-05-20	Introduced the perception plugin system with sensor/perception configuration and auditable environment writeback.
	2026-05-18	Delivered the Session-Centered Runtime MVP with the initial dummy simulation pipeline.
	2026-04-29	Released the hackathon baseline with the plugin HAL and early ReKep, SAM3, grasping, and VLN workflows.

User / Channel / Scheduled Event
              │
              ▼
      AgentLoop + Planner
              │  AgentTask-bound or unbound call
              ▼
       ForgeToolClient ─────────► AgentTask SQLite + evidence
              │                         │
              │ HTTP Tool API           ▼
              ▼                  ForgeTaskVerifier
 Gateway /tools → ToolInvocation        │
              │                  verdict / PlanRevision
              ▼
 ToolEndpoint → Dora → robot/simulator

terminal AgentTask ─────► Experience Coordinator ──► evolution ledger
                                │                         │
                         Skill candidates          scoped Lessons
                                └──────────► workspace Skills

The system keeps three records separate:

1. Execution — what Query ran or what Gateway ToolInvocation was accepted and how it terminated.
2. Evidence — what PAOS observed before the first bound Action and after all bound Actions.
3. Verdict — whether each system-level success criterion is satisfied.

git clone https://github.com/PhyAgentOS/PhyAgentOS.git
cd PhyAgentOS
python -m pip install -e .

# Development and tests
python -m pip install -e ".[dev]"

Python 3.11 or 3.12 is recommended. Concrete Forge Skills and their Runtime artifacts are distributed separately.

Dora is not required for the general Agent or for paos skill install. It is required on PATH when paos skill start launches a managed Forge Skill profile. PhyAgentOS 1.0.0 uses Dora CLI v0.4.1 with dora-message v0.7.0 as its Forge Skill compatibility baseline. On Linux or macOS, install that exact release:

curl --proto '=https' --tlsv1.2 -LsSf \
  https://github.com/dora-rs/dora/releases/download/v0.4.1/dora-cli-installer.sh | sh
dora --version
# dora-cli 0.4.1
# dora-message: 0.7.0

See the user manual for Windows, Cargo installation, and lifecycle checks.

paos onboard

This creates ~/.PhyAgentOS/config.json and the default workspace at ~/.PhyAgentOS/workspace.

The configuration file is serialized in camelCase; snake_case keys are also accepted.

{
  "agents": {
    "defaults": {
      "workspace": "~/.PhyAgentOS/workspace",
      "model": "openrouter/openai/gpt-4o-mini",
      "provider": "openrouter"
    },
    "verification": {
      "serviceEnabled": true,
      "evidenceRetention": "failed",
      "maxReplansPerEpisode": 2,
      "maxVerifierCallsPerRun": 50
    },
    "evolution": {
      "enabled": true,
      "scope": "verified_forge_lineage",
      "promotionMode": "guarded_auto",
      "minSuccessfulEpisodes": 3,
      "minLessonEpisodes": 3,
      "maxLessonsPerSkill": 8,
      "maxEvolutionCallsPerRun": 20
    }
  },
  "providers": {
    "openrouter": {
      "apiKey": "YOUR_API_KEY"
    }
  },
  "forge": {
    "requestTimeoutS": 10,
    "pollIntervalS": 0.5,
    "executionTimeoutS": 300,
    "evidence": {
      "requiredImageSources": ["front"],
      "captureTimeoutS": 5,
      "postCaptureTimeoutS": 5,
      "connectionTimeoutS": 2,
      "maxArtifactBytes": 8388608,
      "associationQuality": "best_effort"
    }
  },
  "resourceRegistry": {
    "url": "https://paos-resource-manager.dev.x-era.com"
  }
}

The front source is only an example. resourceRegistry.url selects a generic package registry; it may be empty when all artifacts are installed from local bundles or a supplied static index. PAOS connects only to the Gateway URL in the manifest of the explicitly started, healthy Skill Runtime. It never starts or downloads a concrete Skill merely because the Agent starts.

Start the required installed Skill Runtime first, then choose one of the PAOS entry points:

# Interactive CLI
paos agent

# One request; the Agent may create an AgentTask and bind Tool API calls to it
paos agent -m "Inspect Forge capabilities, then place the object in the target area and verify the visible result."

# Long-running channels, Cron, Heartbeat, Agent, and Forge Tool API integration
paos gateway

Use paos status to inspect the local model/workspace configuration. Use forge_tool_context for a live ToolSpec, binding, readiness, endpoint status, and frame profile.

A typical non-off contract looks like this:

{
  "mode": "recovery",
  "goal": "The red block is inside the tray.",
  "success_criteria": [
    "The red block is visibly within the tray boundary.",
    "No other object has been displaced outside the workspace."
  ],
  "constraints": [
    "Do not move the blue block."
  ],
  "evidence_policy": {
    "required_kinds": ["rgb_image"],
    "required_sources": ["front"],
    "minimum_association": "best_effort"
  }
}

The context tool is always available for diagnostics. Task and mutating tools require one healthy, explicitly active Skill Runtime and an immutable Skill binding created from the current turn's primary activate_skill result.

Installed Skills are managed explicitly. Registry downloads require either resourceRegistry.url, PAOS_RESOURCE_REGISTRY_URL, or a supplied static index; local bundles are never replaced until their manifest, archive inventory, and locked node executables validate.

paos skill search
paos skill install <skill-name> --version <version>
# Or install a local, independently obtained bundle
paos skill install /path/to/<skill-name>-<version>.tar.gz --local
paos skill inspect <skill-name>
paos skill start <skill-name> --profile <profile>
paos skill status <skill-name>
# With no non-terminal AgentTask, select another installed Runtime
paos skill switch <other-skill-name> --profile <profile>
paos skill logs <skill-name>
paos skill stop <skill-name>

# Verify the executable locked by the installed Skill manifest
paos forge-node install <skill-name> <node-id> --archive /path/to/<node>.tar.gz
paos forge-node verify <skill-name> <node-id>

Each Forge Skill bundle declares its workflow document, required Tool IDs, named runtime profiles, and exact platform/architecture Node locks. Each locked archive has an exact SHA-256 and contains one named root-level executable. For Registry Node downloads, the verified Skill lock supplies the digest when the Registry omits that duplicate field, and the exact size is resolved before the download begins; installation records and verifies the extracted binary hash. python scripts/package_skill.py <bundle-dir> --output-dir <directory> creates a deterministic bundle for publication. The PhyAgentOS source and release packages do not bundle concrete Forge Skills, Forge nodes, models, or simulation assets; obtain only the Skills needed for a deployment and install them explicitly. The integration development guide documents Bundle layout, local validation, immutable publication order, and Registry acceptance.

When agents.evolution.enabled is true, the Agent checks the registered Skill summaries before its first tool call. activate_skill(name, role) loads the complete workflow and applicable scoped Lessons while recording an auditable task-to-Skill binding. A turn may have one primary Skill and multiple supporting Skills; only the primary can be updated automatically. Reading a SKILL.md directly does not create this binding. Verified AgentTasks are reflected on asynchronously:

• workflow-related semantic failures first form normalized observations; three independent AgentTasks with the same failure pattern are required before an abstract scoped lesson can become active and project to skills/<name>/references/LESSONS.md;
• task impossibility, verifier/evidence limitations, infrastructure failures, and uncertain causes remain diagnostic-only and never become Skill lessons;
• semantic successes support a Skill candidate;
• three independent successful AgentTasks promote a validated workspace Skill revision;
• inconclusive, invalid, review-only, and verification=off outcomes never train a Skill.

The applicable active Lessons returned by activated Skills are frozen with the AgentTask binding. Automatic verification, later PlanRevision verification, and review use that same scoped set as advisory workflow context. The verifier must ground every criterion and verdict in the task contract, execution facts, and valid evidence; a Lesson cannot satisfy a criterion or serve as an evidence reference. If no Skill was activated, no learned Lesson is supplied.

Evolution is fail-open and does not change Forge submission, execution, evidence, verification, or recovery. Built-in Skills remain immutable; promoted changes are written as workspace overrides with revision history under .paos/evolution/.

~/.PhyAgentOS/workspace/
├── AGENTS.md / SOUL.md / USER.md / TOOLS.md / SKILLS.md
├── EMBODIED.md / ENVIRONMENT.md / LESSONS.md / TASK.md
├── .paos/agent_tasks/tasks.sqlite3
├── .paos/evolution/experience.sqlite3
├── .paos/evolution/revisions/<skill>/
├── skills/<skill>/SKILL.md
├── skills/<skill>/references/LESSONS.md
└── artifacts/agent_tasks/<task_id>/
    ├── evidence_bundle.json
    ├── before_snapshot.json / after_snapshot.json
    └── evidence/

EMBODIED.md, ENVIRONMENT.md, and SceneGraph remain knowledge surfaces. They are not execution queues. With evolution enabled, the root LESSONS.md is retained as legacy/human-authored material but is not injected into Agent turns or Forge verification; only the activated task's frozen active scoped Lessons may accompany verification as non-authoritative advice. The experience database is authoritative for learned Lessons. PAOS no longer reads or generates the former Runtime Markdown queue files.

PhyAgentOS/
├── PhyAgentOS/agent/          # AgentLoop, tools, memory, experience, verifier integration
├── PhyAgentOS/forge/          # Tool API client, AgentTask aggregation, and observations
├── PhyAgentOS/skill_runtime/  # Bundle validation/install and explicit Dora lifecycle
├── PhyAgentOS/verification/   # Public contracts, request builder, engine, service
├── PhyAgentOS/channels/       # Messaging channels
├── PhyAgentOS/config/         # Configuration schema and loading
├── PhyAgentOS/templates/      # Agent knowledge/workspace templates
└── docs/                      # English, Chinese, operations, integration, Forge docs

python -m pip install -e ".[dev]"
pytest
ruff check PhyAgentOS tests
python -m compileall -q PhyAgentOS tests

Optional black-box tests may use FORGE_GATEWAY_URL to connect to a running compatible Gateway. Tests and PAOS documentation must not mutate the Gateway source or configuration.

PRs and Issues are welcome! Check our development roadmap here → Dev Plan.

Jointly developed by Sun Yat-sen University HCP Lab & Peng Cheng Laboratory & X-Era Lab

       
_{MIT License · Copyright © 2025-2026 PhyAgentOS}