gui-agents

Agent S: an open agentic framework that uses computers like a human

STEP-GUI: The top GUI agent solution in the galaxy. Developed by the StepFun-GELab team and powered by StepFun’s cutting-edge research capabilities.

💻 A curated list of papers and resources for multi-modal Graphical User Interface (GUI) agents.

[ICLR 2026 Oral] ScaleCUA is the open-sourced computer use agents that can operate on cross-platform environments (Windows, macOS, Ubuntu, Android).

An understudy watches. Then performs.

The AI agent that interacts with you in the real world.

[ICLR'25 Oral] UGround: Universal GUI Visual Grounding for GUI Agents

DataClaw: Agentic Tailoring Multimodal Data from Raw Streams — coming soon (code, weights, dataset & DataClaw-val upon acceptance).

Official implementation of UI-Ins: Enhancing GUI Grounding with Multi-Perspective Instruction-as-Reasoning

[AAAI 2026] Test-Time Reinforcement Learning for GUI Grounding via Region Consistency https://arxiv.org/abs/2508.05615

[CVPR 2025] Scalable Video-to-Dataset Generation for Cross-Platform Mobile Agents

Code repo for "Read Anywhere Pointed: Layout-aware GUI Screen Reading with Tree-of-Lens Grounding"

[ACL'25 (Findings)] Explorer: Scaling Exploration-driven Web Trajectory Synthesis for Multimodal Web Agents

Curated resources about automated GUI computer-use via LLMs. Highly opinionated, focus is on quality vs quantity.

PreAct: Computer-Using Agents that Get Faster on Repeated Tasks (arXiv:2606.17929). Compile successful runs into verified state-machine programs and replay them 8.5–13× faster with no per-step LLM calls.

A Gradio-based demonstration for the Microsoft Fara-7B model, designed as a computer use agent. Users upload UI screenshots (e.g., desktop or app interfaces), provide task instructions (e.g., "Click on the search bar"), and receive parsed actions with visualized indicators overlaid on the image.

A curated hub for GUI agent papers, benchmarks, datasets, projects, and computer-use research.

Long-horizon visual puzzle-game evaluation for GUI and computer-use agents.

A Windows-only, agent-agnostic benchmark evaluating desktop agent reliability, safety, and recovery under safe, parameterizable, and reversible chaos injection.