← Back to blog

Oct 17, 2026 · 8 min read

Claude's Computer Use: When AI Learns to Click

Anthropic taught Claude to use computers the way people do: by looking at a screen, moving a cursor, and clicking. It's early, error-prone, and genuinely interesting. Here's what it means.

AIAnthropicAutomation

In late 2024, Anthropic announced something genuinely new: Claude 3.5 Sonnet could use computers the way people do. Not through APIs, not through specialized integrations — by looking at a screen, moving a cursor, clicking buttons, and typing text.

"We're teaching it general computer skills — allowing it to use a wide range of standard tools and software programs designed for people." — Anthropic, October 2024

It was the first frontier AI model to offer computer use in public beta. A year later, the capability has improved, the use cases have clarified, and the limitations are better understood.

How it works

Anthropic built an API that lets Claude perceive and interact with computer interfaces. Developers get three tools:

  • Computer tool: Takes a screenshot and a goal, returns mouse and keyboard actions (move cursor, click, type, take screenshots).
  • Text editor tool: Views file contents, creates files, replaces text, undoes edits.
  • Bash tool: Returns commands to run on a computer system.

Claude doesn't directly control a computer. Instead, it suggests actions (click here, type this), and a developer-built execution layer carries them out. The model sees the result via screenshot and decides the next action.

"On OSWorld, which evaluates AI models' ability to use computers like people do, Claude 3.5 Sonnet scored 14.9% in the screenshot-only category — notably better than the next-best AI system's score of 7.8%." — Anthropic

14.9% is nowhere near human-level (which is generally 70-75% on OSWorld). But it's nearly double the next-best AI system. The capability is real, even if it's early.

What it's actually good for

A year of real-world use has clarified where computer use adds value:

Automating repetitive GUI tasks. Many business processes are "click through this legacy app, copy data from here, paste it there, submit the form." These processes don't have APIs. Computer use can automate them.

Software testing. Claude can navigate an application, fill out forms, click through workflows, and verify that the UI behaves correctly. This is useful for regression testing of applications that are hard to test programmatically.

Data entry and migration. Moving data between systems that don't have APIs is painful. Computer use can bridge the gap by interacting with both systems through their interfaces.

Research and data collection. Browsing websites, navigating to specific pages, extracting information that's behind a login or a complex navigation flow.

What it's not good for

Speed. Claude takes screenshots, analyzes them, decides on an action, executes it, takes another screenshot. Each step takes seconds. A task that takes a human 30 seconds might take Claude 5 minutes.

Reliability. At 14.9% on OSWorld, most complex tasks fail. Claude can get stuck in loops, click the wrong thing, or misinterpret a UI state. It needs supervision.

Security-sensitive operations. Giving an AI model control of a computer that has access to sensitive data or systems is risky. Anthropic recommends using it in sandbox environments for lower-risk tasks.

"Given this technology is early, developers are encouraged to explore lower-risk tasks and use it in a sandbox environment." — AWS, on Claude computer use in Amazon Bedrock

The safety question

Anthropic assessed whether computer use increases frontier risks under their Responsible Scaling Policy and found that it remains at AI Safety Level 2. The key insight: computer use is mainly a way of lowering the barrier to applying existing cognitive skills, not fundamentally increasing those skills.

The chief concerns focus on present-day harms:

  • Prompt injection through screenshots. A malicious website could contain instructions in its content that Claude follows.
  • Access to sensitive data. If Claude can see the screen, it can see whatever's on it, including passwords, financial data, or personal information.
  • Unintended actions. Claude might click something it shouldn't, delete a file, or send an email to the wrong person.

The mitigations are operational: run in a sandbox, don't give access to sensitive systems, supervise the execution, and log everything.

What this means for developers

If you're building automation tooling, computer use opens up a new category: tasks that were previously only automatable with fragile, brittle UI automation tools (Selenium, Playwright) can now be automated with a model that can adapt to UI changes.

But it's not a replacement for API integration. If an API exists, use it. APIs are faster, more reliable, and more secure than GUI automation. Computer use is for the cases where no API exists and building one isn't an option.

Practical guidance:

  1. Use it for tasks that can't be API-automated. Legacy systems, third-party apps you don't control, processes that require human-like interaction.
  2. Run it in a sandbox. A VM or container with no access to sensitive data or systems.
  3. Log every action. You need an audit trail of what Claude did, in case something goes wrong.
  4. Set clear stopping conditions. Don't let it run indefinitely. Define what "done" looks like and what "stuck" looks like.
  5. Have a human review the result. Computer use is not reliable enough for unsupervised production use.

The bigger picture

Computer use is a step toward general AI agents — systems that can accomplish real-world tasks without being narrowly programmed for each one. It's early, it's error-prone, and it's not ready for unsupervised production use. But the trajectory is clear.

The teams that will benefit from this capability are the ones that start experimenting now, understand the limitations, and build the operational practices (sandboxing, logging, supervision) that make it safe to use. The teams that wait for it to be "ready" will be starting from scratch when it is.

If you have repetitive GUI tasks in your workflow that can't be API-automated, computer use is worth a look. Just don't give it the keys to your production systems.

Need help applying this?

Schedule call →