Anthropic’s new AI tool can take ‘CTRL’ of your PC

Artificial intelligence startup Anthropic just announced a tool that can control your computer for you. The new AI model, Claude 3.5 Sonnet, will soon be capable of interacting with desktop applications.

The company’s ‘computer use’ tool can read what’s on a user’s screen and, with prior permission, make changes for them. Intrigued? Read on.

How will Anthropic's computer use tool work?

Earlier this week, the company announced how the new extension will allow Claude to perceive and interact with computers. Anthropic also stated how developers can use the API to translate prompts into computer commands. They will further be able to use the model for repetitive tasks, conduct testing, QA and for performing open-ended results.

Instead of developing individual tools to help Claude complete its tasks, the company plans on teaching the model general computer skills. With this in place, Anthropic’s Claude could soon be able to use a wide range of tools and software programs designed for people.

What can Claude Sonnet 3.5 do?

Anthropic stated how Claude Sonnet 3.5 will be able to use a computer the way a person would. It will be able to move the cursor and click buttons among its other wide range of tasks. Other than that, the company also states how the model can be trained to type text.

ALSO READ: I tried Claude AI, and it’s now my go-to chatbot app

Anthropic cautions, however, that all of the model’s tasks will be erroneous during its teething phase and the company may take some time streamlining them. It also mentions that the tool currently struggles with activities such as scrolling, dragging or zooming in.

In a pre-recorded demo, an Anthropic employee used Claude to figure out the logistics of taking their friend for a morning walk with the views of the Golden Gate Bridge.

The demo video displayed how Claude could perform a Google search for hikes, find a map, check routes, identify sunrise time, and send a calendar invite to the friend. What’s more, it also mentioned what kind of clothes are best suitable for the activity, all without any human input.

How Anthropic is planning to address safety concerns

Despite it being a leap in technology, Anthropic mentions the need to have added security scrutiny for the tool. While there are multiple layers of permissions involved, it still requires a fair number of guardrails before deployment.

Talking about safety concerns, Anthropic points out how users will be ‘nudged away’ from having the tool make any changes on their social media. This will include posting for them, creating accounts or interacting with government-based websites. Developers may also be able to add restrictions on where a user could potentially access the tool.

ALSO READ: How Chandigarh police plan to use AI to track down criminals

With recent films like CTRL highlighting the perils of blindly checking the terms and conditions box, it may take more than permissions and external scrutiny to ensure online safety with a tool like this. That said, Anthropic highlights that several other companies like Canva, Asana and Replit have taken a step in this direction and have begun harnessing AI tasks such as graphic designing.

The model is currently available on API and developers can the computer use tool in beta. This upgrade will also make Claude one of the first AI models that can interact with a computer the way ‘people do’.