Local AI Computer Use: Let an Agent Control Your PC and Keep the Controls

Since version 3.0.6, Locally Uncensored can let a model look at your screen and use the mouse and the keyboard in apps you approve. We call it computer control. It works with a local model, it is off until you switch it on, and it runs in the foreground with your real pointer, so you see every move.

Computer control is in beta. It is new and not proven in every app yet, so watch what it does and stop it when it goes wrong. Our check in the real app ran on Windows. On a Mac it is in the app but not fully tested yet, and on Linux it has not been run in the app.

Step 1: Switch It On

Go to Settings, Agent, Agent Permissions, Computer control. From then on the agent chat can use it. The agent chapter of the handbook explains the permission levels around it.

In Voice, a session can use the screen once Where it may work is set to Whole computer, and choosing that level also switches computer control on. Voice is a beta too, and English only. The Voice guide covers that side.

Step 2: Ask for Something

Write the task in the agent chat the way you would explain it to a person at your desk. For example:

• "Open Notepad and type a shopping list with milk, bread and coffee."
• "Scroll the open web page down to the comments."

Start with short tasks in one app. Small local models misjudge when a task is finished, so a task can end as stopped although the work on the screen is done.

Step 3: The Card in Every New Chat

The first time a chat wants the screen, a card in the conversation asks "Allow computer control in this chat?" with Allow and Deny. Until you click, LU looks at nothing and does nothing: no screenshot, no list of windows, no click.

The yes holds for that one chat. A new chat asks again, and so does every chat after the app was restarted. No setting, no permission level and no autonomous session skips the card, and neither does a key or a spoken word. With Deny the model is told that you did not allow it.

Before you click Allow, close what the model should not see. A picture of the screen shows the whole display, including windows of apps you have not allowed.

Step 4: Questions per App and per Action

Per app. LU asks before it touches an app for the first time. You can allow the app once, for this session or always, or decline. The question starts with the focus on Deny.

Per action. By default a session is supervised: you confirm each click and each piece of typing. The question stands directly above the input field. You answer it with the mouse, so the app LU works in keeps the focus.

Supervised or Autonomous

Switch on Autonomous in a question or in the bar, and LU asks only before anything that sends, buys, deletes or changes permissions. That holds for the session and ends with it. The autonomous mode exists on Windows only.

Sensitive Actions Always Need Your Click

Anything that sends, buys, deletes or changes permissions is asked with the reason in red, in a supervised session as in an autonomous one. The same goes for a click whose target LU cannot name, a button or link without a name, and typing when LU cannot check which field has the focus. On Windows and Linux, Alt with a letter is asked too, because it can press a button or open a menu wherever the focus is in the window. A double click on a program, an installer, a script or a shortcut asks first as well.

What LU Never Touches

These are fixed and cannot be allowed:

• Terminals and shells
• Password managers
• Sign-in and permission dialogs of the system
• System settings for security and accounts
• The Run dialog and system launchers
• Windows that run with administrator rights
• LU's own window

LU does not type into a password field, and no screenshot is taken while a password field has the focus.

Pause, Take Over, Stop

The Pause and Stop buttons in the small window do the same. In an autonomous session, moving the mouse also pauses LU.

A paused LU does nothing. It shows what it did so far and asks: Take over or Let it continue. You can add an instruction for the rest of the task. With Let it continue, LU looks at the screen again before it acts, because you may have changed something. With Take over the task ends like a stop. After a stop nothing stays pressed and nothing more is sent.

What You See While It Works

On Windows the LU window shrinks to a small window in the top right corner of the screen LU works on. It shows the current step, what the model does and the pictures it saw, and it names the two shortcuts. The edge of that screen glows softly in purple while LU works there. Neither the small window nor the glow is in the pictures the model gets.

On Linux the window stays as it is, and a bar at the top says "LU is controlling this computer" with a Stop button.

Each look at the screen appears as a small picture on the block of its tool call. The pictures are kept in memory only, at most 24, and are not written to the conversation. With a cloud model as the model that looks at the screen, the screenshots go to that provider.

Which Model Sees the Screen

The setting is Settings, Agent, Agent Permissions, Computer control, Model that looks at the screen. The default is Same as the chat model.

• A model that can see images gets the screenshots, and on Windows the list of buttons and fields of the window next to them
• A model that cannot see images works on Windows from that list alone
• On macOS and Linux there is no such list, so the model needs image input there

A local model sees images only while it is loaded with its image part, a second file next to the model. You can also pick another local model or a cloud model with image input.

We compared the two ways on Windows on two tasks, three runs each. Both finished all six runs. The runs with the picture took 497 seconds against 381 for the list alone. Two tasks are a small sample, so treat this as a hint.

In our Windows test, a local 9B chat model without image input worked from the list. Typing a note in Notepad by voice took 11 to 22 seconds from the end of the command to the spoken result. That was measured before the card in every new chat existed, so your click on Allow comes on top.

Holo 3.1 9B is in the model catalog as a screen model, marked Beta. In our test on Windows it finished a simple typing task in 1 of 3 runs. It is recommended from 16 GB of video memory and does not fit next to a 9B chat model on a 12 GB card.

Limits of a Task

A screen task ends after at most 25 steps or 8 minutes, and the time you take to answer a question counts. There is no background mode.

Where It Has Been Tested

On a Mac, macOS has to grant LU two permissions first: Accessibility and Screen Recording. They come on top of the card in every chat. Screen Recording takes effect only after LU was quit and opened again. This permission path has not been confirmed in the running app on a real Mac yet. Because LU cannot name what a click would press on a Mac, every click there is asked as a sensitive action.

Under Wayland on Linux an app cannot see other windows or send them input, and LU says so.

Related

• v3.0.6 release notes, everything else in this release
• Talk to a local LLM, the Voice guide
• Locally Uncensored vs LM Studio, the comparison page

Locally Uncensored is AGPL-3.0 licensed and built by PurpleDoubleD. Corrections welcome on GitHub Discussions or in the Discord.

In the handbook

• Agent, the tools, Agent Permissions and approvals
• Settings and troubleshooting, every settings tab and where the log file is