When AI starts clicking the buttons for you.
Most of the AI your team uses today lives inside a chat window or connects to other software through an official back door. A newer kind works differently. It looks at a screen, moves a pointer, types into fields and clicks buttons, the same way a person at a desk would. The industry calls this "computer use," and it is worth understanding before a vendor demo or an eager staff member brings it to you.
What computer use actually means.
Picture handing a temp your laptop, a login and a sticky note that says "download last month's claims report from the insurance portal and put the totals in the shared spreadsheet." Computer use gives an AI model roughly that setup. It sees a picture of the screen, decides what to do next, takes an action, then looks at the new picture to check what happened.
Most automation tools talk to other software through an API, a published connection that one program offers so another can read or write data without a person in between. Those connections are fast and predictable, but only exist where a software company chose to build one. Computer use skips that requirement. If a person can do the task by looking and clicking, the model can attempt it too.
The attempt part matters. Earlier versions of these tools were slow, and they stopped to ask permission every few steps, so watching them felt like coaching someone through a website over the phone. They have gotten faster and better at carrying a long task without checking in at every turn. They are still slower than a person who knows the screen well, and they still make mistakes a person would catch.
The screens nobody built a connection for.
Every organization has a handful of systems that never made it into an integration. A school district logs into a state reporting portal. A manufacturer checks order status on three supplier websites, each with its own layout. A clinic pulls eligibility from an insurer's site, and a town office files permits through a county system that looks like it was designed twenty years ago because it was.
These are exactly the places where people still copy and paste for hours. Nobody automated them because there was no way in except the front door. Computer use offers a way through that front door, which is why it gets attention from teams who have already automated everything that had a proper connection.
Computer use is for the screens your other tools can't reach, and most organizations have more of those than they admit.
It is also a poor fit for work that already has a clean connection. If your accounting software syncs with your bank, an agent clicking through the bank's website to do the same thing is slower, costs more and breaks more often. Keep it for the systems where looking and clicking is the only way in.
Access is the hard part.
A model working a screen needs to be logged in, and logins are built on the assumption that a human is sitting there. Multi-factor authentication, the code sent to your phone, exists specifically to stop software from signing in on its own. That is the first wall most experiments hit.
There are reasonable ways around it, such as a dedicated account for the agent with limited permissions, or a browser session your IT person sets up and controls. Those are decisions for whoever manages your systems and security, not for an individual employee on a Friday afternoon. The one practice to rule out entirely is pasting a shared administrator password into an AI tool's settings so it can get in.
Permission is a separate question from access. Some outside platforms forbid automated use in their terms of service, even when a person is technically allowed to do the same clicks. Read the terms of any third-party site before you point an agent at it.
The cost of looking.
Every step in a computer use session involves the model looking at a fresh image of the screen. Images are expensive for these systems to read compared with plain text, so a task that takes forty clicks costs noticeably more than the same task done through a direct connection. The bill grows with the number of steps.
That doesn't rule it out. If a staff member spends half a day each month pulling numbers from a portal, paying more per run for an agent to do it can still be a good trade. It does mean computer use should be reserved for work with no cheaper route, and that someone should look at the usage charges after the first few weeks.
A practical way to try it.
The best first test is a small, read-only task in a system you already wish you had better access to. Reading is safer than writing. An agent that pulls a report and gets it wrong wastes some time, while one that submits a form wrong creates a mess someone has to clean up with the other organization.
Before anyone on your team runs a first experiment, have them answer these:
- Which system are we using, and does it really lack an integration or export we could use instead?
- Is the task read-only? If it writes, submits or sends anything, choose a different first task.
- Whose account will the agent use, and has the person who manages our systems approved it?
- Do the site's terms of service allow automated access?
- What would a person check to confirm the result is right, and who will check it for the first several runs?
- How long does the task take a person today, so we can compare the agent's time and cost honestly?
Then run it once while someone watches the screen. One real attempt on your own portal tells you more than any demo, including how often the agent hesitates, where it misreads a page and whether the result is worth the cost. If it holds up after a few supervised runs, write down what it does and who owns it, the same as any other automation on your list.
