Chapter 12 · Tools, agents and harnesses2024
Computer-using agents
Some agents operate apps through screenshots, clicks and typing, like a person at a computer.
Chapter 12 · Tools, agents and harnesses2024
Some agents operate apps through screenshots, clicks and typing, like a person at a computer.
The agent takes a screenshot, works out what to do, then moves the pointer, clicks or types. After each action it looks again.
Many agents use tools that talk to an app directly. A computer-using agent does not need that. It works through the screen, like you do.
Task: Send Asha the message “Running late”. Tap boxes on the screen to add clicks. Add the typing from the chips. Then step through and see what really happens.
Showing the blank start screen.
A mock screen with one task. A real agent re-reads the screenshot after every action, so it can correct itself.
What does the agent receive each turn?
Why one action at a time?
You type before clicking a box. What happens in the mock screen?