Skip to content

GitHub Copilot Computer Use Tutorial: Control Desktop Apps Step by Step

GitHub Copilot can now control desktop apps on Windows and macOS. Learn how to enable computer use, run a safe test, automate GUI workflows, and manage permissions.

GitHub Copilot Computer Use Tutorial: Control Desktop Apps Step by Step

On this page

GitHub Copilot can now control desktop applications instead of stopping at code, files, and terminal commands. The new computer-use feature entered public preview on October 1, 2026, in GitHub Copilot CLI and the GitHub Copilot app for Windows and macOS, allowing Copilot to read application content, click controls, type text, press keys, scroll, drag items, and move through workflows across applications. 

This makes it useful for software that has no application programming interface (API), command-line interface (CLI), or Model Context Protocol (MCP) integration. This tutorial shows how to enable GitHub Copilot computer use, test it safely with a read-only task, then give it a controlled desktop workflow without handing an AI agent unrestricted access to sensitive applications.

What GitHub Copilot computer use actually does

Computer use gives Copilot a set of tools for interacting with a graphical user interface (GUI), meaning the buttons, fields, menus, windows, and other visual controls people normally operate with a mouse and keyboard. Copilot can use accessible application information and screenshots when visual context is required, then choose actions such as clicking, typing, scrolling, dragging, or pressing keys. It can also move between applications during a single workflow. 

The distinction from ordinary Copilot commands matters. If an application already exposes a reliable API, terminal command, filesystem operation, MCP server, or dedicated browser tool, GitHub recommends using that more direct interface because it generally provides more structured and predictable results. Computer use is most useful when the software is GUI-only or when a legacy application gives an agent no cleaner way to interact with it. 

Check the requirements before enabling it

Computer use is currently a public preview, so its behavior and availability can change. GitHub documents support for local sessions on Windows and macOS through both GitHub Copilot CLI and the GitHub Copilot app. On macOS, Copilot also needs Accessibility permission to interact with application controls and Screen Recording permission when visual context from application windows is required. 

Before testing it, choose an application that does not contain sensitive information and use a task that can be reversed easily. A browser with a harmless test page, a disposable text document, or a non-production desktop application is a better first target than banking software, an accounting system, a production administration panel, or an application containing confidential customer data.

Enable computer use in GitHub Copilot CLI

Start an interactive GitHub Copilot CLI session and enable the feature with the built-in command /computer on. GitHub also provides /computer show to inspect the computer-use status and /computer off to disable it again. The feature is disabled by default, so turning it on is an explicit step rather than an automatic capability granted to every session. 

/computer on
/computer show

After enabling it, do not immediately ask Copilot to perform a complicated multi-application operation. First confirm that the computer-use components are active, then test a task where the expected result is obvious and no important data can be changed.

Enable computer use in the GitHub Copilot app

If you prefer the graphical Copilot app, open its Settings and select Computer Use. Turn on Enable Computer Use. You can also use /computer on from a session. On macOS, follow the permission prompts for Accessibility and Screen Recording, then confirm that both permissions are granted before testing an application.

The same basic workflow applies after setup: tell Copilot the outcome you want, name the application it should use, and state what it must not do. GitHub specifically recommends describing the desired outcome, the applications involved, and important constraints rather than giving an underspecified command such as “use my computer to do this.” 

Start with a read-only computer-use test

Your first prompt should prove that Copilot can see the application and navigate it without making changes. A read-only test also gives you a chance to observe the approval mechanism before allowing the agent to edit or submit anything.

Open the desktop application I specify and summarize the status
information visible in its main window.

Do not change any values.
Do not click buttons that submit, save, delete, send, or confirm anything.
Do not open another application.
Stop after you have reported the visible status.

When Copilot requests permission to control the application, check that the application and requested action match your prompt before approving it. GitHub's documentation explains that the permission system can allow access for the current session, save an application as always allowed, or decline the request. ([GitHub Docs][1])

Give Copilot a controlled desktop workflow

Once the read-only test behaves correctly, move to a small workflow with a clear starting point and a measurable end state. For example, you could ask Copilot to open a test document, change one harmless piece of text, save it, and report the final state. The useful part of the prompt is not just the action; it is the boundary around the action.

Open the test document in the desktop editor.

Change the heading from "Draft Report" to "Test Report".
Save the document.

Do not change any other text.
Do not open or modify another document.
Do not send, upload, or share the file.
Stop after confirming that the new heading is visible.

This structure gives Copilot four pieces of information: the application, the intended change, the restrictions, and the stopping condition. If the application displays an unexpected dialog or moves into a different workflow, you have a defined point at which the agent should stop rather than improvise.

Use computer use for GUI-only software

The strongest use case is software that forces a person to work through its interface because there is no practical API, CLI, or MCP integration. GitHub gives examples such as reviewing information in a legacy desktop application, updating a presentation, entering information into GUI-only software, and moving information between applications. ([GitHub Docs][2])

For example, imagine a desktop application that displays a report but cannot export its information through an API. Instead of manually opening the report, reading several fields, and copying them into another application, you can ask Copilot to perform the visual workflow. The important qualification is that this is still interface automation: Copilot must interpret the application's current state, so a changed window, unexpected dialog, or redesigned control can affect the result.

Keep approvals enabled while you test

Computer use introduces a different class of risk because the agent can interact with applications rather than merely generate text. GitHub warns that changing application versions, operating systems, window states, timing, or dynamic controls can cause computer use to select the wrong control, enter text in the wrong place, repeat an action, or fail to continue. ([GitHub Docs][2])

Do not treat Always allow as a harmless convenience. GitHub notes that saved approvals can allow future computer-use actions against the selected application without another approval prompt. Avoid saving approval for applications containing sensitive, financial, production, or enterprise data until you have evaluated the workflow carefully. ([GitHub Docs][2])

For early experiments, approve access only when you understand the requested action. If you need to stop an active operation, GitHub documents stopping the app session with the Stop control or the Esc key, while the CLI provides its own interruption controls. ([GitHub Docs][1])

Test the workflow before trusting it with real data

A successful demonstration is not the same thing as a reliable automation. Run the same task several times with different window sizes, delayed application responses, unexpected dialogs, and slightly different starting states. The goal is to discover whether Copilot can recover from normal interface variation or whether it starts interacting with the wrong control. GitHub specifically identifies timing, window state, dynamic controls, application versions, and operating-system differences as factors that can affect computer-use behavior. ([GitHub Docs][2])

Use test accounts and disposable data while evaluating the workflow. Keep production credentials out of the first experiments, and remember that visible application content can contain personal or confidential information that becomes part of the context available to Copilot. GitHub advises users to consider whether they are comfortable providing the visible content of an application as context before using computer use. ([GitHub Docs][2])

Know when not to use computer use

Computer use should not automatically replace a more deterministic integration. If the same task can be completed through an API, filesystem operation, terminal command, MCP server, or dedicated browser automation tool, that route will usually give an agent more structured data and fewer opportunities to misinterpret a screen. GitHub makes this distinction explicitly in its computer-use documentation. ([GitHub Docs][2])

The practical boundary is simple: use direct interfaces for repeatable machine-to-machine operations, and consider computer use when the application itself is the only realistic interface. That makes the new feature particularly interesting for old desktop software and internal GUI tools that have remained outside normal automation systems.

What to try next with GitHub Copilot computer use

Once a simple workflow works reliably, increase complexity one step at a time rather than giving Copilot a large unattended task. Try a cross-application workflow, such as reading a value from one test application and entering it into another, while keeping approval enabled and using non-sensitive data. Then deliberately introduce an unexpected dialog or changed window state and see whether Copilot stops safely instead of guessing.

That testing approach is more useful than simply asking whether Copilot can click a button. Computer use is now capable of operating software that was previously inaccessible to an AI coding agent, but the reliability of the surrounding workflow still depends on the interface, permissions, timing, and consequences of a wrong action. For anything that can modify important data or affect another person, keep a human review step until the behavior has been tested against the actual application and its failure modes.

Muhammad Saleem profile photo

Written by

Muhammad Saleem

I’m Muhammad Saleem, a web developer and the owner of TechWare House, a software house focused on practical web and software solutions. With over 14 years of experience, I’ve built and managed hundreds of websites and custom , PHP/MySQL, Python, Django applications. I share hands-on insights about web development, software, technology, and digital solutions on WizTechnoz.com

148 posts published

All posts by this author

Copy this post

0 Comments

No comments yet. Be the first to share your thoughts.

Join the conversation

Log in or create a free account to leave a comment. You can edit or delete your own comments any time.