Microsoft has announced a new feature coming very soon to the Copilot app on Windows 11 that will allow it to see and interact with what’s on your screen across the OS. The feature is called Copilot Vision, which first debuted in Microsoft Edge last year, but is now coming to Windows 11 as a whole.
Copilot Vision is a feature designed to let Copilot assist you in apps. You can choose to share any app you want with Copilot, which will give it permission to see what’s happening inside the app. From there, Copilot can offer suggestions, help out with tasks, and, interact, according to Microsoft.
Interact in this case means highlight buttons and areas for the user to select, not directly control the app itself. At least for now.
In a blog post, Microsoft says Copilot Vision “will allow you to call upon Copilot while working across multiple applications, browser tabs or files. It will read the screen and interact with the content. You’ll be able to use Copilot to search, change settings, organize files and collaborate on projects without switching between files or apps.”
Of course, privacy concerns immediately arise when discussing such a feature. Microsoft says Copilot Vision only works when you ask it to and that Copilot cannot see or interact with anything unless you give it permission to do so first.
Microsoft originally announced this feature for Windows last year during its Copilot+ event, where it showcased playing Minecraft with Copilot watching and providing feedback in real time. The feature now works with any app, whether that be a video game or a productivity program.
The feature will be ready to test in preview with Insiders in the Windows Insider Program as soon as next week. There is no word as to when it will be ready to ship outside of preview, but that’s likely to come later in the year.
In addition to Copilot Vision for Windows, Microsoft is also bringing Copilot Vision to mobile via the Copilot app. You’ll be able to open Copilot, then open your camera to see the world around you and have Copilot offer context and explain items that are in front of you. Pretty cool.
These new features were annoucing during Microsoft’s 50th anniversary Copilot event, alongside a handful of other new features for the AI tool. Copilot Memory will allow Copilot to learn and remember your likes and dislikes, and Copilot Actions will enable the ability for Copilot to book tickets and reservations on your behalf, like a real personal assistant!
