Microsoft has announced a raft of updates for its Copilot AI suite, including the ability for the assistant to understand the context of what’s on your screen.
The raft of new AI-based features, some of which are in preview, are rolling out for the the Copilot+ PCs as part of a new Windows 11 update. There’s also revamped mobile apps available today.
The features include the aforementioned Copilot Vision, a new Copilot Voice feature coming to the revamped iOS and Android apps, as well as useful tools like Click To Do, Improved Windows Search and Super Resolution to enhance your photos.
Microsoft is also prepping the public for the launch of the controversial Recall screenshotting feature.
Let’s take a closer look at the pick of new Copilot features Microsoft is previewing and launching for Copilot+ PC owners as part of the Windows 11 version 24H2 rollout, and for the new mobile apps.
Save £120 on the Meta Quest 3
John Lewis is offering a huge £120 discount on the Meta Quest 3, courtesy of its refurbishment programme.
- John Lewis§
- Save £120 on RRP
- £359 in refurbished condition
Copilot Vision
The new Copilot Vision feature, which is available to beta test within Copilot Labs, is designed for the Microsoft Edge web browser.
Microsoft says this extension of the Copilot feature understands what you’re looking at on the display, enabling you to ask questions about the content in a natural speaking tone.
In a video previewing the new feature published on October 1, Microsoft used the example of how Copilot Vision could help users find a place to stay from a booking site, by asking: “Hey Copilot, I’m looking for a place to stay” to start a conversation.
From here, the assistant would take you to a particular property and ask what you think. If you say “it’s a bit pricey”, Copilot could find you something cheaper, or with a little more colour, or whatever you like, until you eventually land on something you’d like to book.
Microsoft says in a blog post: “Here your Copilot sits within Microsoft Edge. If you want it to, it can understand the page you’re viewing and answer questions about its content. It can suggest next steps, answer questions, help navigate whatever it is you want to do and assist with tasks. All the while you simply speak it to in natural language. It’s a new feature that takes the ease and practical use of Copilot to a new level.”
Microsoft says the feature will be opt-in and the tests are currently limited to pre-approved websites that are popular among the public.
Copilot Voice
The conversational capabilities are expanding with Copilot Voice for the new iOS and Android apps, enabling users to have a natural chat with the AI assistant to get a little help in a tight spot.
The example Microsoft uses today is buying a housewarming gift for a person who doesn’t drink wine, when you’re already on the way to the party. The softly spoken AI suggested a bunch of artisanal teas, or fancy olive oils. You’ll also get an explanation about why fancy olive oil is necessary and nice to have in the first place.
Microsoft says: “We’re making it easier than ever to connect with your companion using Copilot Voice. This is the most intuitive and natural way to brainstorm on the go, ask a quick question or even just vent at the end of a tough day. Your companion will be personal to you, with four voice options to choose from.”
Click to Do
Elsewhere there are plenty more new features rolling out for Windows 11 users who own Copilot+ PCs. One of which is Click to Do, which places an overlay over your entire screen, which can suggest some quick actions. In Photos, for example, you can perform a visual search with Bing, erase objects with Photos or remove a background with the Paint app – just from these suggestions.
Microsoft explains: “Click to Do is context-aware, and accessible from any Copilot+ PC screen – simply press the Windows key and click your mouse to find Click to Do in Snipping Tool or Print Screen to get started. With a clear focus on enhancing productivity, customers will see additional rich actions added in the coming months.”
Improved Windows Search
The AI is getting better at finding specifics with simple word searches. For example, on Copilot+ PCs you’ll be able to search for “BBQ party”, bringing up related items without needing to remember file names, location, spellings and loads more. it’s coming to File Explorer first and then to Windows Search.
Photos gets a Copilot
A couple of new photography features announced today include Super Resolution, which will use on-device AI to turn a blurry, pixelated, or low resolution photo into something that looks like it’s been taken on a pretty awesome, new camera at a high resolution.
Microsoft says: “Simply use the slider to increase the resolution by up to 8X, and precisely control and adjust the output. And this work won’t take all night. With the power of the 40+ TOPS NPU, super resolution in photos can upscale a photo up to 4K within seconds, for free.”
Windows users are also getting tools akin to Google’s Magic Eraser and Magic Editor. They’ll help users remove unwanted elements from the frame and add stuff that wasn’t there in the first place, like a dog in the back seat of a car, for instance.
