It’s an open secret that Microsoft is gearing up to supercharge Windows 11 this summer with next-gen AI capabilities that will enable the OS to be context aware across any apps and interfaces, as well as remember everything you do on your PC to enhance user productivity and search.
These new capabilities are set to ship as part of a new app internally called “AI Explorer,” which I’m told will be unveiled during Microsoft’s special Windows event on May 20. AI Explorer will utilize next-gen neural processing unit (NPU) hardware to process these machine learning and generative AI experiences locally on the device with low latency.
The feature is also said to be exclusive to devices powered by Qualcomm’s upcoming Snapdragon X series chips, at least at first, as Intel and AMD play catchup in the NPU race. It will also require PCs with at least 16GB RAM. But what exactly is AI Explorer, and what can it do? Here’s everything I know about it so far.
AI that remembers everything you do
According to my sources, AI Explorer will run in the background and capture everything you do on your computer. It will document and triage everything it sees, no matter what apps or interfaces you’re looking at, and turn them into memories that you can recall at a later point.
For example, you can have a conversation with a friend in the WhatsApp app for Windows, and AI Explorer will record and remember the content that was on-screen and process it with AI for you to recall later. AI Explorer can also summarize conversations, emails, web pages, and general UI surfaces just by asking for it during or after the fact.
I’m told that this experience is rendered on-device and does not reach out to the cloud to process information. This is important for privacy reasons, but also for performance reasons. To reduce latency, AI Explorer will rely on NPU silicon to process content that has been recorded. I also understand that users will be able to filter out specific apps from being recorded by the AI Explorer process, or disable AI Explorer entirely.
Search for anything with natural language
Sources say AI Explorer features a UI that runs along the top of the screen, and lets users recall memories based on the user’s inputted search criteria. Because AI Explorer is recording and triaging everything you do on your computer, anything and everything becomes something you can search for.
Looked at some webpages about dinosaurs a few days ago but can’t remember the name of the websites? Just type “show me webpages about dinosaurs I viewed a few days ago” and AI Explorer will be able to pull it up. How about a chat with a friend about the Fallout TV show, but you can’t remember who it was or what app you used? Type “find me conversations I’ve had about the Fallout TV show” and AI Explorer will pull up those chats and the points in time where you had them.
Notably, AI Explorer enables the ability to search for things using natural language. No longer will users be confined to strict search terms that must be accurate for results. If you’re looking for a document that your colleague Bob sent you on Slack, just typing “find me that file Bob sent me on Slack a few weeks ago” should yield results based on memories that AI Explorer has captured.
You can also use basic search terms such as “FY24 earnings,” and every instance where FY24 earnings (and related queries) appeared on-screen will be brought up and shown in AI Explorer for you to look back at.
Contextually aware across any app
AI Explorer is able to do more than just remember the things you do on your computer, it’s also able to analyze what’s currently on-screen and provide contextual suggestions and tasks based on what it can see. For example, if you’re looking at an image in an app or webpage, AI Explorer can present suggested tasks such as “remove the background of this image using Paint,” which can be initiated with a single click.
You can also ask AI Explorer to compare two documents that are on-screen, which will lean on Microsoft Copilot to create a comparison. It can also read, understand, and help craft responses to emails or conversations, no matter the app. This capability is called Screen Understanding, and I’m told one of the big selling points of AI Explorer is that it’s supposed to work across any app, with no developer input required.
There will likely be things developers can do to enhance their apps for AI Explorer, but out of the box it should be to analyze, triage, and provide contextual suggestions for content within apps without any developer input.
A new era of AI experiences on Windows
Sources familiar with AI Explorer describe it as a truly useful AI experience that works for you in the background to enhance your workflow while using a Windows PC. With that in mind, it’s not an entirely new concept. There’s already a third-party app available on Mac called Rewind.ai that does pretty much everything AI Explorer is attempting to achieve.
The existence of Rewind.ai proves that this is a concept that can be done, and Microsoft is essentially building its own version into Windows 11 that offloads the resources required for such a feature onto NPUs to keep the load away from the CPU. AI Explorer will be the defining difference between upcoming AI PCs and a non-AI PCs, my sources say.
Microsoft CEO Satya Nadella himself has teased that personal computing is about to enter a new era thanks to AI, and that this year will be where we really see how AI can aid and improve people’s digital workflows on Windows. New Windows lead Pavan Davuluri has also previously teased what’s coming this year:
“The AI is going to be able to orchestrate across multiple apps, services, and devices, functioning as an agent in your life that can connect and keep context across entire workflows.”
AI Explorer isn’t the only new AI experience Microsoft is working on for Windows 11 in 2024 either. I understand that Microsoft also has more advanced Windows Studio Effects in the works, a new Live Captions feature that can translate languages in real-time, and more on-device generative AI experiences such as local Copilot capabilities, background removal in Photos, image generation in Paint, and more.
We will have hands-on with AI Explorer, and all the new AI features, once they are announced, likely on May 20.
