Share content found on screen within apps with Apple Intelligence thanks to upcoming APIs
Siri could soon be able to view and process on-screen content thanks to new developer APIs based on technologies leaked by AppleInsider prior to WWDC.
On Monday, Apple released new documentation to help developers prepare for the arrival of upcoming Siri and Apple Intelligence features. The company’s latest developer API reveals that Siri will gain significant contextual awareness and that the virtual assistant will, at some point, be able to use information from the content currently on screen.
Siri will undoubtedly become much more useful due to Apple’s changes. The company provided a list of examples, which offer some insight as to exactly what the new-and-improved, AI-infused Siri will be able to do in the future.
Users will have the option to ask Siri questions about the web page they’re currently viewing or about a specific object in a photo. The virtual assistant will also be able to summarize documents and emails upon request, or complete texts by adding more content.
Note that some of these features were already made possible with the first iOS 18.2 developer beta, which introduced ChatGPT integration. Siri can forward a PDF, text document, or image to ChatGPT for certain actions, though only with the user’s permission.
The new developer API indicates that Apple wants to streamline this process further. Instead of the user asking Siri to send a document to ChatGPT, they will be able to ask direct questions about the page on screen or use information from it in some way. There’s plenty of room for improvement here since ChatGPT can currently only access screenshots or documents manually provided by the user.
Siri may soon gain the ability to use on-screen content.
Apple’s idea to have AI use on-screen information was apparent even before Apple Intelligence was announced at WWDC. The company’s published research, particularly concerning the Ferret model, served as an indicator of Apple’s plans in the area of artificial intelligence.
Significant importance was placed on document analysis, document understanding, and AI-powered text generation. In one of our recent reports, AppleInsider outlined the various internal test applications used while Apple Intelligence was still in development.
The test applications and environments, particularly the 1UP app, mirror many of the features currently possible via ChatGPT integration on iOS 18.2 beta. Apple also had a dedicated app for testing Smart Replies in Mail and Messages.
Siri’s new ability to complete and summarize texts, or answer questions about images, documents, and web pages was also revealed ahead of the official announcement. In our reports on the Ajax LLM, as well as the BlackPearl and Greymatter projects, we unveiled many of these features, explained how they would work, and even paraphrased Apple’s AI prompts.
It’s apparent that the iPhone maker takes artificial intelligence quite seriously, given the amount of time, research, and effort that goes into its generative AI projects. Monday’s developer API was only released to help developers prepare for new Siri features, which are rumored to make their debut in 2025 with the iOS 18.4 update.
