AI is going to shape our lives in the years to come, and mobile phones are at the fore-front of what we, the consumers, experience. Features such as Object Eraser, Circle to Search, and identifying landmarks and buildings in images are just a taste of what we can expect as processors and the AI technology itself, become more advanced.
We were lucky enough to have an interview with Chris Bergey, SVP and GM of the Client Business at Arm, to discuss where on-device AI is headed next. Seeing as Arm provides the foundational technology to chipmakers such as MediaTek, Qualcomm, and Apple, which powers 99% of the world’s smartphones, the company should have an intriguing outlook on AI and its increasing role in our smartphones.
Who is Chris Bergey?
With more than twenty years of experience in semiconductors in the mobile and wireless sectors, Chris is now a Senior Vice President and General Manager of Arm’s Client Business division. He’s held positions at well-known technology companies such as Western Digital, Broadcom, and AMD, among others. He’s uniquely placed to give us an insider’s view of where on-device AI is heading. We really appreciate the opportunity to interview someone of Chris’s caliber.
Text-to-Video and Image-to-Video Generation on Mobile Devices
Question: Arm processors are already at the core of mobile innovations, but how close are we to seeing text-to-video and image-to-video generation processed directly on mobile devices instead of relying on cloud-based solutions?
Chris Bergey: “The world of AI is moving at a rapid pace. As an example, at the start of 2024 we saw examples of Meta’s Llama 2 7B model running on a device generating text at 10 to 15 tokens per second. Now, we see the Llama3.2 1B model running on-device, generating text at 50 tokens per second and providing exceptional user experiences.
“We’re also seeing rapid advancements in image generation capabilities with many devices able to generate images in less than one second. These advancements are a result of complex model optimization and software optimizations on the device, through specialized libraries, such as Arm KleidiAI, which utilize the underlying compute platform to its full capabilities.
Follow-up question: What technical challenges does Arm face in bringing such resource-heavy tasks onto mobile platforms, and how does your roadmap address these challenges?
Chris Bergey: “One challenge with GenAI models is the need to reduce the footprint of the model so that it doesn’t take up too much memory on the device. In addition to this, it is equally important to maximize the bandwidth between DRAM and your chosen compute solution, to achieve ultimate performance.
“Arm is actively working across all these areas to accelerate the future of GenAI. Looking at the current trajectory of the market, it feels inevitable that short-form video will be generated on-device in the near future, based on the intersection of model optimization and software optimization to exploit underlying AI features in today’s chipsets.”
Question: As mobile users demand more immersive AI-driven experiences, such as real-time video generation from text or images, how is Arm enabling OEMs to optimize these capabilities for both performance and battery life?
Chris Bergey: “Our focus has always been on the user experience and putting the most intelligent, dynamic, and immersive AI into consumers’ hands. As we see a renewed focus on performance and platform capabilities, Arm is continuously working with its OEM partners to unlock new GenAI use cases on CPU and GPU by building new features into the architecture to push the boundaries of what we can enable our partners to do.
“Arm evolves its CPU architecture annually, and for over a decade, we’ve been working on features that positively optimize it for AI applications. Specifically, in Armv9, we’re building features into the architecture to push the boundaries of what we can enable software developers to do, and we see Armv9 becoming the de facto standard for premium mobile experiences. Two key Armv9 architecture features are
“SVE2 and SME2, which together combine to enable fast and efficient AI workloads on Arm CPU. SVE2 meets the needs of high-performance compute, delivering double-digit performance and efficiency gains across a range of real use cases such as image and video processing – from faster sharpening of images to more efficient video decode. SME2 accelerates AI workloads by enabling matrix math operations for faster processing, efficient power consumption and high throughput at low latency, which is key for AI inference. Armv9 CPUs, now on their 4th generation, are gaining momentum and making their way into important apps and frameworks, bringing benefits in terms of performance and efficiency.
Follow-up Question: How do Arm’s CPU and GPU architectures address the balance between processing power and energy efficiency for AI workloads on smartphones?
Chris Bergey: “From a GPU perspective, we pay close attention to the evolution in mobile gaming content trends to ensure our GPU architecture can address the needs of developers and ecosystem partners. This also requires close collaboration with our OEM partners to deliver value in the right areas. Bringing AI to gaming is a key aspect of this and requires AI compute to be as efficient and performant as the platform allows. In addition to our GPUs delivering exceptional performance and efficiency benefits for the very best user experiences, our optimized drivers ensure that our solutions are tuned to run the latest gaming titles, unlocking performance and efficiency with GPU-based AI.”
Built-in Security Features in the Age of AI
Question: As AI processing becomes more integrated into mobile devices, how critical are Arm’s built-in security features in preventing potential threats from hackers, especially given the increased local processing of sensitive data?
Chris Bergey: “If AI is to meet the promise we all anticipate, AI workloads will need to be embedded into all compute platforms from cloud to edge. There will be a role for all types of compute, and developers will carefully choose where it makes sense to run their workloads, based on a number of factors. One of these factors is privacy and security. AI will both increase the value of personal data but also create a new wave of high-value AI models. Both of these will draw attention from hackers, who will be motivated to gain access to these valuable assets, which cost hundreds of millions of dollars to train over a period of months using the latest training machines.
“Arm’s latest Armv9 solutions are designed with security in mind and represent a step change in computing infrastructure, making software on Arm-based devices safer and more reliable. Features such as Pointer Authentication, Branch Target Identification and MTE can help to eliminate whole classes of bugs that hackers have exploited in the past.
Follow-up question: Could you provide insight into how Arm’s TrustZone or other security solutions are designed to protect both AI data and the models themselves from local or remote exploitation?
Chris Bergey: “When it comes to processing at the edge, a whole host of benefits become apparent including latency, power consumption and cost. But the key challenge remains: protecting AI models and personal data across the hybrid mode when user data, as well as the processing of it, resides across both the cloud and on local devices, is paramount. Arm’s isolation technologies such as TrustZone, as well as the Arm Confidential Compute Architecture (CCA) protects all data and code wherever computing happens, to unlock the power and potential of data and AI from cloud to edge.”
Question: In the context of AI workloads, especially for consumer devices, where do you see the biggest security vulnerabilities that Arm’s architecture addresses today?
Chris Bergey: “Trust in AI will be critical for it to reach its full potential, and a balance of autonomy and human oversight will be essential. As we become more reliant on AI systems, and as AI deployments shift from creative use cases to more business-critical and life-changing ones, this space is evolving quickly, with model sizes, capabilities as well as operator types, and quantization changing rapidly.
Follow-up question: Are there specific enhancements in your latest designs that cater specifically to the growing threats posed by the integration of AI into everyday mobile tasks?
Chris Bergey: “It’s important that the bar for security in all connected devices remains critically high, and we do this by building isolation technologies into the Arm architecture, ensuring privacy where it’s needed and giving only the right users access to privileged information. From a mobile perspective, we are committed to supporting developers deliver better software through removing bugs so that the intended outcomes of any AI-enabled application cannot be hijacked, protecting data and computing in trusted execution environments from bad actors, and also a relentless drive for PPA (Power, Performance and Area) improvements on edge devices to enable more local and personal computing to happen.”
Empowering Consumers with AI on Smartphones
Question: Arm has been a major player in making advanced computing accessible on mobile platforms. What are the next key innovations Arm is working on to put more AI capabilities directly into consumers’ hands without needing to rely on cloud processing?
Chris Bergey: “Arm’s power efficiency DNA sparked the mobile-phone revolution, and the same design philosophy is the foundation for all device types, with AI starting on the power-efficient CPU that is pervasive across the world’s devices. We believe that AI will become a part of everyday mobile engagement. It will enable new foreground activities in everyday apps, as well as running in background tasks, ensuring that your system is running as performantly and efficiently as possible. AI will be heterogenous, across all IP (CPU, GPU and NPU) where the best IP is selected for a particular task.
Follow-up question: How do you envision AI-powered smartphones evolving in the next few years, particularly when it comes to on-device AI applications that go beyond current uses like voice assistants or photography enhancements?
Chris Bergey: “Multimodal models will also become commonplace as we interact with our apps using voice, images, video, and text. The next generation of AI use cases will become highly personalized, with aggregated data across multiple apps being collated to tailor your user experience based on your exact needs, e.g., a personal fitness app making recommendations based on real-time health data.
“Arm believes that the capabilities of AI should be accessible to all, and it is for this reason that we develop software solutions that are scalable across tiers of mobile, but also across generations of our IP.”
Question: Given the rapid pace of AI developments, how does Arm collaborate with smartphone manufacturers and software developers to ensure that AI tools and capabilities are not only powerful but also user-friendly and accessible to the average consumer?
Chris Bergey: “Arm is committed to ensuring that the 20 million developers developing on Arm have the capabilities to write easier, simpler, faster, and more secure software. Earlier this year, we launched Arm Kleidi, which includes Kleidi technology and software libraries, alongside community engagements to give the developer ecosystem automatic access to leading performance. A key part of this is the Arm Kleidi Libraries, such as Arm KleidiAI, which are being integrated directly into popular AI frameworks. Arm KleidiAI will unleash Arm Cortex-A and Neoverse CPU performance across AI workloads – Arm is working with a number of AI framework developers to enable performance in the key frameworks and is doing the direct integration work, so developers don’t have to.
Follow-up question: Could you share examples of where we might soon see AI capabilities that consumers can leverage directly, thanks to Arm’s chip innovations?
Chris Bergey: “KleidiAI has already been integrated into many of the world’s most popular AI frameworks, including MediaPipe (through XNNPACK), PyTorch, ExecuTorch, and Hunyuan. These integrations are accelerating the performance of leading LLMs like Meta’s Llama 3.2 on Arm-based mobile devices. The integration of optimized libraries with the key AI frameworks unlocks the best possible performance on Arm, enabling the innovation behind the most exciting use cases of tomorrow.”
We hope you’ve enjoyed reading this in-depth interview with Arm’s Chris Bergey on where on-device AI is headed next. We also thank Arm for the interview opportunity.
