Today was a big day for granted patents relating to Apple’s future Headset device(s). The U.S. Patent and Trademark Office officially published a series of five newly granted patents covering various attributes that may be incorporated into Mixed Reality Headsets. For those following news of Apple’s future headset(s), there’s a lot to dive into here for rich details such as the use of head motion sensors referred to as ‘Head Odometers;’ Glint-Assisted Gaze Tracking; Event Camera-Based Gaze Tracking using Neural Networks, and more.
Eye Tracking using Head Motion Sensors or ‘Head Odometers’
Today the U.S. Patent and Trademark Office officially granted Apple a patent titled “Eye tracking using low resolution images” that relates to various methods and apparatus for eye tracking in virtual and mixed or augmented reality (VR/AR) applications using head motion sensors referred to as ‘head odometers.’
A key to providing accurate eye tracking is knowing the location of the user’s eyes with respect to the eye tracking cameras. In some embodiments of an eye tracking system, to accurately determine the location of the user’s eyes with respect to the eye tracking cameras, the controller may execute an algorithm that performs a three-dimensional (3D) reconstruction using images captured by the eye tracking cameras to generate 3D models of the user’s eyes.
The 3D models of the eyes indicate the 3D position of the eyes with respect to the eye tracking cameras, which allows the eye tracking algorithms executed by the controller to accurately track eye movement.
Methods and apparatus for tracking relative movement of a device with respect to a user’s head are described in which sensors (referred to herein as head motion sensors or head odometers) are placed at one or more positions in or on the device, for example at or near the user’s ears to primarily track pitch and at or near the bridge of the nose to primarily track y movement.
Signals from the head odometers may be used to detect movement of the device with respect to the user’s eyes. This may allow 3D reconstruction to be performed only when movement of the device with respect to the user’s eyes has been detected, thus significantly reducing power consumption by the eye tracking system.
Apple’s patent FIG. 2 below illustrates an example VR/AR HMD that implements an eye tracking system that includes sensors to detect movement of the HMD with respect to the user’s eyes.
For more on this, review Apple’s granted patent 11,442,540.
Glint-Assisted Gaze Tracker
Apple’s granted patent covers methods and apparatus for glint-assisted gaze tracking in VR/AR head-mounted displays (HMDs). Images captured by gaze tracking cameras may be input to a glint detection process and a pupil detection process, for example implemented by one or more processors of a controller of the HMD.
The glint detection process may detect glints in the images and pass the glint information to the pupil detection process, where the detected glints may be used in detecting the pupil location and contour.
The glint information may also be passed by the glint detection process to a glint-LED matching process that matches the detected glints to particular ones of the light-emitting elements of the gaze tracking system. Results of the glint-LED matching process (detected glints and LED correspondences) and pupil detection process (detected pupil ellipse) are passed to a gaze estimation process, for example implemented by one or more processors of the controller, to estimate the user’s point of gaze.
Apple’s patent FIGS. 9A through 9C graphically illustrate glint-LED matching in image space; FIGS. 11A and 11B show example results of a glint matching in image space method compared to results when a glint geometric matching method is applied in 3D space to detect and correct potential mismatches using the glint matching in image space method.
For more on this, review our original patent application report here for more graphics or review the full patent 11,442,537.
Three Additional MR Headset Patents Granted today include:
11,442,271: Display illumination systems
11,442,539: Event camera-based gaze tracking using neural networks
11,442,543: Electronic devices with monocular gaze estimation capabilities
