Earlier this morning we posted a report titled “More Details Emerge about Apple’s Mixed Reality Headset Suppliers including Sony and LG that will provide OLED on Silicon Displays.” An Apple project for the longer term involves HMD functionality specifically designed for use with a future autonomous vehicle. In an autonomous vehicle, passengers such as office workers or executives could use their transit time to work on projects, 3D documents or participate in virtual meetings. Others could simply enjoy playing a game, read emails or participate in social media or the future metaverse.
Apple patent notes in their patent filing that “embodiments of the VR system may be implemented in autonomous or ‘self-driving’ vehicles where all occupants are passengers.
The VR system may enable safer, smaller, and less expensive autonomous vehicles. Windows in vehicles are inherently unsafe and not structurally sound, and add cost to vehicles. By providing a virtual view of the real environment or of a simulated environment, the VR system may reduce or eliminate the need for windows in autonomous vehicles, allowing the vehicles to be engineered with fewer and/or smaller windows, or with no windows at all.”
Various concept drawings have surfaced over time supporting this vision, with one example presented below.
Apple’s patent FIG. 1 presented below illustrates a virtual reality (VR) system including a head mounted device (HMD) that may be used by passengers in vehicles, according to some embodiments; FIG. 2 illustrates a VR system that projects VR content to a window of a vehicle for viewing by passengers; and FIG. 5 illustrates projecting VR content so that it appears to the viewer to be in space in front of the vehicle.
Apple’s patent FIG. 8 below is a block diagram illustrating active systems in a vehicle that may be used in a VR system to provide synchronized physical effects for passengers that provide an immersive VR experience.
Apple’s patent FIG. 18 below illustrates a VR system monitoring a passenger using a VR system in a vehicle; FIG. 19 illustrates a VR application that adapts a VR experience to accommodate a passenger based on passenger preferences and passenger sensor data.
In some embodiments, the VR system may provide an interface that allows the passenger to specify their preferences for a VR environment to be experienced (e.g., the passenger may request a relaxing, normal, or exciting experience).
In some embodiments, while using the HMD to experience the VR world generated by the VR application on the VR controller, various sensors may be used to monitor the passenger for signs of discomfort or motion sickness (e.g., paleness, sweating, fidgeting, swallowing, burping, pulse rate, breath rate, eye motions, etc.); the passenger sensor data #1830 may be provided to the VR controller via wired or wireless connections.
In some embodiments, the sensors may include sensors #1832 in or on the HMD, for example internal cameras that monitor the passenger’s eyes or other portions of the passenger’s face, external cameras that monitor other parts of the passenger’s body such as the arms and hands, Inertial-measurement Units (IMUs) that detect and track motion of the passenger’s head, and/or sensors that monitor physiological responses such as sweating, swallowing, and breath rate.
Apple began work on this advanced HMD system specifically designed for future autonomous vehicles (Apple’s Project Titan) back in 2016. In today’s follow-up patent filing by Apple, they cancelled 20 patent claims and replaced them with 20 all-new patent claims to better describe and protect their invention. Below are just three of the new patent claims:
A system, comprising: one or more processors; and a memory storing instructions that, when executed on or across the one or more processors, cause the one or more processors to: obtain inputs from one or more sources in a vehicle, wherein the inputs include indications of visual cues of an environment outside the vehicle, and wherein the environment comprises other vehicles; generate virtual content based at least in part on the inputs, wherein the virtual content comprises virtual representations of the other vehicles, and wherein motions and accelerations of the virtual content are synchronized with motions and accelerations indicated in the inputs; and send the virtual content to a display device for a passenger in the vehicle.
A method, comprising: performing, with one or more computing devices: obtaining inputs from one or more sources in a vehicle, wherein the inputs include indications of visual cues of an environment outside the vehicle, and wherein the environment comprises other vehicles; generating virtual content based at least in part on the inputs, wherein the virtual content comprises virtual representations of the other vehicles, and wherein motions and accelerations of the virtual content are synchronized with motions and accelerations indicated in the inputs; and sending the virtual content to a display device for a passenger in the vehicle.
One or more non-transitory, computer-readable storage media storing instructions that, when executed on or across one or more processors, cause the one or more processors to: obtain inputs from one or more sources in a vehicle, wherein the inputs include indications of visual cues of an environment outside the vehicle, and wherein the environment comprises other vehicles; generate virtual content based at least in part on the inputs, wherein the virtual content comprises virtual representations of the other vehicles, and wherein motions and accelerations of the virtual content are synchronized with motions and accelerations indicated in the inputs; and send the virtual content to a display device for a passenger in the vehicle.
To review the full details OF Apple’s patent application number 20220222904 and the other 18 new patent claims, click here.
Considering that this is a patent application, the timing of such a product to market is unknown at this time.
