Today the US Patent & Trademark Office published a patent application from Apple that relates to taking user Avatars to the next level for use with future Mixed Reality Headsets wherein users are in Extended Reality (XR) environments. Beyond creating an Avatar (Animoji) of a user’s head, next generation Avatar creation will allow users to scan in their hands and/or full body so that they could move within Extended Reality (XR) 3D environments.
While Apple defines Mixed Reality and Virtual Reality, Apple defines Extended Reality (XR) this way: An extended reality (XR) environment refers to a wholly or partially simulated environment that people sense and/or interact with via an electronic system. In XR, a subset of a person’s physical motions, or representations thereof, are tracked, and, in response, one or more characteristics of one or more virtual objects simulated in the XR environment are adjusted in a manner that comports with at least one law of physics. For example, an XR system may detect a person’s head turning and, in response, adjust graphical content and an acoustic field presented to the person in a manner similar to how such views and sounds would change in a physical environment.
In some situations (e.g., for accessibility reasons), adjustments to characteristic(s) of virtual object(s) in an XR environment may be made in response to representations of physical motions (e.g., vocal commands). A person may sense and/or interact with an XR object using any one of their senses, including sight, sound, touch, taste, and smell. For example, a person may sense and/or interact with audio objects that create a 3D or spatial audio environment that provides the perception of point audio sources in 3D space.
In another example, audio objects may enable audio transparency, which selectively incorporates ambient sounds from the physical environment with or without computer-generated audio. In some XR environments, a person may sense and/or interact only with audio objects.
Apple’s patent covers methods and interfaces for interacting with 3D environments that include at least some virtual elements (e.g., applications, augmented reality environments, mixed reality environments, and virtual reality environments).
Apple’s patent FIG. 7A below illustrates an enrollment process akin to setting up Face ID initially and then goes on to allow the user to scan in their hands (FIG. 7H) so that they could be used to interact with virtual objects in a VR environment; FIGS. 9B, 9C, 9E and 9F illustrate the user’s scanned in hand now being able to interact with virtual objects.
Apple’s patent FIG. 12A illustrates the user enrollment for creating a full bodied Avatar. Apple further notes that the user can put on a headset device (e.g., an HMD) and use the headset with its multiple cameras to collect image and/or depth data for other physical features of a user, such as the user’s hands, feet, torso, arms, shoulders, etc.
Further to patent FIG. 12A, avatar #1220 displayed in the lower iPad includes portions (1220-1, 1220-2, 1220-3, and 1220-4), which are displayed as a virtual representation of the user (#700) (above the iPad). Portion #1220-1 forms the avatar’s left forearm and hand and has an appearance (e.g., position, pose, orientation, color, shape, etc.) that is determined based on various aspects (e.g., position, pose, orientation, color, movement, etc.) of the user’s left forearm and hand that are detected, for example, by the camera #904 of the iPad.
Similarly, portion #1220-2 forms the avatar’s right forearm and hand and has an appearance that is determined based on various aspects of the user’s right forearm and hand #700-2.
Portion $1220-3 forms the avatar’s head and shoulders region and has an appearance that is determined based on various aspects of user’s head #700-3 and shoulders.
Portion 1220-4 forms the remaining portions of avatar 1220 and has a visual appearance that is different from that of portions 1220-1 to 1220-3. For example, as illustrated in FIG. 12A, portion #1220-4 has an appearance formed of elements #1225 having various colors and, optionally, shapes (optionally different shapes; optionally overlapping or not overlapping), whereas portions #1220-1 to #1220-3 have an appearance that is visually representative of (e.g., similar to, having a shape of one or more human features) the corresponding portions of the user.
The patent goes deeper into every detail of the full Avatar.
Being that this is a new concept, Apple drills down and provides readers with rich details, scenarios and a lot more patent figures related to this invention. To go down the rabbit hole, review Apple’s patent application 20220262080 for more.
