XR_ANDROID_light_estimation OpenXR extension

Name String

XR_ANDROID_light_estimation

Extension Type

Instance extension

Registered Extension Number

701

Revision

1

Extension and Version Dependencies

OpenXR 1.0

Last Modified Date

2025-01-17

IP Status

No known IP claims.

Contributors

Jared Finder, Google

Cairn Overturf, Google

Spencer Quin, Google

Levana Chen, Google

Nihav Jain, Google

Salar Khan, Google

Scott Chung, Google

Overview

This extension allows the application to request data representing the lighting of the real-world environment around the headset. This information can be used when rendering virtual objects to light them under the same conditions as the scene they're placed in.

Inspect system capability

An application can inspect whether the system supports light estimation by chaining an XrSystemLightEstimationPropertiesANDROID structure to the XrSystemProperties when calling xrGetSystemProperties. 

typedef struct XrSystemLightEstimationPropertiesANDROID {
    XrStructureType    type;
    void*              next;
    XrBool32           supportsLightEstimation;
} XrSystemLightEstimationPropertiesANDROID;

Member Descriptions

  • type is the XrStructureType of this structure.
  • next is NULL or a pointer to the next structure in a structure chain. No such structures are defined in core OpenXR or this extension.
  • supportsLightEstimation is an XrBool32, indicating if the current system supports light estimation.

An application can inspect whether the system is capable of supporting light estimation by extending the XrSystemProperties with XrSystemLightEstimationPropertiesANDROID structure when calling xrGetSystemProperties.

If the system is not capable of supporting light estimation, then it will return XR_FALSE for supportsLightEstimation, and XR_ERROR_FEATURE_UNSUPPORTED from xrCreateLightEstimatorANDROID.

Valid Usage (Implicit)

Create a light estimator handle

XR_DEFINE_HANDLE(XrLightEstimatorANDROID)

The XrLightEstimatorANDROID handle represents a light estimator. This handle can be used to access light estimation information using other functions in this extension.

The xrCreateLightEstimatorANDROID function is defined as: 

XrResult xrCreateLightEstimatorANDROID(
    XrSession                                   session,
    XrLightEstimatorCreateInfoANDROID*          createInfo,
    XrLightEstimatorANDROID*                    outHandle);

Parameter Descriptions

The application can use xrCreateLightEstimatorANDROID function to create a light estimator.

The returned light estimator handle may be subsequently used in API calls. If an application wants to indicate to the runtime that it is done accessing light estimation data it must destroy the handle using xrDestroyLightEstimatorANDROID.

Valid Usage (Implicit)

Return Codes

Success

  • XR_SUCCESS
  • XR_SESSION_LOSS_PENDING

Failure

  • XR_ERROR_FUNCTION_UNSUPPORTED
  • XR_ERROR_RUNTIME_FAILURE
  • XR_ERROR_INSTANCE_LOST
  • XR_ERROR_SESSION_LOST
  • XR_ERROR_OUT_OF_MEMORY
  • XR_ERROR_HANDLE_INVALID
  • XR_ERROR_LIMIT_REACHED

The XrLightEstimatorCreateInfoANDROID structure describes the information to create an XrLightEstimatorANDROID handle. 

typedef struct XrLightEstimatorCreateInfoANDROID {
    XrStructureType    type;
    void*              next;
} XrLightEstimatorCreateInfoANDROID;

Member Descriptions

  • type is the XrStructureType of this structure.
  • next is NULL or a pointer to the next structure in a structure chain. No such structures are defined in core OpenXR or this extension.

Valid Usage (Implicit)

xrDestroyLightEstimatorANDROID function releases the estimator and any underlying resources. 

XrResult xrDestroyLightEstimatorANDROID(
    XrLightEstimatorANDROID                     estimator);

Parameter Descriptions

Valid Usage (Implicit)

Thread Safety

  • Access to estimator, and any child handles, must be externally synchronized

Return Codes

Success

  • XR_SUCCESS

Failure

  • XR_ERROR_FUNCTION_UNSUPPORTED
  • XR_ERROR_HANDLE_INVALID

Access light estimation data

The xrGetLightEstimateANDROID function is defined as: 

XrResult xrGetLightEstimateANDROID(
    XrLightEstimatorANDROID                     estimator,
    const XrLightEstimateGetInfoANDROID*        input,
    XrLightEstimateANDROID*                     output);

Parameter Descriptions

Valid Usage (Implicit)

Return Codes

Success

  • XR_SUCCESS
  • XR_SESSION_LOSS_PENDING

Failure

  • XR_ERROR_FUNCTION_UNSUPPORTED
  • XR_ERROR_VALIDATION_FAILURE
  • XR_ERROR_RUNTIME_FAILURE
  • XR_ERROR_HANDLE_INVALID
  • XR_ERROR_INSTANCE_LOST
  • XR_ERROR_SESSION_LOST
  • XR_ERROR_OUT_OF_MEMORY
  • XR_ERROR_TIME_INVALID

A XrLightEstimateGetInfoANDROID describes the information required to get light estimation data. 

typedef struct XrLightEstimateGetInfoANDROID {
    XrStructureType    type;
    void*              next;
    XrSpace            space;
    XrTime             time;
} XrLightEstimateGetInfoANDROID;

Member Descriptions

  • type is the XrStructureType of this structure.
  • next is NULL or a pointer to the next structure in a structure chain.
  • space is the XrSpace defining the reference space in which the returned light direction and spherical harmonics and cubemap rotation are expressed.
  • time is the XrTime that describes the time at which the application wishes to query the light estimation.

Valid Usage (Implicit)

XrLightEstimateANDROID structure contains light estimation data. 

typedef struct XrLightEstimateANDROID {
    XrStructureType                type;
    void*                          next;
    XrLightEstimateStateANDROID    state;
    XrTime                         lastUpdatedTime;
} XrLightEstimateANDROID;

Member Descriptions

To obtain light estimate information for the environment lighting cubemap, ambient light, spherical harmonics, and the primary directional light, applications can chain instances of the following structures, XrEnvironmentLightingCubemapANDROID, XrAmbientLightANDROID, XrSphericalHarmonicsANDROID and XrDirectionalLightANDROID respectively onto XrLightEstimateANDROID::next.

Valid Usage (Implicit)

The XrEnvironmentLightingCubemapANDROID structure contains light estimation data in a cubemap format about the environment lighting of the scene. 

typedef struct XrEnvironmentLightingCubemapANDROID {
    XrStructureType                type;
    void*                          next;
    XrLightEstimateStateANDROID    state;
    const float*                   cubemapRightImage;
    const float*                   cubemapLeftImage;
    const float*                   cubemapTopImage;
    const float*                   cubemapBottomImage;
    const float*                   cubemapFrontImage;
    const float*                   cubemapBackImage;
    uint32_t                       resolution;
    XrQuaternionf                  rotation;
    XrTime                         centerExposureTime;
} XrEnvironmentLightingCubemapANDROID;

Member Descriptions

  • type is the XrStructureType of this structure.
  • next is NULL or a pointer to the next structure in a structure chain. Valid structures are XrSphericalHarmonicsANDROID, XrAmbientLightANDROID, and XrDirectionalLightANDROID.
  • state is the XrLightEstimateStateANDROID representing the state of the light estimate.
  • cubemapRightImage is a float* representing the R32G32B32_SFLOAT buffer of the right side of the cubemap. Note this is HDR so float values can be greater than 1.0.
  • cubemapLeftImage is a float* representing the R32G32B32_SFLOAT buffer of the left side of the cubemap. Note this is HDR so float values can be greater than 1.0.
  • cubemapTopImage is a float* representing the R32G32B32_SFLOAT buffer of the top side of the cubemap. Note this is HDR so float values can be greater than 1.0.
  • cubemapBottomImage is a float* representing the R32G32B32_SFLOAT buffer of the bottom side of the cubemap. Note this is HDR so float values can be greater than 1.0.
  • cubemapFrontImage is a float* representing the R32G32B32_SFLOAT buffer of the front side of the cubemap. Note this is HDR so float values can be greater than 1.0.
  • resolution is a uint32_t representing the width and height of the image of each side of the cubemap.
  • rotation is a XrQuaternionf representing the rotation of the cubemap.
  • centerExposureTime is a XrTime representing the time the cubemap was captured. Useful for interpolation between cubemaps.

Valid Usage (Implicit)

  • The XR_ANDROID_light_estimation extension must be enabled prior to using XrEnvironmentLightingCubemapANDROID
  • type must be XR_TYPE_ENVIRONMENT_LIGHTING_CUBEMAP_ANDROID
  • next must be NULL or a valid pointer to the next structure in a structure chain
  • state must be a valid XrLightEstimateStateANDROID value
  • cubemapRightImage must be a pointer to a valid float value
  • cubemapLeftImage must be a pointer to a valid float value
  • cubemapTopImage must be a pointer to a valid float value
  • cubemapBottomImage must be a pointer to a valid float value
  • cubemapFrontImage must be a pointer to a valid float value
  • cubemapBackImage must be a pointer to a valid float value

XrAmbientLightANDROID structure contains light estimation data about the ambient light in the scene. 

typedef struct XrAmbientLightANDROID {
    XrStructureType                type;
    void*                          next;
    XrLightEstimateStateANDROID    state;
    XrVector3f                     intensity;
    XrVector3f                     colorCorrection;
} XrAmbientLightANDROID;

Member Descriptions

Valid Usage (Implicit)

XrSphericalHarmonicsANDROID structure contains spherical harmonics representing the lighting of the scene. 

typedef struct XrSphericalHarmonicsANDROID {
    XrStructureType                    type;
    void*                              next;
    XrLightEstimateStateANDROID        state;
    XrSphericalHarmonicsKindANDROID    kind;
    float                              coefficients[9][3];
} XrSphericalHarmonicsANDROID;

Member Descriptions

Valid Usage (Implicit)

XrDirectionalLightANDROID structure contains light estimation data. 

typedef struct XrDirectionalLightANDROID {
    XrStructureType                type;
    void*                          next;
    XrLightEstimateStateANDROID    state;
    XrVector3f                     intensity;
    XrVector3f                     direction;
} XrDirectionalLightANDROID;

Member Descriptions

Valid Usage (Implicit)

The XrSphericalHarmonicsKindANDROID enumeration identifies to the runtime, what type of spherical harmonics the application is requesting. 

typedef enum XrSphericalHarmonicsKindANDROID {
    XR_SPHERICAL_HARMONICS_KIND_TOTAL_ANDROID = 0,
    XR_SPHERICAL_HARMONICS_KIND_AMBIENT_ANDROID = 1
} XrSphericalHarmonicsKindANDROID;

The enums have the following meanings:

Enum

Description

XR_SPHERICAL_HARMONICS_KIND_TOTAL_ANDROID

The spherical harmonic coefficients represent the environment light radiance function, including the contribution from the main light.

XR_SPHERICAL_HARMONICS_KIND_AMBIENT_ANDROID

The spherical harmonic coefficients represent the environment light radiance function, excluding the contribution from the main light.

The XrLightEstimateStateANDROID enumeration identifies to the runtime, what type of spherical harmonics the application is requesting. 

typedef enum XrLightEstimateStateANDROID {
    XR_LIGHT_ESTIMATE_STATE_VALID_ANDROID = 0,
    XR_LIGHT_ESTIMATE_STATE_INVALID_ANDROID = 1
} XrLightEstimateStateANDROID;

The enums have the following meanings:

Enum

Description

XR_LIGHT_ESTIMATE_STATE_VALID_ANDROID

The light estimate is valid

XR_LIGHT_ESTIMATE_STATE_INVALID_ANDROID

The light estimate is invalid

Example code for light estimation

The following example code demonstrates how to get all possible light estimation quantities from the runtime.

XrSession session;  // Created at app startup
XrSpace appSpace;   // Created previously.

XrLightEstimatorANDROID estimator;
XrLightEstimatorCreateInfoANDROID createInfo = {
    .type = XR_TYPE_LIGHT_ESTIMATOR_CREATE_INFO_ANDROID};
CHK_XR(xrCreateLightEstimatorANDROID(session, &createInfo, &estimator));

// Every frame
XrTime updateTime;  // Time used for the current frame's simulation update.

XrLightEstimateGetInfoANDROID info = {
    .type = XR_TYPE_LIGHT_ESTIMATE_GET_INFO_ANDROID,
    .space = appSpace,
    .time = updateTime,
};

XrDirectionalLightANDROID directionalLight = {
    .type = XR_TYPE_DIRECTIONAL_LIGHT_ANDROID,
};

XrSphericalHarmonicsANDROID totalSh = {
    .type = XR_TYPE_SPHERICAL_HARMONICS_ANDROID,
    .kind = XR_SPHERICAL_HARMONICS_KIND_TOTAL_ANDROID,
    .next = &directionalLight,
};

XrSphericalHarmonicsANDROID ambientSh = {
    .type = XR_TYPE_SPHERICAL_HARMONICS_ANDROID,
    .kind = XR_SPHERICAL_HARMONICS_KIND_AMBIENT_ANDROID,
    .next = &totalSh,
};

XrAmbientLightANDROID ambientLight = {
    .type = XR_TYPE_AMBIENT_LIGHT_ANDROID,
    .next = &ambientSh,
};


XrEnvironmentLightingCubemapANDROID lightingCubemap {
    .type = XR_TYPE_ENVIRONMENT_LIGHTING_CUBEMAP_ANDROID,
    .next = &ambientLight,
}

XrLightEstimateANDROID estimate = {
    .type = XR_TYPE_LIGHT_ESTIMATE_ANDROID,
    .next = &lightingCubemap,
};

XrResult result = xrGetLightEstimateANDROID(estimator, &info, &estimate);
if (result == XR_SUCCESS &&
    estimate.state == XR_LIGHT_ESTIMATE_STATE_VALID_ANDROID) {
  // use directionalLight, totalSh, ambientSh, ambientLight,
  // lightingCubemap, if each
  // struct has a valid state field
}

// When you want to disable light estimation
CHK_XR(xrDestroyLightEstimatorANDROID(estimator));

New Object Types

  • XrLightEstimator

XrObjectType enumeration is extended with:

  • XR_OBJECT_TYPE_LIGHT_ESTIMATOR_ANDROID

XrStructureType enumeration is extended with:

  • XR_TYPE_LIGHT_ESTIMATOR_CREATE_INFO_ANDROID
  • XR_TYPE_LIGHT_ESTIMATE_GET_INFO_ANDROID
  • XR_TYPE_LIGHT_ESTIMATE_ANDROID
  • XR_TYPE_DIRECTIONAL_LIGHT_ANDROID
  • XR_TYPE_SPHERICAL_HARMONICS_ANDROID
  • XR_TYPE_AMBIENT_LIGHT_ANDROID
  • XR_TYPE_ENVIRONMENT_LIGHTING_CUBEMAP_ANDROID

New Enums

New Structures

New Functions

Issues

Version History

  • Revision 2, 2025-01-17 (Salar Khan)
    • Added environment lighting cubemap support
  • Revision 1, 2024-09-16 (Cairn Overturf)
    • Initial extension description

OpenXR™ and the OpenXR logo are trademarks owned by The Khronos Group Inc. and are registered as a trademark in China, the European Union, Japan and the United Kingdom.