.. _about_real_time_tracking:

Real-time tracking
==================

Once you are receiving a stream of ``TrackingStateSet`` objects (see :ref:`data_access_methods`), 
you can access the ``UserState`` sub-structure which contains the user's tracking state.


.. tab-set-code::

   .. code-block:: c

      EW_BET_TRACKING_STATE_SET_HANDLE tss_handle; // Assume a valid handle
      const EW_BET_UserState *user_state = EW_BET_API_GetUserState(tss_handle);
      if (user_state->timestamp_in_seconds != EW_BET_NULL_DATA_TIMESTAMP) {
         // Use the data
      }

   .. code-block:: c++

      // Assume we hold a valid TrackingStateSet instance
      std::unique_ptr<eyeware::beam_eye_tracker::TrackingStateSet> last_received_tracking_state_set;
      const eyeware::beam_eye_tracker::UserState &user_state = last_received_tracking_state_set->user_state();
      if (user_state.timestamp_in_seconds != eyeware::beam_eye_tracker::NULL_DATA_TIMESTAMP) {
         // Use the data
      }
         
   .. code-block:: python
      
      from eyeware import beam_eye_tracker
      tracking_state_set: beam_eye_tracker.TrackingStateSet  # assume a valid instance
      user_state = tracking_state_set.user_state()
      if user_state.timestamp_in_seconds != eyeware.beam_eye_tracker.NULL_DATA_TIMESTAMP:
         # Use the data

   .. code-block:: javascript

      const trackingStateSet = api.getLastTrackingStateSet(); // assume a valid API instance
      const userState = trackingStateSet.user_state;
      if (userState?.valid) {
        // Use the data
      }

.. _about_user_state:

The ``UserState`` State Structure
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The ``UserState`` contains:

* :ref:`head_pose`  
* :ref:`unified_screen_gaze`
* :ref:`screen_of_attention_gaze`
* :ref:`viewport_gaze`

Each of these elements hold their own ``TrackingConfidence`` field, explained in the next section.


.. _tracking_confidence:

Tracking Confidence
~~~~~~~~~~~~~~~~~~~

The ``TrackingConfidence`` is a reliability measure for its respective tracking data package.

It is **not** a direct indicator of how `accurate` the tracking is, but it does indicate whether you should
discard some data. For example, it would be incorrect to assume that if the ``point_of_regard`` (see :ref:`unified_screen_gaze`) 
have a ``HIGH`` confidence, then it means that the eye tracking result is highly accurate. Instead
it only means that the conditions for the relevant tracking algorithms were deemed to be good
based on some internal heuristics (like measured illumination, eye-tracking on a screen that was
calibrated, head pose orientation or distance, etc.).

For an explaination of the values it can take, see :ref:`c_api_tracking_confidence`.

.. _head_pose:

Head Pose
~~~~~~~~~~

The head pose is provided as a rigid transform such that a 3D point in the head coordinate system :math:`p_{hcs}` is
transformed to the :ref:`world_coordinate_system` by:

.. math::
   p_{wcs} = R \cdot p_{hcs} + t,

where :math:`p_{hcs}` is a column vector defining the x,y,z coordinates in the head coordinate system, :math:`R` (referred to as ``rotation_from_hcs_to_wcs`` in the C API) is a 3x3 rotation matrix and :math:`t` (referred to as ``translation_from_hcs_to_wcs`` in the C API)
is a 3x1 vector translation, with units in **meters**.

The head coordinate system (HCS) is defined as follows:

* The origin is at the nose tip.
* The x-axis points from the nose to the left ear.
* The y-axis points from the nose to the top of the head.
* The z-axis points from the nose towards the front.

.. figure:: /media/head_transform_isometric_rt_axes_wcs.png
  :width: 400
  :alt: Lateral view of the head transform
  :align: center

  Here is a lateral view that shows the :math:`{R, t}` transform, as well as the xyz axes of the WCS and of the head frame



.. figure:: /media/head_transform_frontal_axes.png
  :width: 300
  :alt: Frontal view of the head frame
  :align: center

  Frontal view showing the head axes from the point of view of the person


.. figure:: /media/head_transform_top_rt_axes.png
  :width: 400
  :alt: Top view of the head transform
  :align: center
  
  Top view showing the transform, the WCS frame at the screen center and the head frame on the nose tip

The head pose also contains a ``track_session_uid`` field, which is kept unchanged as long as the
person is being tracked without interruption. However, if the person goes out of frame or turns around
such that they can no longer be tracked for a moment, then this number is incremented once the
person is detected again.

.. _unified_screen_gaze:

Unified Screen Gaze
~~~~~~~~~~~~~~~~~~~

``UserState.unified_screen_gaze`` describes where the user is looking on the logical desktop,
expressed in the :ref:`unified_screen_coordinate_system`.

* ``point_of_regard`` is constrained to screen areas.
* ``unbounded_point_of_regard`` can extend beyond the desktop, for example to animate an overlay
  beyond its edges.
* ``confidence`` describes the tracking conditions; it does not measure gaze accuracy. See
  :ref:`tracking_confidence`. Ignore both points when it is ``LOST_TRACKING``.

Gaze can move between screens, but transitions may still pause at an edge or jump. Physical
gaps assumed between screens can cause an edge pause, while differences in screen size or DPI
can cause a position change when moving between their coordinate mappings. Use
:ref:`screen_of_attention_gaze` when you need Beam's estimate of which screen is being looked at.

.. figure:: /media/UnifiedScreenGaze.svg
   :width: 600
   :alt: Unified Screen Gaze
   :align: center

   Gaze positions share one desktop coordinate system.

.. figure:: /media/UnboundedUnifiedScreenGaze.svg
   :width: 600
   :alt: Unbounded Unified Screen Gaze
   :align: center

   The unbounded point can extend beyond screen boundaries.

.. note::

   Beam Eye Tracker 2.6.12 provides more continuous gaze across screen boundaries, accounting
   for differences in DPI and physical sizes. Existing SDK clients benefit without code changes.
   Older Beam versions keep bounded gaze clipped to the screen Beam estimates the user is
   looking at, and can also clip unbounded gaze during transitions. Updating the SDK alone
   does not change that behavior.

.. _screen_of_attention_gaze:

Screen of Attention Gaze
^^^^^^^^^^^^^^^^^^^^^^^^

``UserState.screen_of_attention_gaze`` describes Beam's best estimate of which screen the user
is looking at:

* ``screen_geometry`` gives that screen's inclusive ``top_left`` and ``bottom_right`` corners.
* ``point_of_regard`` is the unclipped gaze position on that screen's plane. When the user looks
  near the screen's edges, the reported gaze position can extend beyond its logical boundaries.
  It retains that screen's DPI even when it enters another screen's logical area, keeping motion
  consistent relative to the screen of attention, for example for overlays.
* ``confidence`` indicates reliability of the screen estimate, separately from unified gaze
  confidence. Ignore geometry and position when it is ``LOST_TRACKING``.

Geometry and gaze use the :ref:`unified_screen_coordinate_system`.

Beam uses recent gaze history to keep its estimate of the screen being looked at stable.
Near a screen boundary, it may continue reporting the same screen even as gaze enters another
screen, until there is enough evidence that the user's attention has shifted.

.. note::

   Available in Beam SDK 2.3.0 with Beam Eye Tracker 2.6.12 or newer. With older supported
   Beam versions, this field reports ``LOST_TRACKING``.


.. _viewport_gaze:

Viewport Gaze
~~~~~~~~~~~~~

The ``ViewportGaze`` contains the eye gaze coordinates in relation to the :ref:`viewport`,
referred to as ``normalized_point_of_regard``.
The coordinates are normalized such that, if the gaze is inside the viewport, then the values are in the range :math:`[0, 1]`.
The coordinates are not clipped, so they can be negative or exceed :math:`1.0`, if the gaze is outside the viewport.

.. warning:: If the viewport geometry is not updated into the ``API``, then the ``ViewportGaze`` will be unreliable.



