Skip to content

The Hovitron Video Pipeline

Licence

Please check the LICENSE file.

Acknowledgments

This work was supported by the HoviTron project that received funding from the European Union's Horizon 2020 research and innovation program under grant agreement Number 951989.

Authors

  • Universite Libre de Bruxelles (ULB):

    • Gauthier LAFRUIT
    • Laurie VAN BOGAERT
    • Daniele BONATTO
    • Sarah FACHADA
    • Yupeng XIE
    • Mehrdad TERATANI
    • Armand LOSFELD

  • Universidad Politécnica de Madrid (UPM):

    • Jaime SANCHO
    • Eduardo JUAREZ
    • Miguel CHAVARRIAS
    • César SANZ

  • Deutsches Zentrum für Luft- und Raumfahrt (DLR):

    • Michael PANZIRSCH
    • Martin LINGENAUBER
    • Klaus STROBL

  • CREAL:

    • Alex KVASOV
    • Grégoire HIRT

Introduction

The following repository contains the public part of the code for the Hovitron's Video Pipeline.

The module for the HoviTron Video Pipeline can be divided into three categories: Acquisition and Refinement Modules, View Synthesis, and Headset runtime.

Acquisition and Refinement Modules have the responsibility to acquire data (usually from a camera setup), apply post-processing if necessary, and make them ready to be consumed by the pipeline. These modules are compiled in the form of Dynamic Link Libraries (.dll). The Hovitron project pocesses the following Acquisition and Refinement Module:

For the View Synthesis Module , the code is compiled as the executable. Its purpose is to load the DLL of an Acquisition and Refinement Module, communicate with the headset runtime throughtheOpenXR standard, and create virtual views as requested by the headset. For development purposes, it also supports a mode where no headset is required and the result of the view synthesis is displayed on the screen. The Hovitron Video Pipeline possesses one View Synthesis Module:

If depth maps are needed for the SampleStreamer or OpenCVStreamer DLL, those can be generated using the RDE tool. More information can be found here.

The Headset runtime is not part of the present repository. Such OpenXR runtime for headsets must be installed on their own. Refer to the instructions of your headset's manufacturer to install it. Right now the Hovitron Video Pipeline has been validated on:

  • Creal Zorya headset (see (3) in the figure of the homepage)
  • Meta Quest 2 headset

The code present in this repository has been written in C++ and C++ Cuda. It is destined to be used on Windows platform (Windows 10/11). The Hovitron Video Pipeline is further described in Hovitron's deliverables (https://www.hovitron.eu/public-resources) and in the proceedings of the 1rst Workshop on Interactive eXtended Reality (IXR) (https://dl.acm.org/doi/abs/10.1145/3552483.3556461).

graph LR

  subgraph "Sources of the data"

  Raytrix["Raytrix camera"]
  Kinect["Kinect"]
  Disk["Dataset"]

  end

  subgraph "Acquisition and refinment (Dll)"

  RaytrixDll["Raytrix Dll"]
  KinectDLL["KiRT"]
  sampleStreamer["Sample Streamer"]
  opencvStreamer["OpenCV Streamer"]

  end
  ray["Raytrix API"]
  k4a[Kinect Azure API]
   Raytrix ---ray --> RaytrixDll
   Kinect --- k4a--> KinectDLL

   miv["MIV standard"]
   Disk --- miv-->sampleStreamer
   miv--> opencvStreamer

  subgraph "View Synthesis"
  RVSVulkan[RVS Vulkan]
  end

hoviC["Hovitron C API"]
RaytrixDll --- hoviC
KinectDLL --- hoviC 
sampleStreamer --- hoviC
opencvStreamer --- hoviC 
hoviC -->RVSVulkan


subgraph "HMD OpenXR runtime"
metaRuntime["Meta's OpenXR runtime"]
crealRuntime["Creal's OpenXR runtime"]
end

xr["OpenXR"]
RVSVulkan --- xr
xr --> metaRuntime
xr --> crealRuntime

subgraph "Target hardware"
crealZorya["Creal Zorya HMD"]
metaQuest["MetaQuest 2"]

screen["computer screen"]
end 

RVSVulkan --> screen
crealRuntime-->crealZorya
metaRuntime-->metaQuest

Build the code (Windows)

First, clone this repository to obtain the source code of the project.

In the case you want to build the project with a headset support, the code will need the OpenXR loader as dependency. If so, the project must be cloned using a specific command to retrieve this OpenXR dependency.

Clone this repository with : 

git clone --recursive https://gitlab.hovitron.ulb.be/hvt/openxr-ravis-stalf.git

If you forgot to use the --recursive flag please use the following command to retrieve it. 

git submodule update --init --recursive

Then to build the code, install the dependencies defined in the README.md of the different modules you wish to install:

⚠ Warning ⚠ By default the project will only build the module RVSVulkan, SampleStreamer, and AcqKiRT. The module that must be compiled can be specified using variables present in the top-most CmakeLists.txt file.

In the following table are listed the variables used to control the compilation of the different modules.

Variable Default Value Description
"HVT_BUILD_ACQKIRT" ON Compile KiRT DLL
"HVT_BUILD_RAYTRIX_STREAMER" OFF Compile Raytrix DLL
"HVT_SAMPLE_STREAMER" ON Compile SampleStreamer DLL (testing purposes)
"HVT_OPENCV_STREAMER" OFF Compile OpenCVStreamer DLL (testing purposes)

The recommanded IDE and compilator for this project is Microsoft Visual studio. Visual Studio supports CMake, therefore it allows opening the project in it, and use the CMakeSettings.json to specify the module to be compiled with and the correct paths of the necessary libraries.

If you prefer translate the project into a MVSC project, you can use the CMake GUI.

Usage

A typical usage of the software is the following: 

RVSVulkan.exe ${PATH_TO_DLL} [--glfw|--openxr|...]
Where the variable PATH_TO_DLL is the relative or absolute path to one of the DLLs.

Additional arguments can also be given when launching the View Synthesis Module. Please refer to the RVSVulkan README.

Moreover, configuration files for the DLLs often need to be specified using environment variable(s). Refer to the documentation of the modules for further instructions.

In MVSC with a CMake project, you can specify the arguments and environment variables easely with a launch.vs.json file.

Example of the content of a launch.vs.json file: 

{
  "version": "0.2.1",
  "defaults": {},
  "configurations": [
        {
        "type": "default",
        "project": "CMakeLists.txt",
        "projectTarget": "",
        "name": "glfw_ULBUnicorn",
        "args": [
            "../SampleStreamer/SampleStreamer.dll",
            "--glfw"
        ],
        "env": {
            "HVT_JSON_PATH": "C:\\Users\\Administrateur\\Documents\\savData\\dataset\\ULBUnicornV4.json"
        }
        }
    ]
}