Licence

Please check the LICENSE file.

Acknowledgments

This work was supported by the HoviTron project that received funding from the European Union's Horizon 2020 research and innovation program under grant agreement Number 951989.

Authors

Universite Libre de Bruxelles (ULB):
- Gauthier LAFRUIT
- Laurie VAN BOGAERT
- Daniele BONATO
- Sarah FACHADA
- Yupeng XIE
- Mehrdad TERATANI
- Armand LOSFELD
Universidad Politécnica de Madrid (UPM):
- Jaime SANCHO
- Eduardo JUAREZ
- Miguel CHAVARRIAS
- César SANZ
Deutsches Zentrum für Luft- und Raumfahrt (DLR):
- Michael PANZIRSCH
- Martin LINGENAUBER
- KLAUS STROBL
CREAL:
- Alex KVASOV
- Gregoire HIRT

Introduction

The following repository contains the public part of the code for the Hovitron's Video Pipeline.

The module for the HoviTron Video Pipeline can be divided into three categories: Acquisition and Refinement Modules, View Synthesis, and Headset runtime.

Acquisition and Refinement Modules have the responsibility to acquire data (usually from a camera setup), apply post-processing if necessary, and make them ready to be consumed by the pipeline. These modules are compiled in the form of Dynamic Link Libraries (*.dll*). The Hovitron project pocesses the following Acquisition and Refinement Module:

Kinect Refinment Tool Module (KiRT) DLL: To be used with a camera setup composed of Azure Kinect cameras. More information can be found here.
Raytrix Acquisition DLL: To be used with a camera setup composed of Raytrix camera. More information can be found here.
SampleStreamer DLL (for testing purposes): To be used to read dataset on the disk that use MPEG Immersive Dataset (MIV) format and yuv image in 8 or 16 bit (no 10 bit). More information can be found here.
OpenCVStreamer DLL (for testing purposes): slower version of SamplerStreamer DLL that supports yuv with 10 bit depth. More information can be found here.

For the View Synthesis Module , the code is compiled as the executable. Its purpose is to load the DLL of an Acquisition and Refinement Module, communicate with the headset runtime through OpenXR standard, and create virtual views as requested by the headset. For development purposes, it also supports a mode where no headset is required and the result of the view synthesis is displayed on the screen. The Hovitron Video Pipeline possesses one View Synthesis Module:

RVSVulkan: More information can be found here.

The Headset runtime is not part of the present repository. Such OpenXR runtime for headsets must be installed on their own. Refer to the instructions of your headset's manufacturer to install it. Right now the Hovitron Video Pipeline has been validated on:

Creal Zorya headset
Meta Quest 2 headset

The code present in this repository has been written in C++ and C++ Cuda. It is destined to be used on Windows platform (Windows 10/11). The Hovitron Video Pipeline is further described in Hovitron's deliverables (https://www.hovitron.eu/public-resources) and in the proceedings of the 1rst Workshop on Interactive eXtended Reality (IXR) (https://dl.acm.org/doi/abs/10.1145/3552483.3556461).

Build the code (Windows)

First, clone this repository to obtain the source code of the project.

In the case you want to build the project with a headset support, the code will need the OpenXR loader as dependency. If so, the project must be cloned using a specific command to retrieve this OpenXR dependency.

Clone this repository with :

git clone --recursive https://gitlab.hovitron.ulb.be/hvt/openxr-ravis-stalf.git

If you forgot to use the --recursive flag please use the following command to retrieve it.

git submodule update --init --recursive

Then to build the code, install the dependencies defined in the README.md of the different modules you wish to install:

RVSVulkan: Build instructions for RVSVulkan
KiRT DLL: Build instructions for Kirt
Raytrix Streaming DLL: Build instructions for Raytrix Dll
SampleStreamer: Build instructions for SampleStreamer Dll
OpenCVStreamer: Build instructions for OpenCVStreamer Dll

:warning: Warning :warning: By default the project will only build the module RVSVulkan, SampleStreamer, and AcqKiRT. The module that must be compiled can be specified using variables present in the top-most CmakeLists.txt file.

In the following table are listed the variables used to control the compilation of the different modules.

Variable	Default Value	Description
`"HVT_BUILD_ACQKIRT"`	ON	Compile KiRT DLL
`"HVT_BUILD_RAYTRIX_STREAMER"`	OFF	Compile Raytrix DLL
`"HVT_SAMPLE_STREAMER"`	ON	Compile SampleStreamer DLL (testing purposes)
`"HVT_OPENCV_STREAMER"`	OFF	Compile OpenCVStreamer DLL (testing purposes)

The recommanded IDE and compilator for this project is Microsoft Visual studio. Visual Studio supports CMake, therefore it allows opening the project in it, and use the CMakeSettings.json to specify the module to be compilated with and the correct paths of the necessary libraries.

If you prefer translate the project into a MVSC project, you can use the CMake GUI.

Usage

A typical usage of the software is the following:

RVSVulkan.exe ${PATH_TO_DLL} [--glfw|--openxr|...]

Where the variable PATH_TO_DLL is the relative or absolute path to one of the DLLs.

Additional arguments can also be given when launching the View Synthesis Module. Please refer to the RVSVulkan README.

Moreover, configuration files for the DLLs often need to be specified using environment variable(s). Refer to the documentation of the modules for further instructions.

In MVSC with a CMake project, you can specify the arguments and environment variables easely with a launch.vs.json file.

Example of the content of a launch.vs.json file:

{
  "version": "0.2.1",
  "defaults": {},
  "configurations": [
        {
        "type": "default",
        "project": "CMakeLists.txt",
        "projectTarget": "",
        "name": "glfw_ULBUnicorn",
        "args": [
            "../SampleStreamer/SampleStreamer.dll",
            "--glfw"
        ],
        "env": {
            "HVT_JSON_PATH": "C:\\Users\\Administrateur\\Documents\\savData\\dataset\\ULBUnicornV4.json"
        }
        }
    ]
}