Kinect Acquisition and Refinement Tool DLL

This README explains firstly what it can do, how to build the DLL, and how to launch the Kinect Streaming DLL that was developped during the HoviTron project.

Note that the Kinect Streaming DLL has been compiled and tested with a Windows Operating System.

Utility

Kinect devices possess a color sensor and a depth sensor from the time-of-flight (TOF) family. A major issue with these devices is that depth and color images are misaligned while the view synthesis module requires it to be aligned. Please note that this problem does not occurs with Raytrix cameras since the RGBD data from the light field cameras is inherently inlined. While reprojecting the depth map to align it with the color, it creates patches without data in occluded areas. The purpose of the Kinect Refinement Tool (KiRT) is to align the color and depth image and fill the holes caused by these disocclusions. Information from other adjacent Kinect cameras help fill the occluded area. Also, due to its nature of TOF sensors the depth map will also exhibit noise that the KiRT module must filter, especially at the borders and on reflective surfaces. This data acquisition module was used during the user study. In addition, to obtain good-quality results, the Kinect Azure cameras need to be calibrated intrinsically and extrinsically. More info about calibration is given in D1.2 and D5.3 deliverables (https://www.hovitron.eu/public-resources) as well as in IXR paper in point 3.2 (https://dl.acm.org/doi/abs/10.1145/3552483.3556461).

Installation

List of software/librairies needed:

1. CUDA
2. Azure Kinect SDK (\(\geq\) 1.4.1)
   • Link: https://learn.microsoft.com/en-us/azure/kinect-dk/sensor-sdk-download
   • Documentation: https://learn.microsoft.com/en-us/azure/kinect-dk/
3. Windows OS (10 or 11)
4. Visual Studio (2019/2022) with the C++ desktop development
5. OpenCV

If you have any problem during the installation of one of SDKs/librairies, please refer to the documentation of the developpers .

Build

As a Dynamic Library (DLL)

Windows OS

1. Open the folder in Visual Studio
2. Open the CMakeSettings.json of the project
3. Specify the fields:
   1. OpenCV_DIR with the absolute path to the OpenCV build directory
   2. CUDA_TOOLKIT_ROOT_DIR with the absolute path to the CUDA API directory
4. In the AcqKirt directory, create a directory called libs with two subfolders called Release and Debug
5. Copy the DLLs you need into these 2 folders:
   • Debug:
     • OpenCV: opencv_world{OPENCV_VERSION}d.dll and opencv_videoio_ffmpeg_64.dll
     • KinectSDK: k4a.dll, k4arecord.dll, and depthengine_2_0.dll
     • CUDA: all .dll in C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v{CUDA_VERSION}\bin
   • Release:
     • OpenCV: opencv_world{OPENCV_VERSION}.dll and opencv_videoio_ffmpeg_64.dll
     • KinectSDK: k4a.dll, k4arecord.dll, and depthengine_2_0.dll
     • CUDA: all .dll in C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v{CUDA_VERSION}\bin
6. Finally, press simultaneously on CTLR+S and click on Build
   • It is possible that Visual Studio saves only one field when saving

Launch the dll

In this section, two ways to launch the AcqKiRT DLL will be explained.

Through a terminal

> set HVT_CALIBRATION_PATH=${PATH_TO_A_JSON}
> RVSVulkan.exe ../AcqKiRT/AcqKiRT.dll [--glfw|--openxr|...]

Where the variable ${PATH_TO_A_JSON} is the absolute path to a JSON file where the parameters for the AcqKiRT DLL are specified.

Note that RVSVulkan.exe can be launched with some parameters, see RVSVulkan README.

With Visual Studio

1. Open the project in Visual Studio
2. Compile it
3. Right-click on the HoviTron CMakeLists.txt then click on Add a debug configuration
4. After the field name, add the field args, and as arguments add
   1. "../AcqKiRT/AcqKiRT.dll"
   2. Other parameters for RVSVulkan, see RVSVulkan README
5. Add a field env, and specify the environment variable "HVT_CALIBRATION_PATH" with the absolute path to the JSON parameter file
6. Next to the green arrow (in Visual Studio), choose the debug configuration that you created, and click on it to launch it.

To make it clearer, here is an example of a launch.vs.json file where one debug configuration is written. The configuration name is AcqKiRTDLL. The configuratione asks Visual Studio to launch RVSVulkan.exe with the AcqKiRT DLL with a GLFW window.

{
  "version": "0.2.1",
  "defaults": {},
  "configurations": [
    {
      "type": "default",
      "project": "CMakeLists.txt",
      "projectTarget": "",
      "name": "AcqKiRTDLL",
      "args": [
        "--glfw",
        "../AcqKiRT/AcqKiRT.dll"
      ],
      "env": {
        "HVT_CALIBRATION_PATH": "C:\\Users\\Admin\\Desktop\\Hovitron\\HoviTronProject\\RVSVulkan\\AcqKiRT\\calibration_files\\calibration.json"
      }
    }
  ]
}

The paramaters of the DLL will be set according to what is inside the JSON file "C:\Users\Admin\Desktop\Hovitron\HoviTronProject\RVSVulkan\AcqKiRT\calibration_files\calibration.json" given as an environment variable.

Common issue

• Error 126 In case the software returns error 126 when the AcqKiRT DLL is loaded, you can search for the missing DLL using the process monitor (https://docs.microsoft.com/en-us/sysinternals/downloads/procmon).

JSON parameter file

As you could understand, the AcqKirt DLL needs a JSON parameter file where all the needed parameters will be specified. An example is available at JSON example.

In this section, all the parameters are explained so you can write your own JSON parameter file.

Variables	Type	Description
cameraConfiguration	set of variables	Variables are numKinectAzures and color_control.
numKinectAzures	int	Number of cameras used
color_control	set of variables	Variables are exposure, brightness, whitebalance, contrast, saturation, sharpness, backlight_compensation, gain, powerline_frequency, and offset_delay_for_sub
exposure	int	Exposure of the cameras
brightness	int	Brightness of the cameras
whitebalance	int	White balance of the cameras
contrast	int	Constrast of the cameras
saturation	int	Saturation of the cameras
sharpness	int	Sharpness of the cameras
backlight_compensation	int	Backlight compensation of the cameras
gain	int	Gain of the cameras
powerline_frequency	int	Powerline frequency of the cameras
offset_delay_for_sub	int	Offset delay for sub cameras
camera_intrinsics	set of variables	Variables are c0, c1, c2, c3, c4, ... Where ci refers to the camera i
ci	set of variables	Variables are serial_number and calibration_file.
serial_number	String	List of chars specifying the serial number
calibration_file	String	Path to the calibration file of the camera
camera_extrinsics	Set of variables	Variables are p0, p1, p2, p3, ... pi refers camera i
pi	string	Path to the extrinsic calibration file of the camera
process	set of variables	Variables are multiGPU, async loader, and async send.
multiGPU	boolean	To use multiple GPUs during the execution.
async loader	boolean	To make the load of new data asynchronously to the refinement process.
async send	boolean	To make the send of new data asynchronously to the refinement process.