OpenCVStreamer

SampleStreamer and OpenCVStreamer DLL are destined to read dataset with MIV (MPEG Immersive Video) format stored on the disk. Such DLL were developped in the project to serve a debugging purpose and as example for the other Aquisition and refinment modules.

SamplerStreamer is destined to read dataset with yuv format without any conversions while OpenCVStreamer will use the OpenCV library to convert it to RGB format. The code for OpenCVStreamer and the one for SampleStreamer are very similar. Thus, the functionning of the DLL is also very similar.

Initialization

Before reading color and depth images, the DLL will first initialize some components needed.

First, the DLL will load the parameters, written in a JSON file, needed of RVS and itself with the fucntion initSettings. For example, the filenames for the RGBD images and their corresponding camera parameters are loaded. More information about the parameters that can be set can be found in the installation and build README of the DLL. Secondly, the Vulkan GPU is retrieved with its UUID and some Vulkan objects, like the buffer memory, are initialized through the functions initVk and initVkRessources.

Streaming Loop

Since RVSVulkan and the OpenCVStreamer DLL should operate independlently, a thread is created before the streaming loop is launched. This thread acts on his own and will upload new data when it wants. As it is called, the thread is an infinite loop and will only stop if the RVSVulkan thread asks or if the application finishes. The RVSVulkan (main) thread can acquired the uploaded data when needed. In this section, we will focus on the streaming thread. For more details about the RVSVulkan thread, please refer to the documentation of it.

During the initialization, the framerate of the loop is set. Thus, the streaming loop first checks if it should load and upload new data. If it is not the case, it will wait until the time to upload a new frame comes.

Load data

Depth images

To load depth images, the first thing that should to be checked is the bit depth. If the depth was encoded in 10 bits, the file would be read direclty from its memory space. If the bit depth is lower or greater, it will use the OpenCV API and the cv::Mat class as the container. Howerver, if the file extension is .yuv, the OpenCV API cannot read it, thus, the file should be read as a single channel array, then copied into the data memory of the cv::Mat object. Finally, the memory is analyzed to check if missing values are present and should be replaced by 0.

Note that, other encoding and compression formats, as PNG and TIFF, cannot contain more than one frame which does not allow working with dynamic dataset.

Color images

For color images, the process is similar. If the encoding format is YUV, then the data is directlty loaded from the memory space. If another encoding format is used, the image is loaded through the OpenCV API and is contained in a cv::Mat object.

Upload data

After loading the data, the data is uploaded in the Vulkan GPU buffer memory. Some technical differences are present if the depth or the color image is uploaded, but it will not be explained here. For more information about Vulkan memory buffers, please refer to the official Vulkan documentation.

The streaming loop can be resumed by the flowchart below.

flowchart LR

subgraph rvs["RVSVulkan thread"]
    someProc["..."]
    subgraph crstream["createStreamingObject"]
        subgraph opencv["opencvStreamer()"]
            initsett["initSettings()"]
            initvk["initVk()"]
            initvkres["initVkRessources()"]
        end
    end
    crstreamthr["createStreamThread"]
    subgraph looprvs["RVS Loop"]
        getdatastr["acquireStreamFrames()"]
        perfproc["computeProcess()"]
    end
end

someProc --> crstream --> crstreamthr --> looprvs
initsett-->initvk-->initvkres

getdatastr --> perfproc --> getdatastr

subgraph opencvthread["OpenCVStreamer thread"]
subgraph strloop["Streaming loop"]
waitstream["waitUntilFrameShouldBeUploaded()"]
loaddepth["loadDepth()"]
loadcolor["loadColor()"]
upload["uploadDataToMemoryBuffer()"]

memdata(("Memory space"))
opencvapi[["OpenCV API"]]
end
end

loaddepth -.->|IF bitdepth IS 10| memdata
loaddepth -.->|ELSE| opencvapi

loadcolor -.->|IF format IS yuv| memdata
loadcolor -.->|ELSE| opencvapi

waitstream-->loaddepth --> loadcolor --> upload --> waitstream

subgraph gpu["Vulkan GPU"]
membuf(("Memory buffers"))
end



upload-.-> membuf -.->getdatastr 

Figure 1: Flow chart of the OpenCVStreamer DLL