Skip to content

SampleStreamer

SampleStreamer and OpenCVStreamer DLL are destined to read dataset with MIV (MPEG Immersive Video) format stored on the disk. Such DLL were developped in the project to serve a debugging purpose and as example for the other Aquisition and refinment modules.

SamplerStreamer is destined to read dataset with YUV format without any conversions while OpenCVStreamer will use the OpenCV library to convert it to RGB format. The code for OpenCVStreamer and the one for SampleStreamer are very similar. Thus, the functionning of the DLL is also very similar.

Initialization

Before reading color and depth images, the DLL will first initialize some components needed.

First, the DLL will load the parameters, written in a JSON file, needed of RVS and itself with the fucntion initSettings. For example, the filenames for the RGBD images and their corresponding camera parameters are loaded. More information about the parameters that can be set can be found in the installation and build README of the DLL. Secondly, the Vulkan GPU is retrieved with its UUID and some Vulkan objects, like the buffer memory, are initialized through the functions initVk and initVkRessources.

Streaming Loop

Since RVSVulkan and the SampleStreamer DLL should operate independlently, a thread is created before the streaming loop is launched. This thread acts on his own and will upload new data when it wants. As it is called, the thread is an infinite loop and will only stop if the RVSVulkan thread asks or if the application finishes. The RVSVulkan (main) thread can acquired the uploaded data when needed. In this section, we will focus on the streaming thread. For more details about the RVSVulkan thread, please refer to the documentation of it.

During the initialization, the framerate of the loop is set. Thus, the streaming loop first checks if it should load and upload new data. If it is not the case, it will wait until the time to upload a new frame comes.

Load data

Differently from the OpenCVStreamer DLL, the SampleStreamer DLL loads all the data (multiple bit depth images) in the same way. The file is just read direclty from its memory space thanks to the information about the file.

Upload data

After loading the data, the data is uploaded in the Vulkan GPU buffer memory. Some technical differences are present if the depth or the color image is uploaded, but it will not be explained here. For more information about Vulkan memory buffers, please refer to the official Vulkan documentation.

The streaming loop can be resumed by the flowchart below. 

flowchart LR

subgraph rvs["RVSVulkan thread"]
    someProc["..."]
    subgraph crstream["createStreamingObject"]
        subgraph opencv["SampleStreamer()"]
            initsett["initSettings()"]
            initvk["initVk()"]
            initvkres["initVkRessources()"]
        end
    end
    crstreamthr["createStreamThread"]
    subgraph looprvs["RVS Loop"]
        getdatastr["acquireStreamFrames()"]
        perfproc["computeProcess()"]
    end
end

someProc --> crstream --> crstreamthr --> looprvs
initsett-->initvk-->initvkres

getdatastr --> perfproc --> getdatastr

subgraph opencvthread["SampleStreamer thread"]
subgraph strloop["Streaming loop"]
waitstream["waitUntilFrameShouldBeUploaded()"]
loaddepth["loadDepth()"]
loadcolor["loadColor()"]
upload["uploadDataToMemoryBuffer()"]

memdata(("Memory space"))
end
end

loaddepth -.->memdata

loadcolor -.->memdata

waitstream-->loaddepth --> loadcolor --> upload --> waitstream

subgraph gpu["Vulkan GPU"]
membuf(("Memory buffers"))
end



upload-.-> membuf -.->getdatastr

Figure 1: Flow chart of the SampleStreamer DLL