3D Navigation Using Camera Intrinsics and Extrinsics

Input Data

Camera intrinsics and extrinsics For this project, you need to achieve 3D navigation on a static 2D image by using camera intrinsics and extrinsics. Given a pair of color and depth image captured by Kinect, your job is to reconstruct the 3D geometry with texture information using the inverse camera model. Then project the 3D scene into any arbitrary virtual camera and re-render it on a 2D image.

You are provided two image files:

(1) “color.jpg” a regular RGB image

(2) “depth.dat” a depth image that has the same size as the color image in terms of width and height. But the difference is each pixel of the depth image has the depth value instead of color value. The depth represents how far the corresponding scene point away from the camera center (measured in cm). Both the color and depth image are captured by a Kinect sensor at the same time. So their pixels are aligned already, which means for a pixel at (x, y) in the color image, you know the corresponding depth value in the depth image at (x, y). On thing, you need to pay attention is the data type for each image. For the color image, each pixel value is stored as “uchar”; however, for the depth image, each pixel value is stored as “short int”. So be aware to use the corresponding pointer to access the individual pixels accordingly.

What you should achieve?

How to successfully load the color and depth images into memory.

How to successfully reconstruct the 3D geometry by applying camera inverse model.

How to successfully use camera extrinsics to change the virtual view.

How to successfully render the virtual image by using the camera intrinsics. Well developed code with good keyboard interactivity.

Press ‘1’ key to rotate the image about x-axis clockwise, e.g. increasing 5 degrees. - Press ‘2’ key to rotate the image about x-axis anti-clockwise, e.g. decreasing 5 degrees. - Press ‘a’ key to translate the camera along x-axis by 10 units. - Press ‘d’ key to translate the camera along x-axis by -10 units. - Press ‘w’ key to translate the camera along x-axis by 10 units. - Press ‘s’ key to translate the camera along x-axis by -10 units. - Press ‘z’ key to increase the focal length, e.g. +30 pixels. - Press ‘z’ key to decrease the focal length, e.g. -30 pixels. - Press ‘r’ key to restart the virtual camera extrinsics. - Press ‘o’ key to output the virtual image to a local file.

Write up a “.pdf” document with several virtual images rendered from different angles or perspectives and clearly state what you have achieved or what parts you fail to accomplish.