Aquileo | D400 On-Chip Calibration by dorodnic · Pull Request #4658 · realsenseai/librealsense

dorodnic · 2019-08-18T14:32:26Z

This PR introduces handling of the new "lightweight" calibration stack in the RealSense Viewer and DQT:

On-chip calibration depends on firmware 5.13.x that is going to be released soon. Formal white-paper from intelrealsense.com is also expected to be published on the topic

On-Chip Calibration is a set of integrated calibration capabilities designed to make stereo camera calibration process more accessible. It is not a full replacement of Dynamic Calibration Toolchain but can help keep the camera in good health when used regularly. In particular, there may be severe aggregate calibration defects that can be only rectified by the full Dynamic Calibration tool.

On-Chip Calibration does not require any special target, but we recommend pointing the camera at a flat wall when calibrating (ceiling, floor, indoor any white wall will do, outdoor the wall needs to be sufficiently textured)

At the moment, this capability is only provided via the Viewer, but in the future it is likely to be also accessible from the API.

…re the family of devices using it

Introducing auto-calibration process Adding metadata window for stream

ev-mp · 2019-08-20T05:44:24Z

+            if (update_state == RS2_FWU_STATE_INITIAL_PROMPT)
+                ImGui::Text("Firmware updates offer critical bug fixes and\nunlock new camera capabilities.");
            else
+                ImGui::Text("Firmware updates is underway...\nPlease do not disconnect the device");


ev-mp · 2019-08-20T05:51:01Z

+                    to_string() << std::fixed << std::setprecision(1) << timestamp, 
+                    "Frame Timestamp is normalized represetation of when the frame was taken.\n"
+                    "It's a property of every frame, so when exact creation time is not provided by the hardware, an approximation will be used.\n"
+                    "Clock Domain feilds helps interpret the meaning of timestamp\n"


field helps to interpret

ev-mp · 2019-08-20T05:51:37Z

+                    "Timestamp is measured in milliseconds, and is allowed to roll-over (reset to zero) in some situations" });
+                stream_details.push_back({ "Clock Domain",
+                    to_string() << rs2_timestamp_domain_to_string(timestamp_domain), 
+                    "Clock Domain describes the format of Timestamp feild. It can be one of the following:\n"


ev-mp · 2019-08-20T05:55:57Z

+                    "1. System Time - When no hardware timestamp is available, system time of arrival will be used.\n"
+                    "                 System time benefits from being comparable between device, but suffers from not being able to approximate latency.\n"
+                    "2. Hardware Clock - Hardware timestamp is attached to the frame by the device, and is consistent accross device sensors.\n"
+                    "                    Hardware timestamp encodes percisely when frame was captured, but cannot be compared across devices\n"


~~cannot be compared across devices~~ cannot be used for inter-device synchronization.

ev-mp · 2019-08-20T06:00:51Z

+                    "2. Hardware Clock - Hardware timestamp is attached to the frame by the device, and is consistent accross device sensors.\n"
+                    "                    Hardware timestamp encodes percisely when frame was captured, but cannot be compared across devices\n"
+                    "3. Global Time - Global time is provided when the device can both offer hardware timestamp and implements Global Timestamp Protocol.\n"
+                    "                 Global timestamps encode exact time of capture and at the same time are comparable accross devices." });


I think of adding of disclamer - that the global time is an algorithm-augmented mechanism that may introduce interval fluctuations and therefore is best suited for synchronizing Video ( 100fps<) frames.
In addition it requires an initial settle tile

ev-mp · 2019-08-20T06:17:52Z

+                    "                 Global timestamps encode exact time of capture and at the same time are comparable accross devices." });
+                stream_details.push_back({ "Frame Number",
+                    to_string() << frame_number, "Frame Number is a rolling ID assigned to frames.\n"
+                    "Most device do not guaranty consequitive frames to have conseuquitive frame numbers\n"


Most devices do not guarantee consecutive frames to have sequential frame numbers
devices... guarantee

ev-mp · 2019-08-20T06:18:45Z

-                ImGui::SetTooltip("%s","Stream Resolution, Format");
+                stream_details.push_back({ "Hardware FPS",
+                    to_string() << std::setprecision(2) << std::fixed << fps.get_fps(), 
+                    "Hardware FPS captures number of frames per second produced by the device.\n"


the number of frames...

ev-mp · 2019-08-20T06:18:59Z

-            ImGui::SameLine();
+                    stream_details.push_back({ "Display Size",
+                        to_string() << size.x << " x " << size.y, 
+                        "When Post-Processing is enabled, actual display size of the frame may differ from original capture size" });


the actual display...

ev-mp · 2019-08-20T06:21:18Z

+
+            std::map<rs2_frame_metadata_value, std::string> descriptions = {
+                { RS2_FRAME_METADATA_FRAME_COUNTER                        , "A sequential index managed per-stream. Integer value" },
+                { RS2_FRAME_METADATA_FRAME_TIMESTAMP                      , "Timestamp set by device clock when data readout and transmit commence. usec" },


Disclamer - the units are mostly usec, but there are exceptions (SR300 with 100nsec units)

ev-mp · 2019-08-20T06:23:06Z

+
+            if (model->supports_on_chip_calib())
+            {
+                // Make sure we don't spam calibration remainers too often:


remainders...

ev-mp · 2019-08-20T06:25:38Z

+                    msg, manager, false);
+
+                // Recommend calibration once a week per device
+                if (rawtime - last_time < 60)


The weekly remainder could create a false feeling that the device shall be re-calibrated on a weekly basis.
Setting it to a monthly periods would, imho, mitigate this

Why monthly?

ev-mp · 2019-08-20T06:31:49Z

+        if (ImGui::IsItemHovered())
+        {
+            win.link_hovered();
+            ImGui::SetTooltip("Open release notes. Internet connection is required");


the Release Notes...

ev-mp · 2019-08-20T06:34:27Z

+
+        // Set calibration speed: 0 - slowest, 4 - fastest
+        // On-Chip Calib manager will reduce speed if needed
+        void set_speed(int speed) { _speed = speed; }


I would also check the input for validity

It is not user accessible

ev-mp · 2019-08-20T06:35:12Z

+        bool use_new_calib = true;
+        std::string _error_message = "";
+    };
+}


EOL is missing

ev-mp · 2019-08-20T06:36:49Z

-                intrinsics.fy = rect_params[1] * height / resolutions_list[resolution].y;
-                intrinsics.ppx = rect_params[2] * width / resolutions_list[resolution].x;
-                intrinsics.ppy = rect_params[3] * height / resolutions_list[resolution].y;
+                if (width == 256 && height == 144)


Pls put a comment that this shall be used in auto-calibration only

dorodnic added 3 commits August 18, 2019 06:32

Changing rolling-shutter base class to non-monochrome to better captu…

ec3f768

…re the family of devices using it

Handle special intrinsics of 256x144 (auto-calib) resolution

00510df

Refactoring of firmware-update UI base logic into process-manager class

6f2268b

Introducing auto-calibration process Adding metadata window for stream

dorodnic added the calibration label Aug 18, 2019

dorodnic requested a review from ev-mp August 18, 2019 14:34

dorodnic mentioned this pull request Aug 19, 2019

Where is the Intel.Realsense.DynamicCalibrator source file #4659

Closed

dorodnic force-pushed the auto_calib_rebased branch from 50aa36d to 23b9bc2 Compare August 19, 2019 10:51

Code review changes

9cad634

dorodnic force-pushed the auto_calib_rebased branch from 23b9bc2 to 9cad634 Compare August 19, 2019 12:31

Basic TARE calibration

ff485da

ev-mp reviewed Aug 20, 2019

View reviewed changes

Comment thread common/fw-update-helper.cpp

ev-mp reviewed Aug 20, 2019

View reviewed changes

Comment thread common/notifications.cpp

ev-mp reviewed Aug 20, 2019

View reviewed changes

dorodnic added 2 commits August 20, 2019 00:58

More code review comments

f41123b

Ground truth in tare calibration is in uint32_t units (mm)

b6577da

ev-mp approved these changes Aug 20, 2019

View reviewed changes

ev-mp merged commit 033df88 into realsenseai:development Aug 20, 2019

Conversation

dorodnic commented Aug 18, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants