Calibration, Alignment & Mesh Deform

Calibration is the automatic process of calculating camera and projector poses and settings using the blobs detected during Capture. Calibration also results in a consistent 3D point cloud formed from all blobs.

Alignment is a user-guided process to align the calibrated data with the coordinate system in Designer. OmniCal also includes the ReShape feature to adjust 3D model vertices based on the user alignment.

Mesh Deform is an automatic process that enhances the existing 3D model by inserting new vertices based on the aligned point cloud, and adjusting existing vertices accordingly.

OmniCal Stage Plan

In the Stage editor, under Projectors, click OmniCal Calibration to open the Calibrator widget. This widget controls the main OmniCal behaviour.

Once you have created a Capture Plan, you can click on it to open the OmnicalStagePlan widget.

In the OmnicalStagePlan we recommend working through the sections from top to bottom: Capture, Calibration, Alignment, and Mesh Deform (if required).

Calibration

The calibration process triangulates the position of the blobs detected in a capture as well as the relative positions of cameras & projectors and their lens intrinsics. From this point, you no longer need access to the physical stage to continue the calibration process.

Click Setup Calibration… and the widget OmniCal Calibration Setup will appear.
In most cases the default settings suffice. If calibration results are unsatisfactory despite good capture quality, adjust one setting at a time to clearly assess its impact.

Note

Most calibration settings include contextual help within Designer.

Camera Calibration settings

In the Camera Calibration separator we can set the Calibration Model to one of three modes:

• Auto: Attempt to detect appropriate model. Only manually select another mode if ‘Auto’ fails.
• Epipolar: Used with scenes “with depth”, i.e. where blobs are not just projected on a flat surface, but are spread across 3 dimensions.
• Homography: Used with scenes where blobs are mostly projected on a flat surface, or the depth provided is very shallow (e.g. < 1cm).

Bundle Adjustment settings

In the Bundle Adjustment separator we can set Bundle Adjustment target to one of five modes:

• Auto
• Cameras
• Cameras and Projectors
• Cameras then Projectors

Iterations(combined) of Bundle Adjustment to run for cameras and combined cameras & projectors modes. High value will be slow for large numbers of cameras and projectors.

• Cameras then each Projector

Iterations(combined) of Bundle Adjustment to run for Cameras then each Projector modes. Fast so can be set much higher than combined iterations.

Point Cloud Reconstruction settings

A setting under Point Cloud Reconstruction that is often useful is Camera graph optimisation. This should be enabled for camera setups where all cameras overlap with each other, where they form a closed loop, i.e. the first and last camera overlap with each other. Enabling the camera graph optimisation prevents gaps or discontinuities where the point cloud boundaries meet. This will give more consistent reconstruction and calibration results for typical setups for 360-degree projections inside of or on domes, cylinders, spheres or the walls of a room.

Camera graph BA iterations controls a separate Bundle Adjustment pass for the camera graph optimisation. The default is 500, but it is rare to need more than 50 iterations. This step is much faster than later Bundle Adjustment steps that involve cameras and projectors.

Projector Calibration settings

Projector Calibration Algorithm of Projector Calibration has three modes:

• DLT: Direct Linear Transformation
• Zhang
• Auto: will try all available algorithms and select the best result.

Perform Calibration

Once you have finished adjusting the Calibration Settings, click Calibrate in the Plan editor and confirm with clicking “Yes” to start the calibration process. Depending on the number of cameras and projectors, this can take from a few seconds to tens of minutes.

Calibration and Point Cloud visualisation

Once calibration is complete, you can view the point cloud and pixel errors for each projector. To do so, go to the OmniCal Calibration Calibrator widget. In the Visualiser, set the Visualisation Source to Plan and set the Visualisation Mode to All or Only Point Clouds. Set the Point Cloud Visualisation to ProjectorReprojError to see reprojection errors for each individual blob. Observations will show for each Blob by how many cameras it was seen. These and other visualisation modes may help with analysing issues such as interference from other light sources, or movement during a capture.

The generated point cloud will appear in the stage visualiser at this point but will not be aligned to the stage, unless you previously performed an alignment.

Calibration Report

Click Open Report to see a summary report of the calibration. Error values below 1 pixel are considered good, and above 1px usually point to something going wrong in the process (physical setup, scene, configuration, …). The report values are displayed using a colour between green and red to better highlight problematic camera or projector errors. If everything is green, then the calibration was a success.

Re-running calibrations

Re-run a calibration by modifying a setting and clicking Calibrate again. Note that issues introduced during image capture (lighting changes, obstructions, etc.) will require re-running Capture.
Re-running the calibration with different settings is useful:

• If the calibration quality was decent, but not yet good enough. Tweaking camera or projector calibration settings can help in some cases to get better results.
• If there were incorrect calibration settings (e.g. camera calibration was set to “Homography”, but the blobs land on non-planar surfaces).
• Simply to experiment with settings, e.g. when trouble-shooting issues.

Limitations of OmniCal

OmniCal can accurately calibrate projectors, including their particular lens distortion coefficients. Any rectilinear lens is supported, including ultra-wide-angle / ultra-short-throw lenses with throw ratios around 0.35, even if they use built-in mirrors.

However, OmniCal currently does not support fish-eye lenses.

Another minor limitation is that OmniCal cannot correct chromatic aberration in projector lenses. This is a less common phenomenon in projection (compared to photography), but it is noticeable at the corners of some ultra-short-throw lenses.

Alignment

Alignment is the process of registering or aligning the coordinate system of the captured point cloud with that of Designer.

Perform Alignment

1. In the plan editor, roughly edit the point cloud Alignment via Position, Rotation and Scale. This does not need to be exact at all, just close enough in terms of scale and rotation that further steps are made easier.

2. Once the point cloud is roughly aligned (i.e. somewhat close to your projected surfaces in the visualiser), click QuickAlign… to open the tool for detailed alignment. In QuickAlign you should see a reprojected wireframe of the first projection surface. The top two views relate to cameras which can be chosen from the view tab. The bottom two views relate to the cursor location in the camera view. This is essentially a zoomed in view, for better view finding.

3. There are some automatic options to automatically align the point cloud to the stage. Click QuickAlign, and choose Alignment Estimate which is automatic alignment estimation based on point cloud and all projection surfaces.

   Align to plan cameras which is the automatic alignment based on positions of plan cameras.

   Align to plan which aligns to plan cameras and projectors.

   Align to point cloud is useful for alignment using 2D camera reference points. Use this option when you want to apply the current alignment data to the stage.

4. If the Alignment assistance tools don’t provide a good enough alignment, create manual alignment points and line them up in multiple cameras. At least 3 alignment points need to be lined up in at least 2 cameras each. It’s best to edit the alignment points one by one in each camera view. Alignment points are best placed on mesh vertices that correspond to distinct features in the physical model (e.g. a corner of a building). Not every camera will see every alignment point, so additional points may be needed.

5. Create AlignmentPoints (AP). Click near a point on the wireframe to create an AP for this mesh vertex. Alternatively, AlignmentPoints can be imported from existing QuickCal projector ReferencePoints.

6. Drag it to the corresponding point in the image. Do the same for the corresponding point in the second camera view.

7. Repeat this process for a minimum of three points.

   1. A Red point means this point is not being used as part of the alignment, but has been added to the view.
   2. A Yellow point means it has been aligned in the current camera, but it is not being used in the calculations.
   3. A Orange point means it has been aligned in a different camera.
   4. A Green point means it has been aligned in both cameras and is being used in the calculation.
   5. If selected points flash, and you can use the arrow keys to move them around.
   6. When points are selected, use SHIFT + arrow keys to move the point around & hold CTRL & left-click for fine control.

Multi-screen Alignment

If there is more than one screen in the plan, it may be helpful to use Multi-screen alignment. This allows aligning surfaces in the stage based on the captured images and a common master screen. If the relative offset and rotation between all surfaces in the stage matches the relationships of the actual physical surfaces, then this may not be needed.

• The master screen should be aligned first.
• The master screen acts as a registration base for all other screens. Reshaping or scaling of the master screen will therefore affect all other screens.
• When aligning secondary screens, they will be moved to the correct position relative the Master screen.
• When stage automation is used to control physical screen positions, it is best to choose a non-moving surface as the master screen.

An example for how to align multiple surfaces can be seen in the demo video from the Overview page, at around 2:40 minutes.

Alignment Reshape

If the proportions of the model match the physical surface exactly, then the alignment should fit perfectly. But if the proportions are not correct, you may need to perform Alignment reshape. Reshape allows adjustments to mesh vertices based on lined-up alignment points. This can be useful for adjustments such as stretching or squeezing the projection surface, or even moving some internal mesh elements (such as a window) by a certain amount. Typical applications are in theatre where scene elements are built according to a CAD drawing, but the physical construct ends up with minor deviations due to the construction process or limitations imposed by the stage.

The reshape process does not use the captured point cloud, but simply adjusts existing mesh vertices and mesh proportions. This is done according to the calibrated cameras, captured alignment images and alignment point data provided by the user. UV coordinates for a vertex are not modified during Reshape.

Note

When reshape causes stretching between vertices, this will in effect reduce the spatial resolution of the content, since any texture is now covering a slightly larger physical area. If a mesh requires significant changes (such as moving a building’s window by a meter) then it may be a good idea to export the reshaped mesh, clean it up and adjust the affected UVs in a 3D modelling software, then export it again as an OBJ file into Designer. Note that such UV changes will also affect content for this screen, so this may then also need re-rendering.

Enable Reshape

1. To enable Reshape for an alignment point hold SHIFT and left-click it. This will turn it into a reshape point which is indicated by a diamond marker (45 degree rotated).
2. Left-click the point again (without holding SHIFT) to turn it back to a non-reshaped alignment point.

An example for how to use reshape points can be seen in the demo video from the Overview page, at around 4:24 minutes.

Mesh Deform

If aligning and reshaping is not enough to make the projection match the physical object, then this is likely caused by shape differences that are not captured in the current 3D mesh. The mesh will need deforming to match the real world object. A typical situation is that Reshape has handled certain vertex deviations, but there are remaining differences like uneven surface areas that are modelled as flat in the 3D mesh, or differences in curvature between the real world object and the 3D mesh.

Mesh Deform is an OmniCal feature that enhances the 3D mesh so it conforms to the captured point cloud. This is achieved by inserting new vertices into the mesh, and splitting triangles accordingly. The new vertices are based on points in the captured and aligned point cloud. Finally, a sophisticated smoothing algorithm adjusts all existing vertex coordinates according to their neighbours (including any newly inserted vertices).

Mesh Deform is able to deal with deformations in physical projection objects that are not or not adequately represented in the original mesh. For example it can accurately represent the actual surface of a curved screen, including any imperfections that arise from the construction process.

UV coordinates of original mesh vertices are not affected during Mesh Deform. Any inserted vertices get assigned UVs that are interpolated from their neighbours.

Note

Typically Mesh Deform is run once at the beginning of tech setup. As long as the physical projection surface does not change over time, the deformed mesh will be valid for any future capture and calibrations. See also Point cloud size and detail below.

Mesh Deform requirements

To use Mesh Deform correctly, a few requirements must be satisfied by the original mesh and the current state of the project:

• The original mesh must be “watertight” in areas that belong together.
  • If e.g. the North and East wall of a building are not connected in the mesh at their common edge (i.e. vertices are welded together), then they will likely get torn away from each other during Mesh Deform. Vertices with the same 3D position must be welded in the mesh, so that e.g. the top right vertex of the East wall is the same vertex as the top left vertex of the North wall, etc.
  • Holes (e.g. for windows of a building) are allowed, but it may be necessary to prevent unwanted deformations that could introduce twists to the window. This can be done by adding AlignmentPoints to the four corner vertices of each window and anchoring them (see Anchor points below).
• The OmniCal and Designer coordinate system must have been aligned, so the captured point cloud and original mesh overlap properly.
• For best Mesh Deform results it is recommended to first use Reshape to get the best possible match between mesh and point cloud.

Mesh Deform settings

Point cloud size and detail

Ideally, Mesh Deform is performed using an OmniCal capture that uses a high number of projected blobs (see grid size). The denser a point clouds is, the more detail it will provide for Mesh Deform. Therefore a larger grid size will provide a more accurate and higher quality deformed mesh. If the physical projection surface does not have many local deformations, then the default grid size is often sufficient.

Dense captures take a longer time to complete and calibrate, so only use a high grid size when a higher quality Mesh Deform is actually needed. When using high grid sizes, it is recommended to use a separate OmniCal plan with lower grid size for faster (e.g. daily) re-calibration. This is especially helpful if the calibration concerns multiple surfaces, but not all require deforming.

Note

It’s easiest to start off with an OmniCal plan using a normal number of blobs (e.g. grid size of 48 or less), to first figure out correct camera placement and settings. If it’s necessary to perform a higher quality Mesh Deform, then make a copy of this OmniCal plan and simply increase the grid size to a higher value before going through all steps from capture to Mesh Deform.

Distance threshold

This threshold denotes the maximum absolute distance between a point in the point cloud and the nearest mesh surface, along the point normal’s direction. It is used to decide whether or not to add a new vertex for a certain cloud point (and deform its neighbours accordingly), or not. If the cloud point is further away from the current aligned (and potentially reshaped) mesh than this threshold, then Mesh Deform will not add a new vertex at the cloud point’s 3D position.

This threshold is useful to exclude noisy or irrelevant data from the point cloud, e.g. trees in the foreground.

Default value: 0.1 m

Depending on the size and scale of a physical projection surface, this value may need to be increased or decreased to get better looking results.

Angle threshold

This threshold denotes the maximum angle between the normal vector of a cloud point and the normal vector of the nearest mesh surface. It is used to decide whether or not to add a new vertex for a certain cloud point (and deform its neighbours accordingly), or not. If the angle is larger than this threshold, then MeshDeform will not add a new vertex at the cloud point’s 3D position.

This threshold is useful to prevent smoothing over edges and corners (e.g. right angles between floor and wall).

Default value: 30 degrees.

Anchor points

Anchor points in Mesh Deform are vertices that do not get modified during the smoothing operation. They are used as fixed guides. Any other vertices get interpolated based on the original vertex position, its non-anchor neighbours and the anchor points near it.

Mesh Deform treats all newly inserted vertices as anchor points, since they are directly derived from the captured point cloud. Other vertices (like mesh borders or alignment points) can also be treated as anchor points. This will influence the output mesh, since those vertices will keep their original aligned (or reshaped) position. This can make sense if there is for example a high confidence on the calculated location of reshaped points. But in most cases where Mesh Deform is required this is not desirable, so these options are not enabled by default.

Anchor points can be modified using these settings:

• Lock mesh boundaries: Unlocked / Locked
  • Whether mesh boundary vertices are treated as anchor points or not.
  • Has no effect on closed meshes without boundary edges, e.g. spheres.
  • Default: Unlocked.
• Use alignment points: No, Yes, If aligned in 2+ cameras
  • Whether Alignment points from QuickAlign will be treated as anchor points or not.
  • The third option only anchors alignment points that are aligned in more than 2 cameras. Having several camera intersections can provide a higher reliability on the calculated 3D vertex position.
  • Default: No.

Visualising Mesh Deform

You can preview the Mesh Deform results by changing the Point Cloud visualisation mode to Deform in the calibration editor. The results are updated in real-time so you can try out different deform settings using this. The lines indicate where each existing mesh vertex would be moved to. Newly inserted vertices will not be shown, though.

Using Mesh Deform

It is a good idea to try the default Mesh Deform settings first, and carefully inspect the result. If the deformed mesh has included or excluded cloud points incorrectly, then decrease or increase the distance and angle parameters as required and try again.

1. Click Mesh Deform… from the Alignment tab of the Plan editor.
2. Select the screen you wish to deform by left-clicking the screen property in the deform editor.
3. Adjust parameters for this screen, as required.
4. Click Deform.

Mesh Deform Performance

Running a Mesh Deform operation may take a while, depending on the number of vertices and triangles in the original mesh, as well as the size of the captured point cloud.

After clicking Deform, the process will run in the background until it is complete. This may take several seconds (for simple meshes) up to tens of minutes (for complex meshes and dense point clouds). Mesh Deform is very CPU intensive, so although it’s possible to continue using Designer while this is running, the impact on performance is very noticeable.