Sparse-MVS

The following instructions shows how to make a reconstruction in the sparse-mvs setting and submit a evaluation result.

1. Prepare the data

Download the image set and evaluation code (Distance matrix) of DTU dataset from the download page. Pick the testing set from all scan models following the paper SurfaceNet+. Reconstruct all the models and save the results into ply files.

2. Evaluation

2.1 Description

The precision and recall have two metrics: the distance metric and the percentage metric. The distance metric is the original DTU dataset matlab code (download). The percentage metric is changed by the evaluation code provided by Tanks and Temples (download). The overall score for the percentage metric is measured as the f-score, and a similar measurement for the distance metric overall is given by the average of the mean precision and mean recall.

2.2 Offline evaluation

Use the distance metric to count the mean precision(accuracy) and recall(completeness) of each model and count total the mean value of all model in the testing dataset. Use the percentage metric to count the value only at 1mm and 2mm. Count the precision, recall and f-score and count the mean value of all model in the testing dataset.

2.3 Qualitative evaluation

To better show the performance comparison and variation tendency between among methods under different sparsity, we also provide quantitative demonstration with the specific camera poses. The following figure shows the comparision of scan 23 among three different methods under four sparsity:1,3,5,7. Here we only pick one view of one scan for comparison. We use Meshlab software for our visualization tool. You should show one specific view provided by us of three models in DTU dataset: scan 1, scan 23 and scan 114. The corresponding camera poses in meshlab of each scan are:

scan 1 :

     <!DOCTYPE ViewState>
      <project>
       <VCGCamera TranslationVector="-243.076 -83.7638 -411.173 1" LensDistortion="0 0" ViewportPx="1575 978" PixelSizeMm="0.0369161 0.0369161" CenterPx="787 489" FocalMm="31.2669" RotationMatrix="0.55268 -0.754127 0.354735 0 -0.438307 -0.625053 -0.645908 0 0.708825 0.201497 -0.675993 0 0 0 0 1 "/>
       <ViewSettings NearPlane="1.03109" TrackScale="0.00816228" FarPlane="13.0311"/>
       <Render Lighting="0" DoubleSideLighting="0" SelectedVert="0" ColorMode="3" SelectedFace="0" BackFaceCull="0" FancyLighting="0" DrawMode="2" TextureMode="0"/>
      </project>

scan 23 :

     <!DOCTYPE ViewState>
      <project>
       <VCGCamera TranslationVector="-282.768 -47.9743 -516.996 1" LensDistortion="0 0" ViewportPx="1575 978" PixelSizeMm="0.0369161 0.0369161" CenterPx="787 489" FocalMm="31.2669" RotationMatrix="0.354282 -0.851435 0.386708 0 -0.276822 -0.490484 -0.826314 0 0.893226 0.185699 -0.409466 0 0 0 0 1 "/>
       <ViewSettings NearPlane="1.03109" TrackScale="0.00875987" FarPlane="13.0311"/>
       <Render Lighting="0" DoubleSideLighting="0" SelectedVert="0" ColorMode="3" SelectedFace="0" BackFaceCull="0" FancyLighting="0" DrawMode="2" TextureMode="0"/>
      </project>

scan 114 :

     <!DOCTYPE ViewState>
      <project>
       <VCGCamera TranslationVector="-161.594 -152.165 -483.126 1" LensDistortion="0 0" ViewportPx="1575 978" PixelSizeMm="0.0369161 0.0369161" CenterPx="787 489" FocalMm="31.2669" RotationMatrix="0.791183 -0.530364 0.304538 0 -0.0815598 -0.585006 -0.806918 0 0.606117 0.613582 -0.506103 0 0 0 0 1 "/>
       <ViewSettings NearPlane="1.03109" TrackScale="0.0090205" FarPlane="13.0311"/>
       <Render Lighting="0" DoubleSideLighting="0" SelectedVert="0" ColorMode="3" SelectedFace="0" BackFaceCull="0" FancyLighting="0" DrawMode="2" TextureMode="0"/>
      </project>

Here you can just copy the above camera pose text and past it into the meshlab software, then the viewing direction will automatically ture in to the desired angle. Then clip the "save snapshot" button to record the image.

3. Submission

You should provide three different evaluation metrics as described above at sparsity=1,2,3,...,11 . That means, there is a total evaluation number of 9X11=99. Here is an example file you need to submit.

Download the submission file template

We also require the method reference and project page.