Abstract
We present an algorithm to synthetically increase the resolution of a solitary depth image using only a generic database of local patches. Modern range sensors measure depths with non-Gaussian noise and at lower starting resolutions than typical visible-light cameras. While patch based approaches for upsampling intensity images continue to improve, this is the first exploration of patching for depth images.
We match against the height field of each low resolution input depth patch, and search our database for a list of appropriate high resolution candidate patches. Selecting the right candidate at each location in the depth image is then posed as a Markov random field labeling problem. Our experiments also show how important further depth-specific processing, such as noise removal and correct patch normalization, dramatically improves our results. Perhaps surprisingly, even better results are achieved on a variety of real test scenes by providing our algorithm with only synthetic training depth data.
Chapter PDF
Similar content being viewed by others
Keywords
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
References
Yang, Q., Yang, R., Davis, J., Nister, D.: Spatial-depth super resolution for range images. In: CVPR (2007)
Schuon, S., Theobalt, C., Davis, J., Thrun, S.: Lidarboost: Depth superresolution for tof 3D shape scanning. In: CVPR (2009)
Freeman, W.T., Pasztor, E.C., Carmichael, O.T.: Learning low-level vision. IJCV (2000)
Kuster, C., Popa, T., Zach, C., Gotsman, C., Gross, M.: Freecam: A hybrid camera system for interactive free-viewpoint video. In: Proceedings of Vision, Modeling, and Visualization, VMV (2011)
Holz, D., Schnabel, R., Droeschel, D., Stückler, J., Behnke, S.: Towards semantic scene analysis with time-of-flight cameras. In: RoboCup International Symposium (2010)
Shotton, J., Fitzgibbon, A., Cook, M., Sharp, T., Finocchio, M., Moore, R., Kipman, A., Blake, A.: Real-time human pose recognition in parts from single depth images. In: CVPR (2011)
Park, J., Kim, H., Tai, Y.W., Brown, M., Kweon, I.: High quality depth map upsampling for 3D-ToF cameras. In: ICCV (2011)
Fattal, R.: Upsampling via imposed edges statistics. SIGGRAPH (2007)
Yang, L., Sander, P.V., Lawrence, J., Hoppe, H.: Antialiasing recovery. ACM Transactions on Graphics (2011)
Irani, M., Peleg, S.: Improving resolution by image registration. In: CVGIP: Graph. Models Image Process. (1991)
Hahne, U., Alexa, M.: Exposure Fusion for Time-Of-Flight Imaging. In: Pacific Graphics (2011)
Rajagopalan, A.N., Bhavsar, A., Wallhoff, F., Rigoll, G.: Resolution Enhancement of PMD Range Maps. In: Rigoll, G. (ed.) DAGM 2008. LNCS, vol. 5096, pp. 304–313. Springer, Heidelberg (2008)
Izadi, S., Kim, D., Hilliges, O., Molyneaux, D., Newcombe, R.A., Kohli, P., Shotton, J., Hodges, S., Freeman, D., Davison, A.J., Fitzgibbon, A.: Kinectfusion: Real-time 3D reconstruction and interaction using a moving depth camera. In: UIST (2011)
Cui, Y., Schuon, S., Derek, C., Thrun, S., Theobalt, C.: 3D shape scanning with a time-of-flight camera. In: CVPR (2010)
Freeman, W.T., Jones, T.R., Pasztor, E.C.: Example-based super-resolution. Computer Graphics and Applications (2002)
Freeman, W.T., Liu, C.: Markov random fields for super-resolution and texture synthesis. In: Advances in Markov Random Fields for Vision and Image Processing. MIT Press (2011)
Yang, J., Wright, J., Huang, T.S., Ma, Y.: Image super-resolution via sparse representation. IEEE Transactions on Image Processing (2010)
Zontak, M., Irani, M.: Internal statistics of a single natural image. In: CVPR (2011)
Glasner, D., Bagon, S., Irani, M.: Super-resolution from a single image. In: ICCV (2009)
Sun, J., Zhu, J., Tappen, M.: Context-constrained hallucination for image super-resolution. In: CVPR (2010)
HaCohen, Y., Fattal, R., Lischinski, D.: Image upsampling via texture hallucination. In: ICCP (2010)
Gal, R., Shamir, A., Hassner, T., Pauly, M., Cohen-Or, D.: Surface reconstruction using local shape priors. In: Symposium on Geometry Processing (2007)
Golovinskiy, A., Matusik, W., Pfister, H., Rusinkiewicz, S., Funkhouser, T.: A statistical model for synthesis of detailed facial geometry. SIGGRAPH (2006)
Huang, J., Lee, A., Mumford, D.: Statistics of range images. In: CVPR (2000)
Diebel, J., Thrun, S.: An application of markov random fields to range sensing. In: NIPS (2005)
Chan, D., Buisman, H., Theobalt, C., Thrun, S.: A Noise-Aware Filter for Real-Time Depth Upsampling. In: Workshop on Multi-camera and Multi-modal Sensor Fusion Algorithms and Applications (2008)
Schuon, S., Theobalt, C., Davis, J., Thrun, S.: High-quality scanning using time-of-flight depth superresolution. In: CVPR Workshops (2008)
Reynolds, M., Doboš, J., Peel, L., Weyrich, T., Brostow, G.J.: Capturing time-of-flight data with confidence. In: CVPR (2011)
Kolmogorov, V.: Convergent tree-reweighted message passing for energy minimization. PAMI (2006)
Wainwright, M., Jaakkola, T., Willsky, A.: Map estimation via agreement on (hyper)trees: Message-passing and linear programming approaches. IEEE Transactions on Information Theory (2002)
Frank, M., Plaue, M., Rapp, H., Köthe, U., Jähnea, B., Hamprecht, F.A.: Theoretical and experimental error analysis of continuous-wave time-of-flight range cameras. Optical Engineering (2009)
Tomasi, C., Manduchi, R.: Bilateral filtering for gray and color images. In: ICCV (1998)
(USF Range Database), http://marathon.csee.usf.edu/range/DataBase.html
Saxena, A., Sun, M., Ng, A.: Make3d: Learning 3d scene structure from a single still image. PAMI (2009)
Saxena, A., Driemeyer, J., Kearns, J., Ng, A.: Robotic grasping of novel objects. In: NIPS (2006)
Scharstein, D., Szeliski, R.: A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. IJCV (2002)
Silberman, N., Hoiem, D., Kohli, P., Fergus, R.: Indoor segmentation and support inference from RGBD images. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012, Part V. LNCS, vol. 7576, pp. 746–760. Springer, Heidelberg (2012)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2012 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Mac Aodha, O., Campbell, N.D.F., Nair, A., Brostow, G.J. (2012). Patch Based Synthesis for Single Depth Image Super-Resolution. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds) Computer Vision – ECCV 2012. ECCV 2012. Lecture Notes in Computer Science, vol 7574. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-33712-3_6
Download citation
DOI: https://doi.org/10.1007/978-3-642-33712-3_6
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-33711-6
Online ISBN: 978-3-642-33712-3
eBook Packages: Computer ScienceComputer Science (R0)