Skip to main content
Springer Nature Link
Log in

Image Stitching and Rectification for Hand-Held Cameras

  • Conference paper
  • First Online:
Computer Vision – ECCV 2020 (ECCV 2020)

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 12352))

Included in the following conference series:

Abstract

In this paper, we derive a new differential homography that can account for the scanline-varying camera poses in Rolling Shutter (RS) cameras, and demonstrate its application to carry out RS-aware image stitching and rectification at one stroke. Despite the high complexity of RS geometry, we focus in this paper on a special yet common input—two consecutive frames from a video stream, wherein the inter-frame motion is restricted from being arbitrarily large. This allows us to adopt simpler differential motion model, leading to a straightforward and practical minimal solver. To deal with non-planar scene and camera parallax in stitching, we further propose an RS-aware spatially-varying homogarphy field in the principle of As-Projective-As-Possible (APAP). We show superior performance over state-of-the-art methods both in RS image stitching and rectification, especially for images captured by hand-held shaking cameras.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    See our supplementary material for derivations.

References

  1. Albl, C., Kukelova, Z., Larsson, V., Polic, M., Pajdla, T., Schindler, K.: From two rolling shutters to one global shutter. In: CVPR (2020)

    Google Scholar 

  2. Albl, C., Kukelova, Z., Pajdla, T.: R6p-rolling shutter absolute camera pose. In: CVPR (2015)

    Google Scholar 

  3. Albl, C., Kukelova, Z., Pajdla, T.: Rolling shutter absolute pose problem with known vertical direction. In: CVPR (2016)

    Google Scholar 

  4. Albl, C., Sugimoto, A., Pajdla, T.: Degeneracies in rolling shutter SfM. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9909, pp. 36–51. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46454-1_3

    Chapter  Google Scholar 

  5. Bapat, A., Price, T., Frahm, J.M.: Rolling shutter and radial distortion are features for high frame rate multi-camera tracking. In: CVPR (2018)

    Google Scholar 

  6. Brown, M., Lowe, D.G.: Automatic panoramic image stitching using invariant features. Int. J. Comput. Vis. 74(1), 59–73 (2007)

    Article  Google Scholar 

  7. Chang, C.H., Sato, Y., Chuang, Y.Y.: Shape-preserving half-projective warps for image stitching. In: CVPR (2014)

    Google Scholar 

  8. Chen, Y.S., Chuang, Y.Y.: Natural image stitching with the global similarity prior. In: ECCV (2016)

    Google Scholar 

  9. Cox, D.A., Little, J., O’shea, D.: Using Algebraic Geometry, vol. 185. Springer, Heidelberg (2006). https://doi.org/10.1007/b138611

    Book  MATH  Google Scholar 

  10. Dai, Y., Li, H., Kneip, L.: Rolling shutter camera relative pose: generalized epipolar geometry. In: CVPR (2016)

    Google Scholar 

  11. Grundmann, M., Kwatra, V., Castro, D., Essa, I.: Calibration-free rolling shutter removal. In: ICCP (2012)

    Google Scholar 

  12. Haresh, S., Kumar, S., Zia, M.Z., Tran, Q.H.: Towards anomaly detection in dashcam videos. In: IV (2020)

    Google Scholar 

  13. Hartley, R., Zisserman, A.: Multiple View Geometry in Computer Vision. Cambridge University Press, Cambridge (2003)

    MATH  Google Scholar 

  14. Hartley, R.I.: In defense of the eight-point algorithm. IEEE Trans. Pattern Anal. Mach. Intell. 19(6), 580–593 (1997)

    Article  Google Scholar 

  15. Hedborg, J., Forssén, P.E., Felsberg, M., Ringaby, E.: Rolling shutter bundle adjustment. In: CVPR (2012)

    Google Scholar 

  16. Heeger, D.J., Jepson, A.D.: Subspace methods for recovering rigid motion I: algorithm and implementation. Int. J. Comput. Vis. 7(2), 95–117 (1992)

    Article  Google Scholar 

  17. Herrmann, C., et al.: Robust image stitching with multiple registrations. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11206, pp. 53–69. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01216-8_4

    Chapter  Google Scholar 

  18. Herrmann, C., Wang, C., Bowen, R.S., Keyder, E., Zabih, R.: Object-centered image stitching. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11207, pp. 846–861. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01219-9_50

    Chapter  Google Scholar 

  19. Horn, B.K.: Motion fields are hardly ever ambiguous. Int. J. Comput. Vis. 1(3), 259–274 (1988)

    Article  Google Scholar 

  20. Im, S., Ha, H., Choe, G., Jeon, H.G., Joo, K., So Kweon, I.: High quality structure from small motion for rolling shutter cameras. In: ICCV (2015)

    Google Scholar 

  21. Ito, E., Okatani, T.: Self-calibration-based approach to critical motion sequences of rolling-shutter structure from motion. In: CVPR (2017)

    Google Scholar 

  22. Klingner, B., Martin, D., Roseborough, J.: Street view motion-from-structure-from-motion. In: ICCV (2013)

    Google Scholar 

  23. Kukelova, Z., Albl, C., Sugimoto, A., Pajdla, T.: Linear solution to the minimal absolute pose rolling shutter problem. In: Jawahar, C.V., Li, H., Mori, G., Schindler, K. (eds.) ACCV 2018. LNCS, vol. 11363, pp. 265–280. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-20893-6_17

    Chapter  Google Scholar 

  24. Lao, Y., Aider, O.A.: Rolling shutter homography and its applications. In: IEEE Trans. Pattern Anal. Mach. Intell. (2020)

    Google Scholar 

  25. Lao, Y., Ait-Aider, O.: A robust method for strong rolling shutter effects correction using lines with automatic feature selection. In: CVPR (2018)

    Google Scholar 

  26. Lao, Y., Ait-Aider, O., Bartoli, A.: Rolling shutter pose and ego-motion estimation using shape-from-template. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11206, pp. 477–492. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01216-8_29

    Chapter  Google Scholar 

  27. Lee, K.Y., Sim, J.Y.: Warping residual based image stitching for large parallax. In: CVPR (2020)

    Google Scholar 

  28. Li, S., Yuan, L., Sun, J., Quan, L.: Dual-feature warping-based motion model estimation. In: ICCV (2015)

    Google Scholar 

  29. Liao, T., Li, N.: Single-perspective warps in natural image stitching. IEEE Trans. Image Process. 29, 724–735 (2019)

    Article  MathSciNet  Google Scholar 

  30. Lin, C.C., Pankanti, S.U., Natesan Ramamurthy, K., Aravkin, A.Y.: Adaptive as-natural-as-possible image stitching. In: CVPR (2015)

    Google Scholar 

  31. Lin, K., Jiang, N., Cheong, L.-F., Do, M., Lu, J.: SEAGULL: seam-guided local alignment for parallax-tolerant image stitching. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9907, pp. 370–385. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46487-9_23

    Chapter  Google Scholar 

  32. Lin, K., Jiang, N., Liu, S., Cheong, L.F., Do, M., Lu, J.: Direct photometric alignment by mesh deformation. In: CVPR (2017)

    Google Scholar 

  33. Lin, W.Y., Liu, S., Matsushita, Y., Ng, T.T., Cheong, L.F.: Smoothly varying affine stitching. In: CVPR (2011)

    Google Scholar 

  34. Liu, F., Gleicher, M., Jin, H., Agarwala, A.: Content-preserving warps for 3D video stabilization. ACM Trans. Graph. (TOG) 28(3), 1–9 (2009)

    Google Scholar 

  35. Liu, P., Cui, Z., Larsson, V., Pollefeys, M.: Deep shutter unrolling network. In: CVPR (2020)

    Google Scholar 

  36. Liu, S., Yuan, L., Tan, P., Sun, J.: Bundled camera paths for video stabilization. ACM Trans. Graph. (TOG) 32(4), 1–10 (2013)

    Google Scholar 

  37. Lowe, D.G.: Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis. 60(2), 91–110 (2004)

    Article  Google Scholar 

  38. Ma, Y., Košecká, J., Sastry, S.: Linear differential algorithm for motion recovery: a geometric approach. Int. J. Comput. Vis. 36(1), 71–89 (2000)

    Article  Google Scholar 

  39. Ma, Y., Soatto, S., Kosecka, J., Sastry, S.S.: An Invitation to 3-D Vision: From Images to Geometric Models, vol. 26. Springer, Heidelberg (2012)

    MATH  Google Scholar 

  40. Magerand, L., Bartoli, A., Ait-Aider, O., Pizarro, D.: Global optimization of object pose and motion from a single rolling shutter image with automatic 2D-3D matching. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012. LNCS, vol. 7572, pp. 456–469. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-33718-5_33

    Chapter  Google Scholar 

  41. Meingast, M., Geyer, C., Sastry, S.: Geometric models of rolling-shutter cameras. In: Workshop on Omnidirectional Vision, Camera Networks and Non-Classical Cameras (2005)

    Google Scholar 

  42. Mohan, M.M., Rajagopalan, A., Seetharaman, G.: Going unconstrained with rolling shutter deblurring. In: ICCV (2017)

    Google Scholar 

  43. Mur-Artal, R., Montiel, J.M.M., Tardos, J.D.: ORB-SLAM: a versatile and accurate monocular slam system. IEEE Trans. Rob. 31(5), 1147–1163 (2015)

    Article  Google Scholar 

  44. Muratov, O., Slynko, Y., Chernov, V., Lyubimtseva, M., Shamsuarov, A., Bucha, V.: 3DCapture: 3D reconstruction for a smartphone. In: CVPRW (2016)

    Google Scholar 

  45. Oth, L., Furgale, P., Kneip, L., Siegwart, R.: Rolling shutter camera calibration. In: CVPR (2013)

    Google Scholar 

  46. Punnappurath, A., Rengarajan, V., Rajagopalan, A.: Rolling shutter super-resolution. In: ICCV (2015)

    Google Scholar 

  47. Purkait, P., Zach, C.: Minimal solvers for monocular rolling shutter compensation under ackermann motion. In: WACV (2018)

    Google Scholar 

  48. Purkait, P., Zach, C., Leonardis, A.: Rolling shutter correction in Manhattan world. In: ICCV (2017)

    Google Scholar 

  49. Rengarajan, V., Balaji, Y., Rajagopalan, A.: Unrolling the shutter: CNN to correct motion distortions. In: CVPR (2017)

    Google Scholar 

  50. Rengarajan, V., Rajagopalan, A.N., Aravind, R.: From bows to arrows: rolling shutter rectification of urban scenes. In: CVPR (2016)

    Google Scholar 

  51. Rengarajan, V., Rajagopalan, A.N., Aravind, R., Seetharaman, G.: Image registration and change detection under rolling shutter motion blur. IEEE Trans. Pattern Anal. Mach. Intell. 39(10), 1959–1972 (2016)

    Article  Google Scholar 

  52. Ringaby, E., Forssén, P.E.: Efficient video rectification and stabilisation for cell-phones. Int. J. Comput. Vis. 96(3), 335–352 (2012)

    Article  Google Scholar 

  53. Rublee, E., Rabaud, V., Konolige, K., Bradski, G.: ORB: an efficient alternative to SIFT or SURF. In: ICCV (2011)

    Google Scholar 

  54. Saurer, O., Koser, K., Bouguet, J.Y., Pollefeys, M.: Rolling shutter stereo. In: ICCV (2013)

    Google Scholar 

  55. Saurer, O., Pollefeys, M., Hee Lee, G.: Sparse to dense 3D reconstruction from rolling shutter images. In: CVPR (2016)

    Google Scholar 

  56. Saurer, O., Pollefeys, M., Lee, G.H.: A minimal solution to the rolling shutter pose estimation problem. In: IROS (2015)

    Google Scholar 

  57. Schonberger, J.L., Frahm, J.M.: Structure-from-motion revisited. In: CVPR (2016)

    Google Scholar 

  58. Schubert, D., Demmel, N., Usenko, V., Stuckler, J., Cremers, D.: Direct sparse odometry with rolling shutter. In: ECCV (2018)

    Google Scholar 

  59. Szeliski, R., et al.: Image alignment and stitching: a tutorial. Found. Trends® Comput. Graph. Vis. 2(1), 1–104 (2007)

    Article  Google Scholar 

  60. Tran, Q.-H., Chin, T.-J., Carneiro, G., Brown, M.S., Suter, D.: In defence of RANSAC for outlier rejection in deformable registration. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012. LNCS, vol. 7575, pp. 274–287. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-33765-9_20

    Chapter  Google Scholar 

  61. Vasu, S., Mohan, M.M., Rajagopalan, A.: Occlusion-aware rolling shutter rectification of 3D scenes. In: CVPR (2018)

    Google Scholar 

  62. Vasu, S., Rajagopalan, A.N., Seetharaman, G.: Camera shutter-independent registration and rectification. IEEE Trans. Image Process. 27(4), 1901–1913 (2017)

    Article  MathSciNet  Google Scholar 

  63. Zaragoza, J., Chin, T.J., Brown, M.S., Suter, D.: As-projective-as-possible image stitching with moving DLT. In: CVPR (2013)

    Google Scholar 

  64. Zaragoza, J., Chin, T.J., Tran, Q.H., Brown, M.S., Suter, D.: As-projective-as-possible image stitching with moving DLT. IEEE Trans. Pattern Anal. Mach. Intell. 36(7), 1285–1298 (2014)

    Article  Google Scholar 

  65. Zhang, F., Liu, F.: Parallax-tolerant image stitching. In: CVPR (2014)

    Google Scholar 

  66. Zhuang, B., Cheong, L.F., Hee Lee, G.: Rolling-shutter-aware differential SFM and image rectification. In: ICCV (2017)

    Google Scholar 

  67. Zhuang, B., Cheong, L.F., Hee Lee, G.: Baseline desensitizing in translation averaging. In: CVPR (2018)

    Google Scholar 

  68. Zhuang, B., Tran, Q.H., Ji, P., Cheong, L.F., Chandraker, M.: Learning structure-and-motion-aware rolling shutter correction. In: CVPR (2019)

    Google Scholar 

  69. Zhuang, B., Tran, Q.H., Lee, G.H., Cheong, L.F., Chandraker, M.: Degeneracy in self-calibration revisited and a deep learning solution for uncalibrated SLAM. In: IROS (2019)

    Google Scholar 

Download references

Acknowledgement

We would like to thank Buyu Liu, Gaurav Sharma, Samuel Schulter, and Manmohan Chandraker for proofreading and support of this work. We are also grateful to all the reviewers for their constructive suggestions.

Author information

Authors and Affiliations

  1. NEC Labs America, Princeton, USA

    Bingbing Zhuang & Quoc-Huy Tran

Authors
  1. Bingbing Zhuang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Quoc-Huy Tran
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Bingbing Zhuang .

Editor information

Editors and Affiliations

  1. University of Oxford, Oxford, UK

    Andrea Vedaldi

  2. Graz University of Technology, Graz, Austria

    Horst Bischof

  3. University of Freiburg, Freiburg im Breisgau, Germany

    Thomas Brox

  4. University of North Carolina at Chapel Hill, Chapel Hill, NC, USA

    Jan-Michael Frahm

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (mp4 25969 KB)

Supplementary material 2 (mp4 33927 KB)

Supplementary material 3 (pdf 16598 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Zhuang, B., Tran, QH. (2020). Image Stitching and Rectification for Hand-Held Cameras. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, JM. (eds) Computer Vision – ECCV 2020. ECCV 2020. Lecture Notes in Computer Science(), vol 12352. Springer, Cham. https://doi.org/10.1007/978-3-030-58571-6_15

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-58571-6_15

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-58570-9

  • Online ISBN: 978-3-030-58571-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation