Optimal Seamline Detection in Dynamic Scenes via Graph Cuts for Image Mosaicking

Li Li, Jian Yao*, Xiaohu Lu, Renping Xie

School of Remote Sensing and Information Engineering, Wuhan University, Wuhan, Hubei, P.R.China

*EMail: jian.yao@whu.edu.cn

*Web: http://www.scholat.com/jianyao http://cvrs.whu.edu.cn

[] The executable files are released.download (15/1/2015).

[] The test dataset is added.download (15/1/2015).


In this paper, we present a novel method for creating a seamless mosaic with relatively clean background from two geometrically aligned images captured from a scene with dynamic objects at different time through removing artifacts caused by dynamic objects and geometric misalignments and simultaneously compensating the image regions of dynamic objects based on optimal seamline detection in the graph cuts energy minimization framework. To effectively ensure that all seamlines will be detected in the laterally continuous regions with the high image similarity and the low object dislocation despite the presence of dynamic objects, the energy functions adopted in graph cuts combine the pixel-level similarities of image characteristics, including color and gradient, and the texture complexity. To successfully compensate the image regions covered by dynamic objects for creating a mosaic with relatively clean background, we initially detect them in overlap regions between images based on pixel-level and region-level similarities followed by refinement based on segmentation, then determine their image source in probability based on contour matching, and finally integrate all of these into the energy minimization framework. Experimental results on challenging indoor dynamic scenes demonstrate that our method is capable of generating high-quality mosaics with relatively clean background base on the detected optimal seamlines.

Figure 1: Two geometrically aligned input images A and B in (a) and (b) with a moving object in the overlap region as shown in (c). The image mosaics with different seamlines are presented in (d)-(g), which avoid crossing the moving object. The moving object is visible in (d)-(f), however, it is invisible in (g) by the compensation with the background image regions, which is the goal of our method.


In this paper, not only the color and gradient differences were considered into the cost of each pixel in overlap regions between images but also the texture complexity inspired by HOG (Histogram of Oriented Gradient) was integrated into the cost function. In addition, to create image mosaics with relatively clean background, the dynamic objects are detected initially by computing the pixel-level similarity in the CIELAB color space and the region-level similarity between HOG feature descriptors, then refined by segments generated by the Mean Shift, and finally determined from which image they come in probability based on contour matching. All of these are integrated into the graph cuts energy minimization framework for detecting optimal seamlines concealing the parallax and compensating the image regions covered by dynamic objects for image mosaicking, as shown in Figure 1(g).

Figure 2: An illustration example of the optimal seamline detection method via graph cuts. The thickness of lines between adjacent pixels represents the value of the energy cost and the "cut" denotes the minimum cut, which means the optimal seamline.

Figure 3: The three typical regions (top) happened in indoor scenes and their corresponding normalized texture complexity maps (bottom) where the lighter regions indicate higher texture complexity.

Figure 4: (a)-(b) two geometrically aligned input source images with dynamic objects; (c) the color distance map in CIELAB; (d) the HOG distance map; (e) the final distance map; (f) the dynamic objects binary map refined by the Mean Shift algorithm; (g)-(h) the finally mosaicked images with optimal seamlines without/with dynamic object compensation..

Experimental Results

We tested our proposed method on the images captured from several challenging indoor dynamic scenes with moving objects by rotating a Nikon D7100 camera of 24 million pixels with a wide-angle lens on a tripod platform at different times. The faces in some pictures presented in this paper have been blurred due to privacy protection. These images were first aligned and warped into a common coordinate system via the popularly used commercial software PTGui(http://www.ptgui.com/) and then utilized to test our proposed optimal seamline detection algorithm despite the presence of dynamic objects.

(a) (b) (c) (d)
(a) (b) (c) (d)
(a) (b) (c) (d)

Figure 5: The geometrically aligned images in two dynamic scenes in the first and second columns and the final image mosaics with dynamic objects compensation in the fourth column and without the compensation in the third column.

(a) (b) (c) (d)

Figure 6: Two geometrically aligned images captured from a dynamic scene with a local motion of a human head are mosaicked based on the seamlines detected by the hard constraint in the third picture and the soft one in the last one.

Dataset and Codes

1. Executable files: [Win64]

2. Test dataset: [Download]


Li Li, Jian Yao*, Xiaohu Lu, and Renping Xie. Optimal Seamline Detection in Dymamic Scenes via Graph Cuts for Image Mosaicking, submitted to IEEE International Conference on Image Processing (ICIP), 2015.