Globally Consistent Alignment for Mosaicking Aerial Images

1. Abstract

In this paper, we present a robust method to efficiently create a globally consistent and seamless mosaic from aerial images. Firstly, a globally consistent registration strategy is proposed to align the aerial images in a common coordinate system, which combines the affine model with the homographic model effectively. To suppress the error accumulation of perspective deformations via a sequential set of aerial images taken from a wide-range region, we proposed to initially align each image by an affine model and then perform a homographic refinement in groups to increase the global consistency. Secondly, to efficiently conceal the parallax between aligned images in regions with large depth differences where it is impossible for recovering a highly accurate consistent image registration, a novel optimized seamline detection algorithm in the graph cuts energy minimization framework is proposed to find optimal seamlines within overlapped regions for image mosaicking through rounding visually obvious foreground objects. Finally, experimental results on several representive image sets illustrates the superiority of our proposed approaches.

2. Approach

Our approach is mainly composed of two parts: global alignment and parallax removal. The global alignment module aims at mosaicking sequential images with keeping both accurate alignment and global consistency, and the parallax removal module is employed to conceal the parallax between aligned images in regions with large depth differences.

Global alignment module:

• Each image is aligned to the reference frame initially by an affine transformation, and all the previously aligned images with an overlap with the new image will be jointly used in the optimization framework;
• As the image being aligned to the mosaic frame, a fixed number of adjacent images, as a group, are optimized jointly by homographic transformations;
• At last, After the optimization of all groups is sequentially finished, the homographic models of all the aerial images will be further jointly optimized as a whole group;

Parallax removal module:

• To remove the parallax between images in regions with large depth differences for image mosaicking, we proposed an effective optimal seamlines detection algorithm via graph cuts, which try to find seamlines with smallest difference in color and gradient magnitude between images in the overlapped regions.

The details of algorithm are described in the paper.

3. Experimental Results

3.1. Information of Experimental Data

To sufficiently test the performance of our proposed algorithm for global alignment of aerial images, we chose a set of 90 aerial images in three strips with the down-sampling image size of 746×480 and a set of 48 UAV images in three strips with the down-sampling image size of 800×533 for evaluation. These two test image sequences are mainly different in flight altitudes and stability of image acquisition platforms, in which there exist an overlapping rate of about 60% in adjacent images.

Part of the aerial sequence images and the UAV sequence images are listed in Fig. 1 and Fig. 2 respectively.

Fig. 1. Part of thumbnails of of the aerial sequence images.

Fig. 2. Part of thumbnails of the UAV sequence images.

3.2. Evaluation on Selection of Initial Models

In the period of obtaining initial alignment, the selection of the transformation model among rigid, affine and homographic models can make some differences to the final mosaicking result, which are compared through doing experiments on the two datasets above. The visual results and numerical results are expressed below:

Visual results(Table. 1. Click the Thumbnails to view the original images ):

Table 1. Thumbnails of the mosaicking results on the aerial images and the UAV images.

Initial Models	Aerial Images	UAV Images
Rigid
Affine
Homographic

Numerical results(Fig. 3, Fig. 4 and Table 2):

Fig. 3. Numerical comparison of different transformation models chosen for initial alignment
before the final global optimization performed: Areial images(Left); UAV images(Right).

Fig. 4. Numerical comparison of different transformation models chosen for initial alignment
after the final global optimization performed: Areial images(Left); UAV images(Right).

Table 2. RMS of the reprojection errors for different transformation models
chosen for initial alignment (GO: Global Optimization; Unit: pixel).

3.3. Comparative Evaluation

We compared our approach with a commercial software named PTGui. The visually comparative results of mosaicking aerial images and UAV ones are illustrated in Fig. 5 and Fig. 6 respectively,in which the optimal seamline detection and image blending are not applied. You can view the original images by click the images.

Notice that the cyan box is labeled for showing the position of the reference image. Besides, the results of PTGui used for comparisons are preview versions in the processing procedure of PTGui( as you can see the ruler lines on the images), because the mosaicking result from overlapping each image directly is not available.

(a)

(b)

Fig. 5. Visual comparison of our approach (a) and PTGui (b) on aerial sequence images

(a)

(b)

Fig. 6. Visual comparison of our approach (a) and PTGui (b) on UAV sequence images

3.4. Validity of Optimal Seamlines

Based on our proposed optimal seamline detection via graph cuts, the extracted seamlines in overlapped regions between images can visually conceal the parallax. Fig. 7 shows the seamline detection result of our proposed algorithm on 6 images of a single strip, where our optimal seamlines successfully bypassed the tall buildings where there exist large parallaxes between images. You can view the original images by click the images.

(a)	(b)

Fig. 7. An example of optimal seamline detection for parallax removal: overlapped edges (a) and optimal seamlines (b)

4. Supplementary Experiments

4.1. Test on Global Alignment for Image Mosaicking

To further validate the superiority of our approach in global alignment for image mosaicking, we tested another set of aerial images which is selected from the same data set as the 90 aerial images mentioned above. This set of aerial images, 90 images of 3 strips, were mainly captured on the building-compact districts. Besides, we also try to mosaic images selected from a single strip.

The mosaicking results of a 3-strips and two single-strip are showed in Fig. 8, Fig. 9 and Fig. 10, and you can view the original images by click the images. What’s more, the numerical results are available in Table 3.

Fig. 8 Mosaicking result of the 3-strips

Fig. 9 Mosaicking result of the Single-Strip1

Fig. 10 Mosaicking result of the Single-Strip2

Table 3. RMS of the reprojection errors of different data (Unit: pixel).

Data	Matches	RMS	RMS(GO)
3-Strip	52284	0.988	0.907
Single-Strip2	11131	0.647	0.636
Single-Strip1	12994	0.827	0.790

4.2. Test on Different Values of λ

Besides, we did experiments on two sets of images from two single strips with different values of λ in the cost function, and the two single strips are Single-Strip1 and Single-Strip2 mentioned above, which represent the scene with large depth differences and the scene of small depth differences, respectively. The mean RMS of reproject-error of mosaicking result with different λ are showed in Fig. 11, and several representative mosaic images with different value of λ are compared in Table 4 to show the difference in global consistency(detail differences are labelled by red boxes).

Judged by both the global consistency of mosaicking result and alignment precision, we find that the value between 0.03 and 0.11 can always give a good mosaicking result for both case, and the optimal one should be set according to the property of data, such as 0.06 and 0.09 are respectively chosen for Single-Strip1 and Single-Strip2 in our experiments.

(a)	(b)

Fig. 11. Alignment precision of different values of λ on the small-depth-difference case (a) and the large-depth-difference case (b)

Table 4. Thumbnails of the mosaicking results with different values of λ on Single-Strip1 and Single-Strip2
(Click the Thumbnails to view the original images).

Value of λ	Single-Strip1	Value of λ	Single-Strip2
0.0		0.0
0.06		0.09
0.19-0.57		0.25-0.57

Citation

Menghan Xia, Jian Yao*, Li Li, and Xiaohu Lu. Globally Consistent Alignment for Mosaicking Aerial Images, submitted to IEEE International Conference on Image Processing (ICIP), 2015. PDF