Eurographics'99 - short paper demo

Registration methods for harmonious integration of real and computer generated objects

G. Simon, V. Lepetit and M.-O. Berger


This page is an extension of our paper entitled "Registration methods for harmonious integration of real and computer generated objects". We focus in this paper on the problem of adding computer-generated objects in video sequences. We propose a robust method for viewpoint computation which utilizes 3D knowledge on the scene as well as 2D/2D correspondences of key-points. Next, we extend this method to the case of unknown focal length varying from image to image. Finally, we describe how to solve possible occlusions between the computer generated objects and the real scene. Many videos illustrating these different points are presented here.

Registration

Previous results

We recently proposed an autonomous model registration system, capable of tracking an object, the model of which is known, in an image sequence. This system integrates tracking, pose determination and updating of the visible features. The heart of our system is the pose computation method, which allows us to compute the viewpoint from 3D/2D correspondences of various features: points, lines and free-form curves. Our method minimizes the reprojection error of the model features in the image. However, one of the limitations of this method originates in the spatial distribution of the model features: the reprojection error is likely to be large far from the 3D features used for the viewpoint computation. In the following example, the viewpoint has been computed using curves on the building in the background of the scene (the Opera). We add a computer generated car on the square which moves from the background to the foreground of the scene. As the car moves away from the Opera, the reprojection error increases and the car seems to hover. The mixing method allows us to obtain much better results.

mpg file

Result obtained by using 3D/2D only: the car seems to hover as it moves far from the Opera.
mpg file

Much better results are obtained by using the mixing method.

The mixing method

In order to improve viewpoint computation, we propose to use 2D/2D point correspondences between consecutive frames. Previous approaches attempted to recover the viewpoint from 2D/2D correspondences alone ; unfortunately, this approach turns out to be very sensitive to noise in image measurements. For this reason, points correspondences between frames are here used to provide additional constraints on the viewpoint computation: the viewpoint is defined as the minimum of a cost function which incorporates 2D/3D correspondences between the image and the model as well as 2D/2D correspondences of key-points that are automatically extracted and matched in two consecutive frames.

mpg file

Model free-form curves are tracked over the sequence (2D features are drawn in yellow, projections of the 3D corresponding features in green).
mpg file

Points correspondences between frames are used to provide additional constraints on the viewpoint computation (arrows join the previous image key-points to the current image corresponding key-points).
The computed viewpoints accuracy is assessed here. Once the viewpoints are known over the sequence, computer-generated objects can be added in the real scene. The following video shows a virtual car moving on the square.



mpg file



Resolving occlusions

Once the viewpoint has been computed for each frame of the sequence, the computer-generated objects can be added in the scene. However, as the virtual object can be occluded by real objects (see the statue in our example, whose 3D model is not available), we have to determine the visible part of the virtual object. To do this, 3D stereo reconstruction of the scene is performed in the area where the virtual object is to be incrusted. Ideally, depth should be computed at every point of the incrustation area. However, for sake of accuracy, we prefer to compute the depth only for features which are easily matched. Therefore, depth is computed for contour points and key-points. For each feature point, the estimated depth is then compared with the depth of the virtual object. Finally, the shape of the occluding object is determined from this set of points.



The set of contour points and key-points which stand in front of the computer-generated object are drawn in white and the occlusion mask is shown in gray.



mpg file

A flying helicopter passing behind the statue is added in the scene.


Registration with varying focal length

Some results obtained by using a varying focal length camera are now presented. The scene consists of a toy cottage standing on a calibration grid. The trajectory of the camera projected on the horizontal plane and the focal changes (computed by using the calibration grid and called "real" parameters in the following) are shown below. Note that a zoom in is followed by a translation along the optical axe, which is difficult to discriminate. Previous studies on zoom-lens cameras prove that the image transformation resulting from varying focal length can be described using an affine model with 3 parameters. For each frame of the sequence, the hypothesis of a zoom against the hypothesis of a camera motion is tested. When the hypothesis of a camera motion is retained, the camera viewpoint is computed by using the mixing method. Comparison between estimated and real camera parameters can be found below. The accuracy of the results is also visually assessed by the reprojection of the 3D model and virtual objects are added in the scene (see below).



Estimated (blue) and real (green) trajectories of the camera projected on the horizontal plane.



Estimated (blue) and real (green) focal changes.
mpg file

The quality of the intrinsic parameters is visually assessed by the good accuracy of the reprojection of the 3D model.
mpg file

Virtual objects are added in the real scene (a rough reconstruction of the scene which is automatically obtained from the estimated viewpoints is used to compute the shadows between virtual and real objects).


Gilles Simon
Last modified: Fri Apr 23 15:56:59 MET DST 1999