Fusion and regularisation of image information in variational correspondence methods

Resumen

In this work we study, and improve, applicability of variational correspondence methods, used for calculating dense optical-flow and disparity fields, to real scenes with realistic illumination conditions. It is well known that under realistic illumination conditions and image noise, proper image representation is crucial in order to generate both correct and temporally coherent optical-flow and disparity fields. We have studied 34 different image representations and ranked these with respect to accuracy, robustness and a combination of accuracy plus robustness. In the case of well known test images with optimal (or quasi optimal) illumination conditions, effects of image representation are not that important. On the other hand, with scenes from real applications, such as robotic grasping or vision in automotive industry, influence of image representation is crucial. Also we have extended the basic models to include both spatial- and temporal-constraints. In the case of optical-flow, for example, temporal constraint reduces `flickering' of estimations. By flickering we mean temporal changes in the displacement fields due to lack of or ambiguity of spatial features in the images. We show that by using spatial constraints in the disparity estimation, considerable improvements are possible. These constraints are due to (a) what we know of the solution before hand (e.g. roads are relatively flat surfaces, sky is far away) or (b) what we can deduce from the scene itself. Effectively, we show how these constraints can be obtained and refined in a hypothesis-forming-validation loop (HFVL). In the example that we give of a HFVL loop, we segment an initial disparity map and form constraint(s) based on the segments and feed back these into the disparity calculation. Not only do the disparity estimations improve, but we also segment the depth map into meaningful surfaces Apart from introducing the principal results obtained, we also explain in detail how the models that we have used can be solved. Therefore, this work (along with the related Matlab/MEX code available at \url{http://atc.ugr.es/~jarnor/}) will hopefully help other scientists understand and further improve the variational methods for calculating stereo-disparity and optical-flow.