Ronen Basri: Research

Harmonic Representations of Lighting

 

Lighting can have a significant effect on the appearance of objects. By analyzing the effect of lighting we can hope to develop object recognition algorithms that that can overcome variations in lighting as well as shape reconstruction algorithms that make full use of the brightness values of the image. In recent years we proposed the use of harmonic representations to lighting. We showed that, as a function of the surface normal, the reflectance values of Lambertian surfaces are smooth versions of the lighting and can be captured by low order spherical harmonics. Consequently, the images of stationary Lambertian objects can be well approximated with more than 99% accuracy by nine (or even fewer) dimensional spaces spanned by the harmonics. These harmonic representations capture the appearance of objects under complex lighting that may include arbitrary combinations of point (directional) and extended sources.  Further applications of harmonic representations led to the development of photometric stereo methods in which both the 3D shape of an object and the lighting configuration are recovered from a set of images of the object obtained under different lightings as well as 3D reconstruction methods for objects in motion and fast indexing (pose estimation) under unknown lighting and viewing position (see further details on this in here).

What lies ahead? Harmonic analysis can model Lambertian surfaces under complex lighting. There is still however need to develop effective methods to deal with specularities, cast shadows, and inter-reflections.  Also, much work lies ahead in order to incorporate lighting analysis into common vision problems such as motion analysis and recognition.

Spherical harmonic basis for the space of images of a human face.

Photometric stereo with unknown lighting using harmonic representations.

The harmonic representations were first derived in an ICCV paper in 2001. A more comprehensive journal version has appeared in
                Ronen Basri and David Jacobs, “Lambertian reflectance and linear subspaces,” IEEE Transactions on Pattern Analysis and                 Machine Intelligence, 25(2): 218-233, 2003.

Additional analysis in the case of near lighting can be found in
                Darya Frolova, Denis Simakov, and Ronen Basri, “Accuracy of spherical harmonic approximations for images of                 Lambertian objects under far and near lighting,” European Conf. on Computer Vision (ECCV-04), Prague, 574-587, 2004.

The photometric stereo method was introduced in
                Ronen Basri, David Jacobs, and Ira Kemelmacher , “Photometric stereo with general, unknown lighting,” International                 Journal of Computer Vision, 72(3): 239-257, 2007.

Shape reconstruction of objects in motion in
                Denis Simakov, Darya Frolova, and Ronen Basri, “Dense shape reconstruction of a moving object under arbitrary, unknown                 lighting,” 9^th IEEE Int. Conf. Computer Vision (ICCV-03), Nice: 1202-1209, 2003.

And  indexing
                Ira Kemelmacher and Ronen Basri, “Indexing with unknown illumination and pose,” IEEE Conf. on Computer Vision and                 Pattern Recognition (CVPR-05), San Diego: 909 – 916, 2005.

Additional papers include
                Roman Dovgard and Ronen Basri, “Statistical symmetric shape from shading for 3D structure recovery of faces,” European                 Conf. on Computer Vision (ECCV-04), Prague, 99-113, 2004.

                David Jacobs, Peter Belhumeur, and Ronen Basri, “Comparing images under variable illumination,” IEEE Conf. on                 Computer Vision and Pattern Recognition (CVPR-98), Santa Barbara: 610-617, 1998.

 

The data used to evaluate our photometric stereo algorithm — both the photometric stereo images and the 3D laser scans of the objects — are available for download.