## CSC2515 :: Machine Learning :: Final Project## On Learning Surface Light FieldsYou can download a pdf version of the paper here.## IntroductionThe five-dimensional surface light field function
[Miller 1998]
represents the exitant radiance for all surface points in a scene. Its
value incorporates all light transport effects such as shadows,
reflection and refraction. In the past, tabulated or re-sampled color
values [Miller 1998] [Wood 2000]
have been used to approximate the surface light field function and the
exitant radiance for new views has been interpolated from the stored
values. ## Related WorkRelated work to our approach can be found in the Machine
Learning and the Computer Graphics literature. |

Table 4: Estimated color values for the GBF (left) and the VMBF (right) and correct image. The average error rates over all vertices are 0.0097 for the GBF and 0.0087 for the VMBF. |

Our work is limited in many respects. We want to address
this in future
work. First of all, the experiments described in the previous section
have to be repeated on much large data sets and scenes with more
complicated lighting environments. One question is how much the optimal
values for the number of basis functions and the necessary training set
size depend on the scene characteristics and if these can be estimated
efficiently, for example based on the variance of the color values in
the training set.

A crucial prerequisite for performing more experiments is to
investigate more efficient ways to obtain the optimal parameters. As
already mentioned, the convergence rate of the numerical minimizer we
used so far is not optimal, especially for the VMBF. Here, it has to
been explored if an adaption of a minimization technique to a function
defined on the sphere could provide a better performance. An
alternative would be to employ the EM algorithm for our problem.
Originally, we discarded this possibility as the literature does not
provide a consistent view if this could lead to improved performance
[Ungar 1995] [Orr 1998] but given the
current situation, we might consider this option again. In [Zickler
2005]
different other efficient techniques for obtaining the parameter of an
RBF are mentioned. It might be worth to explore these. Another idea is
to implement the optimization process on a graphics processing units
(GPU). The results of Hillesland et al. [Hillesland 2003]
and the advance in GPU technology suggest a speedup up to a factor of
20. The simplicity of the RBF network, given the number of basis
functions is constant and sufficiently small, suggests to also
implement the re-rendering on the GPU. Here, we expect real-time
performance even for complex scenes.

In the longer term more general questions regarding our approach
can be addressed. One is the investigation of other spherical basis
functions such as Spherical Wavelets [Schröder 1995]
or splines defined on the sphere [Wahba
1981].
Improvements might also be possible when relaxing the assumption that
the vertices are independent. For example instead of computing the
error per sample point one could compute the error for a neighborhood
in a rendered image. In [Green
2006]
it is denoted that interpolating Gaussians between the vertices instead
of the final colors leads to a significantly improved visual
appearance, similar to the difference between Gouraud and Phong
Shading. We think this can be adapted for our technique.

Another area of future work can be to move beyond the tabulation
of the spatial domain. Learning the whole five-dimensional surface
light field function function would be one possibility. Interesting
would also to employ the VMBF to learn the transfer function and
compare the results to [Green
2006].

Further work is also necessary to understand why we cannot see the
superiority of the VMBF over the GBF reported in [Jenison 1995].

We presented a learning technique for the surface light
field function
based on radial basis function networks. We showed that Gaussian basis
functions outperform the von Mises basis function for high numbers of
basis functions
but both perform equally well for ,
which has the most relevance in practice. Therefore, our particular
spherical basis function does not provide an advantage over basis
functions defined in cartesian space for approximating surface light
fields. Unfortunately, both, VMBF and GBF, are not capable to
reconstruct high frequency effects such as the sharp shadow.

Compared to existing techniques for surface light fields, our
approach has the advantage of a low memory consumption without
additional compression techniques. However, this comes to the prize
that we do not achieve the quality of previous techniques, such as
[Wood 2000],
for high-frequency effects.
The current approach is mainly limited by the training time for all
vertices of a scene and due to the limited amount of data used for this
report, no general applicability of our results can be
assumed.

Here, we present some additional graphs which could not be included in pdf version of the report.

Figure 1: Performance of RBF with VMBF for all samples of the hyperparameterspace exploration. Clearly visible are the clusters of vertices. |

Figure 2: Spatial distribution of the test error in the scene shown in Table 2. The region with the high error is the ground plane where the color value changes because of shadows. |

Figure 4: Estimated color value and ground truth for a cube vertex of the scene in table 1. The fall off of the surface for increasing color values is caused by the cosine-term of the shading model. |

Figure 3: Ground truth and estimated color value for the training data for one vertex in the ground plane in Figure 1. |

References:

- Arnold 1941
- K. J. Arnold,
*On spherical probability distributions*, PhD Thesis (unpublished), Massachusetts Institute of Technology, 1941 - Bishop 1995
- C. M. Bishop,
*Neural Networks for Pattern Recognition*, Oxford University Press, 1995 - Fisher 1987
- N. I. Fisher,
*Statistical analysis of spherical data*, Cambridge University Press, 1987 - Gortler 1996
- S. J. Gortler, R. Grzeszczuk, R. Szeliski, M. Cohen,
*The Lumigraph*, SIGGRAPH 1996, August 1996 - Green 2006
- P. Green, J. Kautz, W. Matusik, F. Durand,
*View-Dependent Precomputed Light Transport Using Nonlinear Gaussian Function Approximations*, Proceedings of ACM 2006 Symposium in Interactive 3D Graphics and Games, 2006 - Hastie 2001
- T. Hastie, R. Tibshirani, J. Friedman,
*The Elements of Statistical Learning*, Springer-Verlag, 2001 - Hillesland 2003
- K. E. Hillesland, S. Molinov, R.Grzeszczuk,
*Nonlinear optimization framework for image-based modeling on programmable graphics hardware*, SIGGRAPH 2003, August 2003 - Jenison 1995
- R. Jenison, K. Fissell,
*A Comparison of the von Mises and Gaussian Basis Function for Approximating Spherical Acoustic Scatter*, IEEE Transactions on Neural Networks, 6, 5, 1995, pp. 1284-1287 - Jenison 1996
- Rick L. Jenison and Kate Fissell,
*A Spherical Basis Function Neural Network for Modeling Auditory Space*, Neural Computation, Vol. 8, Issue 1, pp. 115-128, January 1996 - MacRobert 1948
- T. MacRobert,
*Spherical harmonics; an elementary treatise on harmonic functions, with applications*, Dover Publications, 1948 - Miller 1998
- G. Miller, S. Rubin, and D. Ponceleon,
*Lazy Decompression of Surface Light Fields for Precomputed Global Illumination*, Eurographics Workshop on Rendering 1998, June 1998 - Orr 1998
- M. J. L. Orr,
*An EM algorithm for regularized RBF networks*, International Conference on Neural Network and Brain, 1998 - Schröder 1995
- Peter Schröder, Wim Sweldens,
*Spherical wavelets: efficiently representing functions on the sphere*, SIGGRAPH 1995, September 1995 - Ungar 1995
- L. Ungar, R. De Veaux,
*EMRBF: A statistical basis for using radial basis function for control*, Proceedings of ACC 95, 1995 - Veach 1997
- Eric Veach,
*Robust Monte Carlo Methods for Light Transport Simulation*, PhD dissertation, Stanford University, December 1997 - Wahba 1981
- G. Wahba,
*Spline interpolation and smoothing on the sphere*, SIAM Journal on Scientific and Statistical Computing, 2, 5-16, 1981 - Wood 2000
- D. N. Wood, D. I. Azuma, K. Aldinger, B. Curless, T.
Duchamp, D. H. Salesin, W. Stuetzle,
*Surface light fields for 3D photography*, SIGGRAPH 2000, July 2000 - Zickler 2005
- T. Zickler, S. Enrique, R. Ramamoorthi, P.
Belhumeur,
*Reflectance Sharing: Image-based Rendering from a Sparse Set of Images*, Eurographics Symposium on Rendering. 2005, June 2005

lessig (at) dgp.toronto.edu | 2005/12/22 |