- Particle, Wave and Cartoon Sketch
ACK: River Far sequences are from MIT Temporal Texture DataBase. The other sequences were taken by our group.
(click each sequence to see more details)
Natural scenes contain rich stochastic motion patterns which are characterized by the movement of a large amount of particle and
wave elements, such as falling snow,water waves, dancing grass, etc. We call these motion patterns "textured motion".
2. The Objectives and criterions of Textured Motion Learning.
The analysis and synthesis of such motion patterns are important for a variety of applications in both vision and graphics, and stimulate growing interest of the two communities.
Vision community:
sufficient and general (being alike & various)
effective (i.e. low dimensional)
Applications: motion segmentation, recognition, annotation, etc.
Graphics community:
realistic (fire in Cartoon ‘Shrek’),
stylish (sketch of grassland, captures the spirit of motion)
controllable (our fireworks take place at anytime and place)
Applications: Movie industry, computer gamesBy building up a generative model for the textured motion, we try to learn realistic motion patterns from real data, separate the motion dynamics with photometric and geometric styles, and thus achieve good controllability and interpretation of motion (by motons' trajectories, sources, sinks, force map, shadings, etc.).
3. A Generative Model of Textured Motion.
Generative methods combined with statistical models and algorithms are built up for both texture and motion analysis. The generative method includes the following four statistic models:
An image is represented as a superposition of linear bases using an over-complete dictionary, such as Gabor or Laplacian bases for particle objects, and Fourier bases for wave patterns. Such base representation is known to be generic for natural images, and it is low dimensional as the number of bases is often 100 times smaller than the number of pixels.
Each moving element (moton), such as the individual snowflake and bird, is represented by a deformable template which is a group of several spatially adjacent bases. Such templates are learned through clustering.
The motons are tracked through the image sequence by a stochastic algorithm maximizing a posterior probability. A classic second order Markov chain model is adopted for the motion dynamics. The sources and sinks of the motons are modeled by birth and death maps. We adopt an EM-like stochastic gradient algorithm for inference of the hidden variables: bases, motons, birth/death maps, parameters of the dynamics.
An symbolic representation for cartoon animation.
Given an observed image sequences I[0,t] as training data, we want to achieve two objectives:
- Infer those motons' cable structures, so that it makes the reconstructed sequence approximate to the observed one (phi).
- Fit all parameters of the motion model, so that it captures the spirit of those motons' motion style, birth and death (gamma).
We adopt the stochastic gradient algorithm as our learning process to approximate the global optimal 'theta'. It iteratively sampling the hidden variables, update the dynamics (gamma) and cable templates (phi). The computation is realized by data driven Markov Chain Monte Carlo techniques under coarse-to-fine scheme.
