CVPR 2015 Tutorial on CVFX

The 2015 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR) will include a tutorial on Computer Vision for Visual Effects with several expert speakers from industry and academia. The tutorial will take place the day before the main conference begins, from 8:30 AM to 12:00 PM on June 7, 2015 in Room 105 of the Hynes Convention Center in Boston, Massachusetts. See more details at this page.

Videos now available

I’ve been lax on blogging, though I have a few topics in the wings… but in the meantime, I’m beginning to post Youtube videos from my Spring 2014 course at Rensselaer Polytechnic Institute based on the book, targeted at beginning graduate students and advanced undergraduates. Each roughly hour-long video recording covers 1-2 sections of the book and contains my voice over a screen capture of handwritten notes, figures from the book, running code on real images, and webpage/video views. Check them out at this page.

Visual Effects Overview

Here’s a talk I recently gave that overviews the field of computer vision and its applications to visual effects in movies and television:

The first 24 minutes are a general introduction to computer vision, why it’s difficult, and what kinds of problems computer vision researchers in academia and industry study. The rest of the talk overviews computer vision problems that are encountered in the design and production of visual effects, with lots of stills and videos from movies and TV. The main categories of problems (Matting, Image Compositing and Editing, Features, Dense Correspondence, Matchmoving, Motion Capture, and 3D Acquisition) parallel the chapters in my book.

Thanks to the RPI Cognitive Science Department for hosting and recording the talk.

Matchmoving Movie Titles

Matchmoving (also known as camera tracking) is the first step of any visual effects problem in which CGI objects must appear to “live in” three-dimensional space. Often this involves fully-CGI characters interacting with actual sets or background plates (e.g., Optimus Prime battling in Chicago, Dobby in Harry Potter’s room, the Hulk smashing up New York City). However, there are many more subtle applications of matchmoving; one striking effect that’s been used many times recently is the insertion of 3D text into the credits of movies. Below are two examples from movies that you wouldn’t necessarily associate with visual effects (Easy A and Panic Room):

For Easy A in particular, in addition to matchmoving there was also a fair amount of matting and compositing required; for example, every time a person passes in front of the text, the edges of his/her body had to be outlined or rotoscoped. This was also probably a tricky shot considering its length (although since a single CGI object isn’t continuously present throughout the shot, they might have been able to get away with estimating the camera track in pieces).

One of my favorite uses of this effect is from Stranger Than Fiction, seen below.

Actually, in many of these shots the camera is stationary or purely panning/zooming. In the first case, the compositor could probably eyeball where to put the floating text in 3D and tie it to the motion of a single tracked point (like the end of the toothbrush or a point on Will Ferrell’s body). In the second case (also known as a “nodal pan”) the background pixels in any two frames are related by a projective transformation, so drawn text in one frame of a shot can be pushed to other frames of the shot. However, if the camera’s moving, even a little bit, matchmoving is required.

Once you start thinking about this effect, you’ll notice it in many places. Other examples include Zombieland, Watchmen, Scott Pilgrim, and Fringe. Even local law firm commercials! The same idea is also involved with inserting CGi objects into sports broadcasting; see the earlier post on visual effects for the Olympics.

(Not entirely related, but Art of the Title is a pretty cool site.)