One of the hardest video effects to fake is slow motion. It requires software to stretch out a clip by creating hundreds of non-existent in-between frames, and the results are often stuttered and unconvincing. But taking advantage of the incredible image-processing potential of deep learning, Nvidia has come up with a way to fake flawless slow motion footage from a standard video clip. It’s good thing The Slow Mo Guys both have day jobs to fall back on.
Slowing a video clip from 30 frames per second to 240 frames per second requires the creation of 210 additional frames, or seven in-betweens for every frame originally captured. Simply blending or morphing the before and after frames to create the new interstitials just isn’t enough to keep the motion as buttery smooth as real slow-mo footage appears. This is why slow motion in sports always looks far less cinematic than it does in the movies.
Plugins for high-end visual effects applications, such as RE:Vision Effect’s Twixtor, are able to improve the results of faked slow motion, but they require complex analysis of the motion in the clip, and often take hours to render. Nvidia has taken a different approach, and based on the results in the sample footage it’s released, an even better one.
Using a deep-learning AI that was trained on over 11,000 reference videos of slo-mo sports footage filmed natively at 240-frames per second, the neural network was able to predict how the 210 missing in-between frames were supposed to look, based on the preceding and following frames.
Smartphones and even high-end digital cameras are already able to capture slow-motion footage at these speeds, but as the framerate increases, the resolution drops due to the high bandwidth of data being created on the fly. Nvidia’s AI approach is a much cheaper alternative to dropping tens of thousands of dollars on a high-speed camera from the likes of Phantom because it all happens after the video is recorded.
The results aren’t as instant as with a high-speed camera – even with Nvidia’s high-end graphics processors powering the AI, it still needs time to process. But as smartphones become increasingly more powerful, eventually you’ll be able to fake stunning 8K slo-mo footage with a simple button click.
[Nvidia via Prosthetic Knowledge]