There’s a saying that the eyes are the window to a person’s soul, but according to a team of researchers from the University of Maryland, the eyes might instead be a mirror, providing enough data to reconstruct exactly what a person in a short video clip was looking at as an interactive 3D model.
A lot of factors made CSI, and its spin-offs, a popular franchise for CBS, but what drew many viewers back, week after week, was the over-the-top technology the crime scene investigators had at their disposal. A grainy, low-resolution frame of video captured in the middle of the night by a security camera could be repeatedly enhanced until it revealed the individual fibres on a suspect’s shirt. Most of the tools used to solve crimes on the show still only exist in the writers’ imagination, but researchers are making impressive advances when it comes to the amount of data that can be extracted from just a few frames of video footage.
Leveraging previous research on neural radiance field (NeRF) technology, where complex scenes or objects can be fully recreated in 3D using just a partial set of 2D images captured at various angles, and the fact that the shape of the eye’s cornea is more-or-less the same for all healthy adults, the University of Maryland researchers were able to generate 3D recreations of simple scenes based on imagery extracted from eye reflections. Don’t expect the results to be used to solve any crimes, though.
This approach, as detailed in a recently published study, comes with some very unique challenges. For starters, recreating 3D models from 2D images usually starts with high-quality source material, like video captured from a modern smartphone’s digital camera. Here, the reflections are extracted from a tiny, low-res portion of each frame, and they’re layered atop the complex textures of the eye’s iris, which also varies in colour from person to person, requiring extensive post-processing to clean up the imagery.
Further complicating this approach is the fact that the series of 2D images being used to create the reconstruction are all originating from the same location, with only slight variations introduced as the subject’s eye looks around. If you’ve ever tried to generate a 3D model of an object or a room using a smartphone’s camera and an app, you know you have to move around and record it from all sides and angles in order to get optimal results. But that’s not an option here.
As a result, the 3D models generated using this new technique are very low resolution and low on details. You can still identify objects like a plush dog, as the researchers did, but these results were also achieved using optimal conditions: very basic scenes, very deliberate lighting, and high-res source imagery.
When the researchers attempted to apply their approach to footage they weren’t responsible for capturing but was still created using ideal lighting conditions, such as a clip of Miley Cyrus from her Wrecking Ball music video sourced from YouTube, it’s impossible to discern what you’re looking at in the resulting 3D model. The vague blob is probably a hole in a white shroud the camera lens was looking through in order to achieve the specific look of this shot, but not even Grissom’s CSI team could confirm that using this 3D model. As fascinating as this research may be, it’s going to be a while before it becomes a tool that has any practical applications.