With its new Pixel 2 phones, Google introduced something called Fused Video Stabilization which fixes the so-called "terminator effect" the original Pixels suffered. It's a proprietary combination of optical image stabilization, electronic image stabilization, and the company's fancy machine learning tech. Its results are stellar, giving Google's Pixel 2 phones one of the best, if not the best, video capturing results out there. Since we know that it works well, now the burning question is: how does it work?
We could post examples of Google's Fused Video Stabilization all day, but suffice it to say that the results are unarguably an improvement over OIS or EIS alone. And they need to be. No matter how careful you are, unwanted movements are almost unavoidable unless your phone is either stationary or mounted on some prohibitively expensive hardware. But the new system in the Pixel 2 phones can alleviate a wide range of problems.
Undesirable camera effects
According to Google, there are four main issues caused by motion present in smartphone video that are corrected by Fused Video Stabilization: camera shake, motion blur, rolling shutter distortion, and focus breathing. The first two problems should be self-explanatory. Cameras, like the hands that hold them, shake when you record a video, and that's undesirable. Those movements can result in motion blur, or intermittent changes in sharpness during motion.
Global shutter simulation on the left, rolling shutter simulation on the right
The rolling shutter effect requires a bit more explanation, though anyone familiar with cameras probably already understands it. Basically, a digital camera takes a picture or video frame from top to bottom, and that requires a tiny bit of time. If movement is fast enough (or the process to capture a frame is slow enough) by the time the camera is recording the bottom of the image, it's in a slightly different physical position than it was at the top, resulting in a bit of stretching/distortion.
Focus breathing wasn't a term I was immediately familiar with, but most of us have probably experienced its effects. As focus changes, we can see a slightly wider or narrower angle of a scene. While we do want the focus to change, we don't want the background to be expanding and contracting when it does.
Thankfully, all four of these problems can be mitigated.
Fused Video Stabilization
A lot happens simultaneously while you record a video on the Pixel 2, so let's go through the complete process.
A flowchart for the Fused Video Stabilization video capture pipeline
To begin, OIS end EIS are used simultaneously while recording. Optical image stabilization (or OIS, for short) uses a mechanically suspended camera, allowing it to be quickly moved by electromagnets as a way of compensating for some motion. Electronic image stabilization (EIS) is also applied. EIS crops a video, sacrificing part of the frame as a buffer against movement. Each EIS system is proprietary and may work differently, but at its simplest, EIS tries to edit each frame to keep motion smooth. Sometimes, as in the case of the Pixel 2, EIS augments a visual recognition system with hardware data from a gyroscope or accelerometer.
That gyroscope and OIS lens motion data are captured with the frame. Frames are stored into a buffer, giving us a bit of time to analyze the data collected over multiple frames and allowing us to look at movement data to interpolate future camera motions based on a trained neural model in something called lookahead filtering.
Gaussian filtered video on left, stabilized lookahead filtered result on right
All these bits—the virtualized expectation of future camera motion, the original movement data, and the corresponding frame—are combined together with a few other optimizations (like some added motion blur to mask movements OIS can't remove and which EIS exaggerates) to create the ultimate result: a stabilized frame. Repeat this process as necessary for each subsequent frame that is recorded, and you get the silky smooth results of Google's Fused Video Stabilization.
There's a lot more to it all than that. This is very much the short version, but it should give you an idea for how it works, and perhaps provide a new appreciation the next time you fire up the camera app on your Pixel 2 to capture a quick video.
I strongly encourage the curious to read up on the full details behind how it all works. It does get a lot more complicated, and Google's Research Blog post on the subject is excellent and well-written.