Google's Street View has become ubiquitous. It's one of the pioneering features of Google Maps, and it has become such an integrated part of our modern lives that we might take it for granted. Not long ago, looking around from an arbitrary point on a map would have sounded like science fiction, but today that's just another way to see where something is. Now Google's making that experience even better, by improving how Street View stitches together its elaborate 360-degree panorama images.
If you're interested in the long version, you can read Google's full post on the subject over at the Research Blog, and for a short overview and visual companion on our exploration, check out Google's four-minute video on the subject just below.
At its simplest, all of the panoramas in Street View are just a series of images thrown onto a single 360-degree canvas, but the secret sauce comes from precisely how they're all stitched together. Before now, Google was able to make some simple tweaks on images to get those panoramas to look a bit better, but the new tech in Google's reworked Street View panoramas is much better, and more complicated. Aligning each individual image to the one next to it might seem like an easy task at first, but there are a lot of variables to account for, and geometry isn't always ideal.
A visualization of parallax, by Booyabazooka - Parallax Example.png, CC BY-SA 3.0
For one, there's parallax. Google's Street View camera assembly is called a "rosette," and although all of the cameras are close together, they still don't occupy precisely the same point in space. That gap in the camera's different placements means that each captures slightly different geometry, so some objects may appear in slightly different positions. Combine that with overlap, and details in neighboring images may not align.
Camera rosette (left), and a visualization of the coverage of each camera (right) overlapping areas in dark gray
In addition to camera placement, there's also the question of timing. Since not all of the cameras are pointed at the same thing, you can't be sure that each will fire off with the same settings, and they may even be individually miscalibrated. As a result, individual frames from each camera in the rosette could have further distinctions.
Old method for assembling panoramas (left), new method (right)
This all means that individual photos captured all at once may not correctly line up. Before, they were just sort of blended together into a panorama anyway, resulting in weird lines and breaks between objects sometimes. But now, Google can fix it—at least, in some cases—with a new optical flow algorithm.
Optical flow is a complicated subject, but at its most basic in this particular application, it means looking at multiple frames to account for differences. Here, that means looking at overlapping sections of panorama images.
Two images with a shared border are analyzed, and corresponding points between the two are found. To make things easy Google "downsamples" these points for later use, decreasing their effective resolution for the purposes of alignment (resulting in the pixelated abstraction you see in the overlap above).
With these shared points, alignment offsets can be corrected, and a better transition between the two images can be arranged. Basically, it recognizes bits of the two different photos and moves them to fit on top of each other better.
That fixes the transition between two images, but it's not a full solution. Remember, we have an entire 360-degree panorama, so we have to make sure that moving one image for better alignment doesn't mess up how it might align with others. That's where the next step comes into play.
Now that we can align images better, we need to tweak the overall geometry of the scene and its constituent parts to make sure things still all fit together. We do this by "warping" the individual photos.
Google's particular method is complicated, there are a lot of factors it takes into account to prevent new issues from occurring. But basically, it just stretches and compresses bits of each individual image to get things to fit, using the downsampled points found during the optical flow process as references for proper alignment. This case-by-case stretching also helps to mitigate parallax in some instances.
This is very much the short version. There's a lot more to how it works, it gets quite technical. And it's still not quite perfect, either. Even with these new tools, the resulting panoramas might not line up precisely with real-world geometry or exactly as Google wants. But this system does a much better job aligning the parts of a panorama, and it works without requiring any new images.
In fact, Google is able to apply this new model to all its existing Street View data, and it's improving and restitching existing panoramas as we speak. Eventually, all preexisting Street View panoramas will have Google's new tech applied to them.
- Google Research Blog