The Sins of 3D #2
For years, Side-by-Side (SBS) has been treated almost as if it were the natural format for stereoscopic 3D video. Download a 3D video and there is a good chance it will be SBS. Upload stereoscopic footage and SBS is often the first option people consider. Many 3D displays even include dedicated SBS modes.
But there is a fundamental problem with this idea:
SBS was never a proper 3D video format. It was a workaround for storing two views inside video formats designed for one.
And perhaps the biggest mistake was not inventing SBS in the first place. As a compatibility solution, it was clever, simple and useful.
The mistake was allowing that temporary solution to become permanent.
And this problem was not limited to video. Stereoscopic photography followed a remarkably similar path. Stereo pairs were commonly distributed as SBS JPEGs, even though formats such as MPO already provided a way to store multiple views inside a single image file. Today, Spatial HEIC takes that idea much further, showing how naturally stereoscopic information can coexist with an ordinary image.
The underlying problem is therefore bigger than SBS video itself: for too long, we have treated 3D as a special visual arrangement of two images instead of additional spatial information that a media file should understand and preserve.
Modern multiview codecs such as MV-HEVC demonstrate what stereoscopic video should have been all along: a single video file containing multiple properly identified views, rather than two images squeezed into one conventional video frame.
Had this philosophy become mainstream much earlier, the history of consumer 3D might have been very different.
SBS solves a problem by pretending there is no 3D
The elegance of SBS is also its greatest weakness.
A traditional video codec expects a sequence of rectangular frames. It has no inherent concept of a left eye or a right eye. SBS works around that limitation by taking the two stereoscopic views, resizing them and placing them next to each other inside a single conventional frame.
To the video codec, nothing special has happened.
It simply sees one image.
A 1920×1080 half-SBS video, for example, is technically just a conventional 1920×1080 video containing two horizontally compressed 960×1080 images.
The codec does not necessarily know that one half belongs to the left eye and the other to the right. The video itself does not inherently need to describe the relationship between them. A normal video player simply displays both images next to each other.
It is effectively stereoscopic information disguised as an ordinary flat video frame.
That was an ingenious compatibility hack when proper stereoscopic infrastructure was unavailable.
But a hack should not become the architecture of an entire medium.
We adapted 3D to video instead of adapting video to 3D
This distinction is important.
With SBS, stereoscopic video was forced to conform to infrastructure designed exclusively around a single image stream.
Instead of saying:
“This video contains a left-eye view and a right-eye view,”
we effectively said:
“This is one unusually shaped image. The display will have to know what to do with it.”
The intelligence was moved away from the file and into the user, player or display.
Someone has to know that the video is SBS. Someone has to tell the display that it is SBS. Someone has to know whether it is half-SBS or full-SBS. Someone may have to determine the correct left/right ordering. Software has to split the image and reconstruct the two views.
This is exactly the sort of technical friction that should disappear inside a properly designed media format.
Users should consume content.
They should not have to explain its internal structure to their displays.
SBS also created unnecessary compromises in image quality
Half-SBS became particularly popular because it fit easily into existing Full HD video pipelines.
But it achieved that convenience by sacrificing information.
Instead of storing two complete 1920×1080 views, half-SBS squeezes both into the same 1920×1080 frame. Each eye therefore receives only half the horizontal resolution before the display stretches the image back to its intended proportions.
Again, this was understandable as a bandwidth-saving compatibility measure.
What is harder to justify is how normalized it became.
Stereoscopic 3D is fundamentally about delivering two images. Treating those two images as though they must somehow fit inside the space allocated for one was always a compromise.
Full-SBS avoids that particular resolution loss by using a frame twice as wide, but at the cost of encoding substantially more image data and without efficiently exploiting the relationship between the two views. But the fundamental conceptual problem remains: two independent views are still being represented as regions of one artificial composite image rather than as two views belonging to the same video.
A real multiview format understands what the content actually is
MV-HEVC approaches the problem differently.
Instead of hiding two views inside one picture, the format can represent multiple related video layers or views within the same encoded bitstream. Often, it uses the same motion differences between keyframes to encode the second view at a much lower bitrate while preserving full quality, avoiding the near-doubling of bitrate required by Full-SBS.
That distinction changes everything.
The file can contain both stereoscopic views while preserving their identities. A compatible system does not need to infer that the left half of some giant image represents one eye and the right half represents another.
It knows that there are multiple views.
This also provides an elegant answer to one of stereoscopic 3D’s oldest and most ridiculous problems: inverted stereo.
Rather than relying on the physical position of images inside a composite frame, the appropriate views can be identified and selected for the left and right eyes.
The concept of left and right belongs in the media structure and playback chain — not in instructions that users have to remember.
One file can serve both 2d and 3D users
But arguably the greatest advantage of a proper multiview format has nothing to do with existing 3D enthusiasts.
It is what it can do for everyone else.
One of the most damaging consequences of SBS was that it effectively separated stereoscopic content from ordinary video.
An SBS video looks wrong on a normal 2d player. You see two squashed or adjacent images.
That immediately divides distribution into two worlds:
Here is the normal version.
And here is the special 3D version.
Different files. Different downloads. Different listings. Different workflows. Different audiences.
And that distinction helped condemn 3D to becoming a tiny niche.
With an appropriately authored multiview file, this separation does not have to exist in the same way.
A 2d-compatible playback path can simply use the primary view. The user can watch an ordinary image without needing to care that additional stereoscopic information exists in the file.
A compatible 3D system, meanwhile, can access the additional view and reproduce the stereoscopic presentation.
The same piece of content can belong to both worlds.
That is enormously important.
3D should have been an enhancement, not a separate ecosystem
Imagine if this had been the dominant philosophy from the beginning.
Instead of downloading Movie.mp4 and Movie_3D_SBS.mp4, there could simply have been:
Movie
A conventional display plays it normally.
A 3D display discovers that another view is available and presents it stereoscopically.
The 2d viewer does not need to know anything about SBS, Top-and-Bottom, frame packing or stereo modes.
They might not even know that the file contains additional stereoscopic information.
And they do not need to.
That is precisely how technology becomes mainstream: advanced capabilities should gracefully disappear when the hardware does not support them, rather than making the content unusable.
SBS did almost the opposite.
It made the existence of 3D impossible to ignore because the video became visibly incorrect without a device or player configured to interpret it.
We turned 3D from an additional capability into a separate type of content.
By now, the ideal situation would be for stereoscopic information to be just another capability embedded in mainstream video files whenever available. Owning a 3D display should simply determine whether you can take advantage of that additional information, much like owning a display with HDR support. The content itself should not have to belong to a separate 3D ecosystem.
The missed opportunity of MK3D
The Matroska ecosystem actually provided an opportunity to establish something better.
The .mk3d extension was created specifically for stereoscopic Matroska video. Matroska could also carry metadata describing stereoscopic video modes.
But by the time such ideas appeared, SBS and similar frame-compatible arrangements had already become deeply established.
The community had learned to think of a “3D file” as an ordinary video containing two images arranged within the frame.
So instead of moving decisively toward actual multiview video, much of the ecosystem continued packaging SBS content inside containers.
The container might know that the video was stereoscopic, but fundamentally we were still often storing the stereoscopic views using the same visual workaround.
The opportunity to redefine 3D video as something structurally integrated with ordinary video was largely missed.
Fortunately, 3D Blu-ray took the right approach. You don’t have to tell your TV which 3D format the movie uses or manually select how the two views should be interpreted. You simply insert the disc and press play, just as you would with any other movie. The player and the display already know what they are dealing with and handle the stereoscopic presentation accordingly. No formats to identify, no modes to select, no technical knowledge required from the viewer. That’s exactly how 3D should work.
The temporary solution became the standard habit
This is perhaps the most frustrating part of the SBS story.
SBS made perfect sense as a temporary compatibility mechanism.
Have a video pipeline that understands only ordinary frames?
Put both images in one frame.
Have a television that cannot receive two independent video streams through a particular playback chain?
Send an SBS image and let the television split it.
Need to upload stereoscopic footage to a service that has no idea what stereoscopic video is?
SBS gets both views through the system.
Excellent.
That is exactly what a workaround was for.
But somewhere along the way, the workaround stopped being treated as a workaround.
People built libraries around it. Distribution practices formed around it. Users learned it. Software was designed around it. Displays added manual SBS buttons.
Manual “3D modes” should never have been necessary
Think about how strange one of the most familiar 3D TV interactions actually was.
You start a video.
Two images appear side by side.
You pick up the remote control.
You open the 3D menu.
You tell the television that the two images currently visible on screen are actually a stereoscopic pair.
You select SBS.
The television splits them and sends the appropriate view to each eye.
For enthusiasts, this became completely normal.
But from a user-interface perspective, it is absurd.
The content itself should communicate what it is.
Imagine having to tell a television manually whether every movie is HDR. Or having to explain to an audio receiver that six channels inside a file are actually 5.1 surround sound.
Modern media systems work because formats describe their own content and compatible hardware interprets it automatically.
Stereoscopic video deserved the same treatment.
SBS also perpetuated the left/right problem
There is another consequence closely related to the problems created by cross-eyed stereo images.
When stereoscopic views are encoded according to their position inside a composite SBS frame, reversing the two halves reverses the perceived depth.
And such reversed files exist.
An experienced 3D user may immediately recognize pseudoscopic depth and swap the views if their software or display permits it.
A newcomer may simply conclude that the 3D looks uncomfortable or wrong.
A properly structured multiview workflow gives us the opportunity to move beyond this ambiguity.
Views can have identities.
Software can know what it is receiving.
Playback systems can select and route them appropriately.
We should not still be relying on whether somebody happened to put one image on the correct side of another image when creating a file.
We have even found scenes in commercially released 3D content where the left and right views are accidentally reversed, despite the rest of the video having the correct stereo orientation. In commercial content! This should simply be impossible in a proper multiview workflow. If each view is explicitly identified as belonging to the left or right eye throughout the production chain, editing software should always know which is which and prevent them from being accidentally swapped in the final render — even when working with source footage that arrives in different or opposite layouts.
The fact that such mistakes can reach a commercial release shows the fundamental weakness of treating stereo orientation as the physical placement of two images rather than as an intrinsic property of two properly identified views. Imagine watching a movie and suddenly finding that some scenes have inverted colors. That would be considered an unacceptable production error — and reversed stereo should be no different.
SBS helped create a 3D ghetto
The most important consequence, however, may have been cultural rather than technical.
SBS helped create a separate ecosystem for 3D content.
3D enthusiasts downloaded special 3D files, visited specialized 3D repositories, configured special playback software and learned terminology that ordinary viewers never needed to understand.
Meanwhile, mainstream users consumed ordinary video.
The two worlds became increasingly disconnected.
This matters because technologies spread more easily when their content remains interoperable with the mainstream.
If a video containing stereoscopic information can still circulate as an ordinary playable video, then 3D can travel wherever normal video travels.
Someone might receive the file today and watch it in 2d.
Years later, they might open exactly the same file on a 3D display and discover that a stereoscopic view was there all along.
A streaming platform could theoretically serve compatible content without maintaining an entirely separate conceptual category for users.
Creators could preserve stereoscopic information without making their videos inconvenient for the majority of their audience.
3D would become an additional layer of information rather than a fork in the road.
Photography already showed us another way
The same problem has existed in stereoscopic photography.
The simplest way to distribute a stereo photograph has traditionally been very similar to SBS video: place the left and right images next to each other and save the result as a conventional JPEG. JPS formalized the use of JPEG for stereoscopic imagery and could provide stereoscopic information, but in practice the underlying concept still commonly revolved around packing both views into a single composite image.
Again, this was convenient because practically anything could store, copy and transmit the file. But open such an SBS image without stereoscopic-aware software and you still see two images instead of one normal photograph.
Photography, however, also gave us MPO.
MPO demonstrated a much more sensible concept: multiple complete images could exist inside a single file instead of having to be physically combined into one artificial canvas. This was already much closer to what stereoscopic media actually is.
Unfortunately, MPO never became a universally transparent mainstream image format. Ordinary software and online services did not consistently preserve, interpret or display its stereoscopic information, and the format largely remained associated with dedicated 3D cameras and devices.
Today, the standardized Spatial HEIC format provides an even better example of the philosophy 3D should follow. It is not a proprietary Apple format: it is based on open industry standards that other manufacturers, platforms, and software developers can implement. Software or platforms that supports HEIC can show a Spatial HEIC as a conventional image, even if it does not understand or make use of its additional spatial information.
Telegram also supports HEIC, but to preserve the original resolution and Spatial information, images must be sent as files rather than photos — as with any other service that resizes or processes uploaded images, regardless of the original format. Telegram’s built-in viewer simply displays the primary view as a regular 2d image, while the Spatial information remains preserved in the file for compatible 3D software and devices. You can see this in action with original Spatial HEIC images shared in our privacy-friendly, exclusive Telegram community.
Support for HEIC continues to expand across conventional 2d devices and software, WebP Cloud allows HEIC files to be used on websites to be viewable with any browser, and WordPress added HEIC upload support in version 6.7 (is currently supported in Chromium browsers -Chrome, Edge, Brave- on Windows with HEVC support installed and on macOS, and in Safari on macOS), potentially bringing the format to millions of websites if browsers extend those existing HEVC decoding capabilities to displaying HEIC still images directly — something Safari and other browsers on iOS already do. Most importantly, compatibility with ordinary 2d workflows does not require sacrificing the spatial information: a conventional device can simply display the primary image, while the additional view remains embedded in the same file, ready for compatible 3D hardware to use. And the number of devices from various brands (Samsung, Acer, Pico, Meta, and more) supporting Spatial images inside HEIC files is also increasing.
You can see this compatibility in action with the image below. If your browser supports HEIC, it will load the original Spatial HEIC file directly, displaying its primary view as a regular 2d image while keeping the additional Spatial information intact. If HEIC is not supported, your browser will automatically fall back to a conventional AVIF version instead. Either way, you see the same photograph — the difference is that compatible software and 3D devices can extract much more from the original Spatial file. So, if you’re reading this article on Vision Pro or Pico 4 Ultra, you should be seeing the Spatial image below in all its 3D glory.
Same photograph, same page — but a richer file whenever your device can handle it.
A Spatial HEIC image does not need to announce itself to an ordinary user as a strange double photograph. Actual software can present and edit a normal image, without altering the additional data inside the file, while spatially aware hardware and software can access the additional information required to reproduce depth.
That is exactly the relationship 3D should have with mainstream media.
The ordinary user should not need a special copy.
The 3D user should not need a separate SBS version.
And the creator should not have to choose between making the file convenient for conventional users or preserving its stereoscopic information.
The 2d world does not need to make room for 3D; 3D can simply travel with it.
Think of 3D more like HDR or surround sound
This is perhaps the easiest way to understand what went wrong.
A movie with surround sound is not generally considered a fundamentally different movie file simply because your television only has two speakers.
The additional audio information is there for equipment capable of using it.
Likewise, modern video ecosystems can carry information that only compatible displays fully exploit.
This is the philosophy stereoscopic video needed.
3D should have been a capability of video, not a strange-looking video format of its own.
A multiview file can conceptually say:
“Here is the main view. And here is additional visual information that allows compatible hardware to reconstruct stereoscopic vision.”
SBS instead says:
“Here are two pictures squeezed together. Good luck.”
The difference in philosophy is enormous.
The MV-HEVC standard finally shows the right direction
MV-HEVC is not important merely because it is newer or more technically sophisticated than SBS.
It is important because it embodies the correct abstraction.
A stereoscopic scene contains multiple views.
So store multiple views as multiple views.
Keep them associated within one file. Preserve their relationship. Allow playback systems to access the view or views they need. Let compatible 2d workflows use the primary view while stereoscopic systems take advantage of the additional information.
No visually mutilated frame is necessary.
No manual SBS interpretation should be fundamental to the experience.
No separate SBS copy should need to exist simply because one audience has a 3D display and another does not.
And no user should have to understand how two images were physically packed into a rectangular frame before they can watch a video.
That is what a proper stereoscopic video architecture should look like.
We agree that legacy hardware and software compatibility is a real issue, but that is precisely why we should move toward proper standards instead of perpetuating workarounds such as SBS!
MV-HEVC is a standardized extension of HEVC, not an Apple-exclusive format (in fact, MV-HEVC was standardized in 2014, three years before Apple even introduced HEVC and HEIF support for ordinary 2d content with iOS 11), and other manufacturers and software developers can implement it as well. Samsung did. Acer did. AndroidXR did. Pico and Meta did. Apple adopting it does not make the underlying video codec proprietary to Apple; they simply found a better use for the format and helped make it more widespread than others did.
The fact that some applications may ignore or even discard additional views when rewriting a file is not an argument against multiview formats either; it is largely a consequence of stereoscopic metadata and additional views being niche features that many developers simply never considered worth preserving. If multiview video became commonplace, software would be expected to recognize those additional views just as editors already recognize multichannel audio. An editor does not silently assume that a 5.1 soundtrack must be reduced to stereo simply because two channels are enough for many users: the channels are identified, the software understands what they are, and the user decides whether to preserve them or deliberately downmix them. Additional video views should work in exactly the same way.
The solution to poor support for standardized multiview information is therefore not to keep disguising two views as one SBS frame forever, but to make proper multiview support common enough that preserving those views becomes an ordinary part of the media workflow.
HEVC is already far more widely supported than HEIC — ironically, the technology needed to decode HEIC is already part of the HEVC ecosystem. This means an MV-HEVC file can travel through the same channels as any ordinary 2d HEVC video, giving it the same opportunity to become widespread. Software that does not understand spatial video can simply play the primary view, leaving the additional view untouched; software that does understand it can unlock the full 3D experience. The extra view only needs to be discarded when the file is deliberately recompressed or flattened — just as a multichannel 5.1 audio track can be reduced to stereo 2.0 by software or by user choice. Until then, the depth can simply remain hidden inside the same file, waiting for a compatible 3D device to awaken it.
One file for everyone — with depth waiting for those who can see it.
Perhaps 3D would be more widespread today
It is impossible to know how different the history of stereoscopic 3D would have been if proper multiview distribution had become commonplace earlier.
SBS was certainly not responsible for all of consumer 3D’s difficulties. Hardware limitations, glasses, display brightness, inconsistent quality, industry decisions, content availability and many other factors played major roles.
But SBS may have contributed to a deeper structural problem:
It separated 3D video from ordinary video when the goal should always have been to integrate the two.
Once 3D became a special file that looked broken on a normal screen, it became easy for platforms to ignore it.
Once consumers needed separate 3D downloads, it became easy for distributors to stop providing them.
Once creators had to choose between publishing something universally playable and publishing something specifically for 3D users, 3D became the minority branch.
A proper multiview ecosystem could have reduced that divide.
The mainstream file would also have been the 3D file.
And that could have made stereoscopic information much harder to abandon.
Spatial is not only for Vision Pro. There are millions of devices from various brands that support it.
SBS was useful. Keeping it forever was the mistake
None of this means SBS should disappear completely.
It remains extremely useful for compatibility, testing, transmission through legacy systems and situations where software accepts only a conventional image or video stream.
Sometimes a workaround is exactly what you need.
But we should recognize it for what it is.
It solved the absence of a proper multiview video ecosystem by hiding two views inside one conventional frame. That was clever when compatibility demanded it.
Building decades of stereoscopic distribution around that limitation was not.
Today, formats such as MV-HEVC give us the opportunity to correct that historical direction.
The objective should not be to invent better ways of squeezing two views into video systems that pretend there is only one.
The objective should be to make video systems understand that a video can have more than one view.
One file.
One piece of content.
A normal experience on a 2d display.
A stereoscopic experience when suitable hardware is available.
No separate world for 3D users.
No strange double images for everyone else.
No need for the viewer to understand how the content is internally arranged.
SBS was a clever temporary solution to a technical limitation.
The real mistake was allowing the temporary solution to define what 3D became.
I have already taken that step with my own stereoscopic photography. I stopped saving my stereo pairs as SBS JPEGs and moved to the Spatial HEIC format. We had JPS, which adapted JPEG to stereoscopic imagery, and MPO, which already demonstrated that multiple views could live inside a single image file. Spatial HEIC now takes the concept where it always needed to go: a regular user simply sees the photo, while a 3D user can see that same photo with depth.
One file. Two experiences. No separate 3D version required.
This is how 3D photography should work. This is how 3D video should work. In fact, this is how 3D itself should have worked from the beginning: not as a separate category of media, but as an additional spatial dimension available whenever the display is capable of showing it.
I have already stopped treating my 3D photos as special SBS files and started saving them as Spatial images instead — preserving HDR tone maps, depth maps, and even Live Photo data when available, all within the same file. One file, all the information, ready for whatever the viewing device can make of it.
It’s time we did the same with video.