Bringing HDR video to Reels

• Meta has made it potential for individuals to add excessive dynamic vary (HDR) movies from their cellphone’s digicam roll to Reels on Fb and Instagram.
• To indicate commonplace dynamic vary (SDR) UI components and overlays legibly on high of HDR video, we render them at a brightness degree akin to the video itself.
• We solved numerous technical challenges to make sure a clean transition to HDR video throughout the various vary of outdated and new gadgets that folks use to work together with our providers daily.

Over the previous yr, the Video Infrastructure groups at Fb and Instagram have seen a big enhance within the quantity of HDR content material being uploaded to our apps, with tens of millions of HDR movies uploaded daily. Consequently, we’ve got been working to carry true HDR video assist to our household of apps, beginning with Reels.  At the moment, individuals now have the power to add HDR movies from their cellphone’s digicam roll to Reels and to have that video playback in full HDR . To make this potential, we wanted to beat just a few technical challenges to make sure a clean transition to HDR video throughout the various vary of outdated and new gadgets that folks use to work together with our providers daily. 

The journey to HDR might be higher understood if we have a look at how the introduction of shade tv, as an example, was a recreation changer, permitting viewers to look at applications in full shade, a vastly totally different expertise from the black-and-white broadcasting of the previous. This marked a step change in bringing video nearer to actuality, setting the stage for future developments, similar to high-definition (HD) content material, that proceed elevating the bar for the viewer expertise.

Now we’re on the daybreak of the subsequent era of video with the transition to HDR video. In contrast to SDR video, HDR video has a wider vary of luminosity and shade, which ends up in brighter whites, darker blacks, and a bigger potential variety of seen colours for extra vivid and true-to-life photographs. The rising adoption of HDR cameras and shows, particularly on cellular gadgets, permits a wider viewers to expertise its advantages. 

Problem: Differing machine assist for HDR

The rollout of HDR cameras and shows has resulted in an in depth vary of gadgets with disparate capabilities, which makes implementing end-to-end HDR video creation and supply difficult. It’s extra difficult than merely enabling HDR video creation and add for supported gadgets. From a product perspective, we have to make sure that HDR is accurately preserved by means of our complete media pipeline to offer a top quality person expertise all the best way from the creator’s machine by means of our server-side video processing/storage and, finally, view-side supply and playback.

One of many major challenges with having a variety of gadgets with numerous capabilities is guaranteeing compatibility throughout totally different gadgets. HDR video requirements have particular {hardware} necessities for each creation and consumption, and never all gadgets assist these new requirements. Which means some shoppers might not be capable of view HDR content material. Even worse, they could see a degraded model of the video with incorrect colours, leading to a much less satisfying viewing expertise.

HDR video additionally comes with the problem of various HDR codecs, similar to HLG and PQ, both of which can include HDR10+ or Dolby Imaginative and prescient dynamic metadata. Completely different machine producers might select to implement their very own requirements, with every having distinctive capabilities by way of the vary of luminosity and shade gamut they assist. Completely different sensors additionally end in totally different colours. These distinctions all end in further traits for us to account for to make sure that a given video seems persistently and accurately throughout all gadgets. For instance, if a video is encoded in HDR10 and performed on a tool with an 8-bit show, in some instances the video might look worse than if it had been an SDR video, with washed-out or unnatural-looking colours. If we didn’t handle this, it could be tough for creators to precisely showcase their work and assure the viewing expertise of their audiences.

Problem: Correct tone mapping

Since not all gadgets assist HDR shows, we have to present backward compatibility by providing an SDR illustration of the HDR video. Tone mapping is the method of cutting down the dynamic vary and shade house of a picture or video whereas aiming to protect the unique look of the picture. Improper tone mapping might trigger shade shifts that look unnatural, similar to bushes that seem blue. Tone-mapped movies could also be too shiny or too darkish. In knowledgeable studio, guide tone mapping or shade grading entails adjusting the parameters till the outcomes look pleasing. Nevertheless, we’d like an answer that may run mechanically on billions of movies while not having human evaluation with the tone-mapped SDR encoding intently resembling the unique HDR video.

Refined tone mapping algorithms can be utilized to protect high quality and decrease any artifacts that had been launched. Goal visible high quality metrics for tone mapping stay an open analysis drawback within the business, however based mostly on our inside testing, we had been in a position to tune the favored Hable tone mapping operator to supply outcomes that moderately characterize the creator’s unique intent.  

Tone mapping on the consumer

When creators first started importing HDR video content material, our media pipeline was not ready to deal with 10-bit colours. So HDR movies had overexposed colours, leading to a dissatisfying expertise. iOS gadgets had been the primary to allow HDR by default and, as such, had been the place we started to see HDR uploads coming from our customers. To mitigate this, we transformed all compositions containing HDR content material to SDR on the machine previous to add with client-side tone mapping. 

We used Apple’s native tone mapping APIs to rapidly launch a repair, mapping HDR movies to SDR shade house previous to importing to our servers. This manner all uploads had been assured to look good throughout all gadgets, even when they had been coming from newer telephones that captured HDR video.

On Android the story was a bit totally different. With a extra numerous machine panorama that lacked standardization for HDR in its early phases, we weren’t in a position to depend on OS-level tone mapping to keep up a constant look. Some gadgets had been creating HDR movies with the PQ switch operate, whereas others used HLG. These totally different requirements resulted in several representations of shade and, because of this, there was no one-size-fits-all resolution for tone mapping all Android HDR uploads into correct SDR representations.  

 Bhumi Sabarwal, a software program engineer at Meta, discusses the challenges round enabling tone mapping on Android. (from: Video @Scale 2022) 

For Android, we wanted to implement a deeper resolution. The early expertise for Android HDR Reels resulted in washed out colours for customers with newer gadgets. So we constructed customized tone mapping shaders to precisely convert each PQ and HLG HDR into correct SDR. First, we extracted the video metadata whereas decoding the frames to find out which switch operate was used (e.g., PQ or HLG). Subsequent, as soon as we had every body in YUV colorspace, we might apply applicable transformation matrices to transform into the goal SDR colorspace (BT.709). As soon as we had the SDR rendition, the remainder of the creation course of, together with artistic results, AR filters, and complicated media composition, had been in a position to operate appropriately.

With client-side tone mapping in place, we had alleviated the problem with washed out colours, however we had been nonetheless not delivering a real HDR expertise for these creators who had HDR content material. 

Tone mapping on the server

With client-side tone mapping, we had been in a position to mitigate the visible high quality degradation related to HDR video processed in a media pipeline that solely supported SDR. This fastened the problems we had been observing, however our final purpose was nonetheless to unlock the ability of newer gadgets with HDR shows and ship a full HDR expertise to them. This meant constructing strong end-to-end HDR assist whereas additionally supporting a satisfying person expertise for older gadgets that won’t assist HDR.  

As a part of the creation course of, we already transcode all uploaded movies into totally different resolutions and bitrates to offer a clean playback expertise for all gadgets and community circumstances with adaptive bitrate streaming (ABR). With HDR movies, nonetheless, this could get a bit extra difficult since they require 10 bits per shade part per pixel, and are sometimes encoded with newer codecs, similar to HEVC, VP9, or AV1. These traits enhance the decode complexity, and thus require increased efficiency gadgets to assist clean decoding and playback. If we delivered HDR content material to all gadgets, together with these with out sufficient assist, we might introduce degraded efficiency as the upper requisite bitrates end in wasted bandwidth, which results in elevated buffering, extra frequent in-play stalls, and decrease battery life.  

Due to this fact, to construct an expertise optimized for all customers, we have to have a technique to ship SDR encodings to gadgets that may’t make the most of the advantages of HDR. To deal with the problem of delivering HDR video throughout a various machine ecosystem, we constructed tone mapping into our server-side processing.

With server-side tone mapping, we add content material to our servers with the unique HDR shade info intact, and generate each HDR and SDR representations for supply.  Doing each HDR and SDR encodes doubles our compute for the HDR movies. Leveraging our Meta Scalable Video Processor for HDR processing, we’re in a position to deal with this load with out elevated power necessities.

If a tool doesn’t assist HDR, solely the SDR illustration will probably be delivered for playback. Moreover, the tone-mapped SDR variants of HDR movies are helpful for combined eventualities, just like the Instagram Explore web page or the Fb Feed, the place the person expertise is greatest with a uniform brightness throughout all thumbnails and previews.

Problem: Managing brightness

Inside our apps, we additionally confronted challenges round sustaining the consistency of the app expertise when introducing HDR brightness and colours within the context of the person interfaces designed round SDR video. This typically resulted in inconsistent or unsatisfying experiences in our testing.

As talked about above, HDR doesn’t solely increase the vary of colours, but in addition distinction by enabling increased ranges of brightness. This begs the query of how an already shiny show can change into even brighter to sufficiently accommodate the dynamic vary wanted for HDR, whereas additionally guaranteeing that SDR content material stays accurately and comparatively represented. On iOS gadgets, the default system habits is kind of fascinating. The brightest ranges of the show change into reserved for HDR content material, and the SDR content material truly turns into dimmer.

Inconsistent brightness when displaying HDR and SDR in a combined situation. The leftmost video within the third row is an HDR video and is way brighter than the opposite movies.