Native Body Charge Playback. This text talks a couple of novel HDMI… | by Netflix Expertise Weblog | Jun, 2023
7 min read
Maximizing immersion for our members is a vital objective for the Netflix product and engineering groups to maintain our members entertained and absolutely engaged in our content material. Leveraging a great mixture of mature and cutting-edge consumer gadget applied sciences to ship a clean playback expertise with glitch-free in-app transitions is a vital step in the direction of attaining this objective. On this article we clarify our journey in the direction of productizing a greater viewing expertise for our members by using options and capabilities in client streaming gadgets.
If in case you have a streaming gadget linked to your TV, comparable to a Roku Set Prime Field (STB) or an Amazon FireTV Stick, you might have come throughout an choice within the gadget show setting pertaining to content material body fee. Gadget producers typically name this characteristic “Match Content material Body Charge”, “Auto alter show refresh fee” or one thing related. In case you’ve ever questioned what these options are and the way they’ll enhance your viewing expertise, preserve studying — the next sections cowl the fundamentals of this characteristic and clarify the main points of how the Netflix software makes use of it.
Netflix’s content material catalog consists of video captured and encoded in one among numerous body charges starting from 23.97 to 60 frames per second (fps). When a member chooses to observe a film or a TV present on a supply gadget (ex. Set-top field, Streaming stick, Sport Console, and many others…) the content material is delivered after which decoded at its native body fee, which is the body fee it was captured and encoded in. After the decode step, the supply gadget converts it to the HDMI output body fee which was configured based mostly on the capabilities of the HDMI enter port of the linked sink gadget (TV, AVR, Monitor and many others). Generally, the output body fee over HDMI is routinely set to 50fps for PAL areas and 60fps for NTSC areas.
Netflix affords restricted excessive body fee content material (50fps or 60fps), however the majority of our catalog and viewing hours may be attributed to members watching 23.97 to 30fps content material. This basically implies that more often than not, our content material goes via a course of known as body fee conversion (aka FRC) on the supply gadget which converts the content material from its native body fee to match the HDMI output body fee by replicating frames. Determine 1 illustrates a easy FRC algorithm that converts 24fps content material to 60fps.
Changing the content material and transmitting it over HDMI on the output body fee sounds logical and easy. In truth, FRC works nicely when the output body fee is an integer a number of of the native body fee ( ex. 24→48, 25→50, 30→60, 24→120, and many others…). Then again, FRC introduces a visible artifact known as Judder when non-integer a number of conversion is required (ex. 24→60, 25→60, and many others…), which manifests as uneven video playback as illustrated beneath:
It is very important observe that the severity of the judder is determined by the replication sample. Because of this, judder is extra distinguished in PAL areas due to the method of changing 24fps content material to 50fps over HDMI (see Determine 2):
- Complete of fifty frames have to be transmitted over HDMI per second
- Supply gadget should replicate the unique 24 frames to fill within the lacking 26 frames
- 50 output frames from 24 unique frames are derived as follows:
- 22 frames are duplicated ( complete of 44 frames )
- 2 frames are repeated thrice ( complete of 6 frames )
As a overview, judder is extra pronounced when the frequency of the variety of repeated frames is inconsistent and unfold out e.g. within the state of affairs talked about above, the body replication issue varies between 2 and three leading to a extra distinguished judder.
Now that we now have a greater understanding of the difficulty, let’s overview the options that Netflix has invested in. As a result of fragmented nature of gadget capabilities within the ecosystem, we explored a number of options to handle this subject for as many gadgets as attainable. Every distinctive answer leverages present or new supply gadget capabilities and comes with numerous tradeoffs.
The primary answer we explored and just lately enabled leverages the potential of present supply & sink gadgets to alter the outgoing body fee on the HDMI hyperlink. As soon as this characteristic is enabled within the system settings, gadgets will match the HDMI output body fee with the content material body fee, both precisely or an integer a number of, with out consumer intervention.
Whereas this appears like the right answer, gadgets that help older HDMI applied sciences e.g. HDMI v<2.1, can’t change the body fee with out additionally altering the HDMI information fee. This ends in what is commonly referred as an “HDMI bonk” which causes the TV to show a clean display momentarily. Not solely is that this a disruptive expertise for members, however the period of the clean display varies relying on how briskly the supply and sink gadgets can resynchronize. Determine 3 beneath is an instance of how this transition appears:
Enhancements within the current HDMI requirements (HDMI 2.1+) now permit a supply gadget to ship the video content material at its native body fee with no need an HDMI resynchronization. That is attainable via an modern expertise known as Quick Media Switching (QMS) which is an extension of Variable Refresh Rate (VRR) focused for content material playback situations. QMS permits a supply gadget to take care of a continuing information fee on the HDMI hyperlink even throughout transmission of content material with totally different body charges. It does so by adjusting the quantity of non-visible padding information whereas preserving the quantity of seen video information fixed. As a result of fixed HDMI information fee, the HDMI transmitter and receiver don’t must resynchronize, resulting in a seamless/glitch-free transition as illustrated in Determine 4.
HDMI QMS is positioned to be the perfect answer to handle the issue we’re presenting. Sadly, at current, this expertise is comparatively new and adoption into supply and sink gadgets will take time.
Aside from the above HDMI specification dependent options, it’s attainable for an software like Netflix to govern the presentation time stamp worth of every video body to reduce the impact of judder i.e. the appliance can current video frames to the underlying supply gadget platform at a cadence that may assist the supply gadget to reduce the judder related to FRC on the HDMI output hyperlink.
Allow us to perceive this concept with the assistance of an instance. Let’s return to the identical 24 to 50 fps FRC state of affairs that was coated earlier. However, as an alternative of excited about the FRC fee per second (24 ⇒ 50 fps), let’s broaden the FRC calculation time interval to three seconds (24*3 = 72 ⇒50*3 = 150 fps). For content material with a local body fee of 24 fps, the supply gadget must get 72 frames from the streaming software in a interval of three seconds. Now as an alternative of sending 24 frames per second at an everyday per second cadence, for every 3 second interval the Netflix software can determine to ship 25 frames within the first 2 seconds (25 x 2 = 50) and 22 frames within the third second thereby nonetheless sending a complete of 72 (50+22) frames in 3 seconds. This strategy creates a good FRC within the first 2 seconds (25 frames replicated twice evenly) and within the third second the supply gadget can do a 22 to 50 fps FRC which can create much less visible judder in comparison with the 24->50 fps FRC given a extra even body replication sample. This idea is illustrated in Determine 5 beneath.
NOTE: This answer was developed by David Zheng within the Associate Expertise Expertise staff at Netflix. Be careful for an upcoming article going into additional particulars of this answer.
Given the attainable options out there to make use of and the related advantages and limitations, the Netflix software working on a supply gadget adapts to make use of one among these approaches based mostly on components comparable to supply and sink gadget capabilities, consumer preferences and the particular use case throughout the Netflix software. Let’s stroll via every of those features briefly.
Each supply gadget that integrates the Netflix software is required to let the appliance know if it and the linked sink gadget have the power to ship and obtain video content material at its native body fee. As well as, a supply gadget is required to tell whether or not it could help QMS and carry out a seamless playback begin of any content material at its native body fee on the linked HDMI hyperlink.
As mentioned within the introduction part, the presence of a system setting like “Match Content material Body Charge” usually signifies {that a} supply gadget is able to this characteristic.
Even when a supply gadget and the linked sink can help Native content material body fee streaming (seamless or non-seamless), a consumer may need chosen not to do that by way of the supply gadget system settings e.g. “Match Content material Body Charge” set to “By no means”. Or they could have indicated a desire of doing this solely when the native content material body fee play begin can occur in a seamless method e.g. “Match Content material Body Charge” set to “Seamless”.
The Netflix software must know this consumer choice with a purpose to honor their desire. Therefore, supply gadgets are anticipated to relay this consumer desire to the Netflix software to assist with this run-time determination making.
Despite supply gadget functionality and the consumer preferences collectively indicating that the Native Content material Body Charge streaming needs to be enabled, the Netflix software can determine to disable this characteristic for particular member experiences. For instance, when the consumer is shopping Netflix content material within the house UI, we can’t play Netflix trailers of their Native body fee as a result of following causes:
- If utilizing Resolution # 1, when the Netflix trailers are encoded in various content material body charges, switching between trailers will end in display blanking, thereby making the UI shopping unusable.
- If utilizing Resolution # 2, sending Netflix trailers of their Native body fee would imply that the related UI parts (motion of cursor, asset choice and many others) would even be displayed on the decreased body fee and this may end in a sluggish UI shopping expertise. It is because on HDMI output from the supply gadget, each graphics (Netflix software UI) and video parts will exit on the similar body fee (native content material body fee of the trailer) after being blended collectively on the supply gadget.
To deal with these points we comply with an strategy as proven in Determine 6 beneath the place we allow the Native Body Charge playback expertise solely when the consumer selects a title and watches it in full display with minimal graphical UI parts.
This text introduced options that intention to enhance the content material playback expertise on HDMI supply gadgets. The breadth of accessible technical options, consumer selectable preferences, gadget capabilities and the appliance of every of those permutations within the context of assorted in-app member journeys signify a typical engineering and product determination framework at Netflix. Right here at Netflix, our objective is to maximise immersion for our members via introduction of latest options that may enhance their viewing expertise and preserve them absolutely engaged in our content material.
We wish to acknowledge the onerous work of plenty of groups that got here collectively to ship the options being mentioned on this doc. These embody Core UI and JS Participant improvement, Netflix Utility Software program improvement, AV Check and Tooling (earlier article from this staff), Associate Engineering and Product groups within the Consumer Engineering group and our information science mates within the Data Science and Engineering group at Netflix. Diagrams on this article are courtesy of our Associate Enterprise Platform XD staff.
