When it comes to live video, it’s hard to deliver an actual, real live streaming experience. The latency problem is especially noticeable when video is distributed using online streaming platforms : you may have heard the stories of watching a sports event and hearing the neighbours cheer before seeing the start of an attack. In this article, we’ll go into details of the cause of the latency observed between an action being recorded and shown on the screen, and how it can be reduced to provide a better viewer experience to your viewers.
Latency is often described as “the time between cause and effect of some physical change”. Within a streaming environment, it is most easily experienced by measuring the time when something is recorded in real life and when it is seen on the screen, for example waving your hand in front of a camera and watching it move on the screen a certain time later.
Having a high latency on streams , and especially in interactive live experiences, can have a significant impact on user experience . Imagine a live stream with a latency of about 20s with an interactive chat box where the audience can ask questions: this would mean the person starring in the video could perform some action at a certain time T0. This image would show up on viewers’ screens 20s later. During this time, the broadcaster would simply continue with whatever he is doing. However, if a user would then type in and ask a question about the action at T0, showing up on the broadcaster’s screen, the broadcaster might be doing something completely different already, and have lost context. In return a (wrong) answer would still take another 20s before the answer gets back to the viewer!
An end-to-end media pipeline is complex and consists out of a range of components, each attributing to latency. Depending on the different components, and how this pipeline is configured, latency can be impacted significantly. Because the audio and video information needs to pass through this pipeline, it is prone to accumulate latency while passing through the components. When we would look at a common architecture, we would see there are a few big components contributing to this latency :
When looking at latency, the impact of configuration of the different components in the media pipeline can be of utmost importance. Often configurations cannot be changed haphazardly, but have to be carefully considered depending on the business requirements. There are also a number of other factors at play when deciding this configuration , being scalability and quality.
While new technology and compression algorithms will shift the achieved results , and make lower latency ever achievable, finding the right balance within this latency trade-off triangle will always be important. Impact of technological improvements can already be seen in the advances made by cloud computing which significantly reduced scaling challenges, as well as new media codecs such as VP9, HEVC and AV1 , reducing the required bandwidth significantly compared to older techniques.
Specific customer use cases are usually the best way to decide where on the latency trade-off triangle one should end up. When latency is crucial, for example for video conferencing, or video security monitoring, often scale or quality can be traded in. On the other side, when doing mass delivery of high quality media content in linear broadcast, latency can often be increased slightly. Ideal positions on this latency trade-off triangle will depend heavily on the targeted use case . In most situations, the position can however be shifted by configuration of the different components in the media pipeline.
We already made the mention of measuring latency between camera and screen by waving your hand and counting the delay. This technique was also at the base of the name “hand-waving latency” which is often used to describe latency. A similar explanation can be found for the name “glass-to-glass latency”, referring to the glass lenses of the camera and glass in the screen of your monitor, or “end-to-end latency”, referring to the start (capture) and end (display) of the media pipeline. However, there are a number of other factors which often influence latency, or are perceived as latency: join latency and switch latency.
While join and switch latency are not latency in the end-to-end kind of way, they do impact user experience, and in some configurations are actually a part of the end-to-end latency as a whole.
While optimising latency is surely important, a question we often get is “How low should your latency be?”. The answer will depend on the business case. In general, we advise our customers to make a number of changes to reduce latency already to be within the 18-30s range. This can be achieved with minimal effort and cost. For use cases where latency is crucial, it is interesting to have a look at recent advancements in protocols such as the newer Low Latency CMAF of Chunked CMAF standards being implemented by vendors. In case a business case requires ultra low latency, or real time latency, solutions such as RTMP or WebRTC are to be employed with a high cost when scaling and increasing quality.
There is a broad scala on approaches and protocols being delivered to replace existing HLS and MPEG-DASH based pipelines in an attempt to reduce latency. There are a number of different types of solutions in play. While some solutions try to focus on first-mile protocols, others focus more on the last-mile protocols. For these protocols, the key focus points are:
Keeping these in mind, there are four main approaches observed in the industry:
In a next article we will go deeper into Low Latency CMAF and Chunked CMAF. Feel free to subscribe to our blog and receive an update when the new article is published.
As a frontrunner in player technology, THEOplayer already supports various low latency approaches such as Low Latency CMAF and LHLS and is continuously working to reduce latency and improve user experience. For any further questions on reducing latency, don’t hesitate to reach out to our team : they would be more than happy to guide you towards an optimal solution for your use cases.