News for hackers. Stuff that matters.
Re:Avoid FIFA World Cup CDN Issues by isostatic on Tuesday June 12, 2018 @12:21PM (#17294797)
What amazes me is the lack of concern on latency for web streaming.

We use the internet (and IP in general) to stream video. At high bitrates (200mbit+) we aim for sub 100ms end to end, for compressed services we're happy with 500ms, maybe upto a second if it's something like Sydney to London over the internet.

I was in a control room a couple of weeks ago watching some football. There were two displays, one end was the feed from the stadium, one was the feed from the web streaming service.

There were cheers and then groans from the live end of the room. nearly a minute later someone on the web end started running up the field to score. Of course I knew at that point that it wouldn't be a goal, as not only did the people watching the live stream tell me, but twitter was abuzz.

1 minute end to end delivery latency is shocking for this type of program. Heck 10 seconds is bad enough.

Re:Avoid FIFA World Cup CDN Issues by mv4 on Tuesday June 12, 2018 @12:34PM (#17294942)
You comment mentions two types of latency:

1. network latency, in milliseconds, that affects stream quality and stability

2. the delay (lag) between real-time capture and what the end user is seeing; this one is usually measured in seconds. A stream needs to be ingested, transcoded, sent from distribution servers to edge servers in each target region - with each step adding to the delay.

Minimizing the lag is very hard, because stripping all buffers (to reduce the delay) makes the stream very sensitive to network conditions (which reduces quality). With most commercial CDN providers you will get 5..10 seconds. It can be reduced to 2..3 sec if you know what you're doing.

edit: in case anyone is interested - in the second scenario, where we achieved 2..3 sec broadcast lag vs real-time, stream source (ingestion) was in the US, and the viewers were in mainland China. Network latency was over 600 msec. Wasn't easy!

Re:Avoid FIFA World Cup CDN Issues by isostatic on Tuesday June 12, 2018 @12:49PM (#17295113)
60 seconds in the case of iplayer for the fa cup. 1m20 in the case of BBC News channel right now. HLS tends to be packetised in something like 15 second chunks at the end of the process.

I know why there's a delay, I'm just amazed that people aren't concerned about it. The BBC used to offer multicast sources of live TV, which is a far more sensible solution, far more bandwidth efficient and allows end-to-end transmission in the satellite (or even less) range.

Wowza did a talk at demuxed last year about how to do "3 second latency end to end at scale", which I found amusing given that TV people have been doing sub millisecond latency at scale for nearly 100 years, so at least some people in the industry recognize the problem (which is mainly for sports events)

Re:Avoid FIFA World Cup CDN Issues by vdnkh on Tuesday June 12, 2018 @01:17PM (#17295454)
Hey, I was also at Demuxed last year! I help maintain Hls.js; it should have some form of LHLS support by soon(tm). There's no standard yet so actual adoption will be tough (current implementations use non-standard EXT-X tags to signal LHLS). But pretty much everyone does the same thing: early signaling of segments, chunked transfers, and on the client side, progressive demuxing. HTTP-based solutions typically achieve the aforementioned 3 seconds, but anyone talking about LHLS now (Akamai and Wowza) have their own protocols; it remains to be seen what the rest of us will get.

Twitch has recently implemented LHLS (looks like a "periscope-style" implementation) and I was seeing 1.2s glass-to-glass.

Re:Avoid FIFA World Cup CDN Issues by shittyadmin on Tuesday June 12, 2018 @01:10PM (#17295383)
I hate that I try to stream a game for my friends and they don't see what I did til 10-15s later. I'm trying to have discussions with them in real time on our voice chat server and any input they give me is inherently dated. Mumble gives extremely low latency, so achieving this kind of thing over the internet isn't exactly impossible, the bandwidth requirements of video would probably make it a bit more iffy, but should still be doable with some random issues.

Maybe I just need to get away from the public streaming services which use HLS, switch to UDP streams and sub-1s buffer sizes.

Re:Avoid FIFA World Cup CDN Issues by Deathmax on Tuesday June 12, 2018 @06:14PM (#17298567)
Twitch has introduced low latency mode for a while now, and I've seen sub-2s latencies.

https://help.twitch.tv/customer/en/portal/articles/2935005-l...

Re:Avoid FIFA World Cup CDN Issues by mv4 on Tuesday June 12, 2018 @01:08PM (#17295361)
As for what TV people have been doing - YES.

No matter what anybody says, the Internet just isn't well suited for massive live streaming events. That's what television is for!

Re:Avoid FIFA World Cup CDN Issues by mv4 on Tuesday June 12, 2018 @01:01PM (#17295271)
Honestly, I think people are concerned, but everyone's experience to date with big streaming events has been that something invariably goes wrong - people can't connect, login issues, event won't start, buffering, etc (e.g. the McGregor Mayweather PPV). If you can get a high-quality stream at all, perhaps the time shift is secondary!
Re:Avoid FIFA World Cup CDN Issues by isostatic on Tuesday June 12, 2018 @03:13PM (#17296881)
I only usually deal with the distribution side as an end user, and personally I tend to watch about 2 live events a year (I'd watch new years, but that's clearly pointless, so that leaves eurovision and maybe an election program), so I don't have much experience with that side.

It does amuse me when we were looking at latency for a program from a ropey bit of connectivity which we were using ARQ on. We were discussing whether we could push the latency up from 2 seconds to 6 seconds (it kept dropping out for 2 or 3 seconds at a time), as it's sport. Then we realised there was a good 30-40 seconds downstream before it even left to the CDN!

I still don't understand half of what Streampunk [1] are trying to do with their nmos grain workflows, but they are talking about sub-frame HTTP units

  This is not an approach that supports line-synced timing and may not be appropriate for live sports action that requires extremely low latency. 
  However, for many current SDI workflows that can tolerate a small delay, this approach is sufficient.
With UHD you're talking 20MBytes for a single frame, or each "grain" (a subdivision of a frame) being in the order of a millisecond/megabyte.

I think I prefer this approach to the SMPTE 2110 approach to be honest, especially with the timing windows that 2110 requires (it doesn't lead well to a COTS virtualised environment when your packets have to be emitted at a specific microsecond)

But I digress, this is all very off topic

[0] https://github.com/Streampunk/arachnid [1] https://www.streampunk.media/

Re:Avoid FIFA World Cup CDN Issues by mv4 on Tuesday June 12, 2018 @03:42PM (#17297243)
I am amazed that video delivery is not a solved problem still.
Re:Avoid FIFA World Cup CDN Issues by bradleybuda on Tuesday June 12, 2018 @12:56PM (#17295202)
In the US we've seen a half-dozen over-the-top TV providers launch in the past couple of years. They all seem to understand that live sports is their core feature, yet none of them tries to compete on latency. I regularly see two or more minutes from reading tweets about a goal or touchdown (from broadcast or cable viewers) until I see that TD live on Sling / PSVue / DirecTV NOW / YouTube TV (I've tried them all). Second-screening live sports is impossible with the OTT apps, and is very likely to drive me back to Comcast.

There's a real opportunity for a sports-oriented OTT company to compete on latency, a DVR that actually works, and expansive rights (I never have to guess if I have access to any sporting event).

Re:Avoid FIFA World Cup CDN Issues by mtberatwork on Tuesday June 12, 2018 @01:06PM (#17295318)
Doesn't "second-screening" while watching a sports event interfere with the act of watching the sports event in the first place?
Re:Avoid FIFA World Cup CDN Issues by NoNotTheDuo on Tuesday June 12, 2018 @01:30PM (#17295613)
For games like American Football, there's only 11 actual minutes of action, but it takes 3+ hours to complete the game[0]. That leaves a lot of time to watch a second screen.

[0]: https://qz.com/150577/an-average-nfl-game-more-than-100-comm...

Re:Avoid FIFA World Cup CDN Issues by isostatic on Tuesday June 12, 2018 @03:15PM (#17296912)
Second screening even in things like drama is very popular.

If you're watching a sports game, I can see that having a second stream (perhaps curated) with easy to access stats on that game, or a different angle to what the director thinks you want, or whatever.

I don't see the appeal of second stream in drama, but in things like sport, yes

Re:Avoid FIFA World Cup CDN Issues by mayneack on Tuesday June 12, 2018 @01:20PM (#17295492)
I stream a lot of sports from home. I would gladly trade delays for stream stability/quality. I frequently have to switch from a legal stream that I pay for to a more robust illegal stream.

It would be annoying if there were multiple devices nearby on different delays, but for me in my living room. I don't care if it's 30 seconds or 3 minutes. I've gotten spoiled by twitter feeds a few times, but it's not the end of the world.

Re:Avoid FIFA World Cup CDN Issues by jonknee on Tuesday June 12, 2018 @12:30PM (#17294889)
"Live" TV broadcasts are also on a delay, so I guess it's all relative. I had a friend who lived near enough to an NFL stadium that you could hear when something big happened and the TV delay made it impossible to enjoy a home game there.
Re:Avoid FIFA World Cup CDN Issues by isostatic on Tuesday June 12, 2018 @12:41PM (#17295026)
Very minor delay in the UK -- well under 2 seconds from pitch to TV on DTT in the case of the FA Cup. Sure enough to cause some grief when you can hear the crowd, but not enough to get notifications on twitter or whatever.
Re:Avoid FIFA World Cup CDN Issues by jonknee on Tuesday June 12, 2018 @12:44PM (#17295058)
They intentionally delay it ~7 seconds in the US because we're scared of an errant boob or curse word to make the broadcast.
Re:Avoid FIFA World Cup CDN Issues by Someone1234 on Tuesday June 12, 2018 @12:26PM (#17294850)
Interesting factoid, but some press have agreed to intentional latency as a security feature around certain events and dignitaries. The theory being that a sniper or drone operator cannot use the "live TV" footage to know their exact position.

In some cases international viewers may see the "live" footage before local ones.

Re:Avoid FIFA World Cup CDN Issues by Harvey-Specter on Tuesday June 12, 2018 @12:43PM (#17295048)
> The theory being that a sniper or drone operator cannot use the "live TV" footage to know their exact position.

How is this helpful in either of these cases? A sniper needs eyes on, a drone operator probably has live video from the drone. I don't see how a TV delay would have any effect.

Re:Avoid FIFA World Cup CDN Issues by newest_user_ on Tuesday June 12, 2018 @01:14PM (#17295425)
I'm going to assume the example was a bad one. I could see something like a remotely activated bomb leveraging live TV to inflict the most damage.
Re:Avoid FIFA World Cup CDN Issues by isostatic on Tuesday June 12, 2018 @01:36PM (#17295689)
So watch the international feed
Re:Avoid FIFA World Cup CDN Issues by _trampeltier on Tuesday June 12, 2018 @12:42PM (#17295043)
Already since over 20 years, the german TV chanel "RTL" is about 15 seconds later than all other TV chanels on live events like formula 1. But I have no idea why it is.
Re:Avoid FIFA World Cup CDN Issues by mv4 on Tuesday June 12, 2018 @12:49PM (#17295111)
There no technical reason for it to be that high. Perhaps I should reach out to them.
Re:Avoid FIFA World Cup CDN Issues by LunaSea on Tuesday June 12, 2018 @01:24PM (#17295535)
Luxembourgish channel