News for hackers. Stuff that matters.
Re:Old TV caused village broadband outages for 18 months by bayindirh on Tuesday September 22, 2020 @03:57AM (#24552033)
> that presumably shows up in the logs

Ah, no. SNR problems in the shared medium is not something wathced or logged. I remember losing internet connection on heavy rains sometimes because an odd cable shorted at an underground gallery. It wasn't FTTX then, so it was not easy to find.

> You have a problem that's repeatable right on schedule.

Similar problems rooted to bandwidth scarcity happened all the time, so in a village I'm not sure there's a lot of people who can discern the difference between, ah it's slow again and Oh, I lost connection, look at that SNR value. I better write a technical report to ISP.

> It should be a no-brainer to send someone with test equipment out there to measure things in the field while the problem was happening.

Even the simplest problem, if it's happening 1 in 10.000 is not a no-brainer because, you won't think that'd happen. Especially because of an old TV.

Re:Old TV caused village broadband outages for 18 months by Nextgrid on Tuesday September 22, 2020 @04:03AM (#24552071)
> SNR problems in the shared medium is not something wathced or logged

This is a problem then and should be addressed.

> problems rooted to bandwidth scarcity happened all the time

But the ISP should be able to tell whether their uplink is saturated (and actually upgrade it if it happens too often), so if they're consistently getting complaints around a certain timeframe and the uplink is not saturated during those timeframes then it must be something else.

> Especially because of an old TV

But the diagnostic procedure had nothing to do whether it's an old TV or something else. It was caused by EMI and is detectable with readily-available test equipment which indeed detected it immediately.

Furthermore monitoring packet loss at the PPP level can be done (in fact, Andrews & Arnold - a small ISP in the UK - does it) so if an ISP doesn't have this in place to detect widespread faults then something is also wrong there - why are they operating with zero visibility on their network?

Re:Old TV caused village broadband outages for 18 months by bayindirh on Tuesday September 22, 2020 @04:11AM (#24552119)
> But the diagnostic procedure had nothing to do whether it's an old TV or something else. It was caused by EMI and is detectable with readily-available test equipment which indeed detected it immediately.

They concluded that something is fishy because normal procedures didn't fix anything. I'm not sure that this is a problem they diagnose 5 times a week.

> But the ISP should be able to tell whether their uplink is saturated (and actually upgrade it if it happens too often), so if they're consistently getting complaints around a certain timeframe and the uplink is not saturated during those timeframes then it must be something else.

Good luck escalating this problem from a village to correct people in a big-shiny-ISP(TM). They're too busy(!) to look for 10 customers' problem. I accept that 18 months in unacceptable but, this is the reality we're in. Support and Ops teams are always overworked because management thinks that "the load just doesn't justify more hands in that department".

> This is a problem then and should be addressed.

The backbone operators and ISPs doesn't monitor SNR levels because they're transient and doesn't provide meaningful data. If you want to do it, it's millions of connections. Also, Not everyone use the standard equipment provided by the ISP so, they cannot get info from all devices (e.g.: from mine). So, endpoint ops are complex if you're working on a WAN with random equipment.

Instead, using bandwidth graphs to find problems is more practical and reliable. Since the problem starts at 07:00, it shows like "They're not using their connections much, eh?" in the said graphs.

Disclosure: My workplace also happens to be the national academic network operator. I'm two hops away from a country-wide core router.

Re:Old TV caused village broadband outages for 18 months by Nextgrid on Tuesday September 22, 2020 @04:25AM (#24552198)
> because normal procedures didn't fix anything

But why did it take 18 months to realize that? You've got an issue that occurs consistently and is preventing customers from getting the service they're paying for (and are presumably in contracts so they can't even cancel without a major fight).

> They're too busy(!) to look for 10 customers' problem

So we're in agreement that this is unacceptable and resolving this problem has been grossly mismanaged and someone should be held accountable?

> doesn't monitor SNR levels because they're transient and doesn't provide meaningful data

This seems weird to me. You've got a nationwide network designed to run on old, poor quality wiring (the article doesn't specify whether it's DSL or coax, but it would apply to both). Wiring issues should be expected and automated processes in place to monitor it and quickly detect issues like this one.

> Not everyone use the standard equipment provided by the ISP

Can't this be collected at the DSLAM or CMTS level? Or in case of DSL, even at the PPP level? (Andrews & Arnold - a small DSL ISP in the UK - measures packet loss at that level and while it's not true SNR, it would be enough for these purposes as localized packet loss would pretty much always mean bad SNR or failing equipment)

Re:Old TV caused village broadband outages for 18 months by donalhunt on Tuesday September 22, 2020 @07:18AM (#24553180)
I can guarantee that the customer reports came in as "my broadband isn't working again this morning...". And that's for the customers who called in.

Unless you had 50+% of the village calling up every morning for a week, it would not have been identified as a cluster event (and I suspect many ops orgs don't have the means to identify clusters anyway).

Re:Old TV caused village broadband outages for 18 months by brutusborn on Tuesday September 22, 2020 @06:30AM (#24552898)
What does the (!) after "busy" mean? P.s. Thankyou for sharing your experience, it's very insightful.
Re:Old TV caused village broadband outages for 18 months by nefitty on Tuesday September 22, 2020 @07:00AM (#24553055)
They’re probably using it to express incredulity.

“The mechanic told me it would cost $5000(!) to fix, so I decided to scrap the car instead.”

Re:Old TV caused village broadband outages for 18 months by tonyb on Tuesday September 22, 2020 @12:44PM (#24556731)
This isn’t accurate. The amount of data available on a DOCSIS network is incredible. Per modem SNR, power levels, preEq data, etc. If the data is used properly it cans find a loose screw on a tap or a damaged cable down the the foot.

That being said impulse noise is one of the harder things to track down. If the modern tools don’t work most MSOs would end up doing the “unplug a leg and see if the SNR gets better” method until they found the offending house.

Re:Old TV caused village broadband outages for 18 months by C1sc0cat on Tuesday September 22, 2020 @04:19AM (#24552159)
This is openretch (openreach) not Virgin so not sure where you get shared medium from

I do know people who worked in radio interference and they told me one time they traced a fault to a Budgie cage.

The Budgie had a swing and when it was in use it caused a short which effected the subs (customer) line.

Re:Old TV caused village broadband outages for 18 months by bayindirh on Tuesday September 22, 2020 @04:21AM (#24552171)
If you're running network over coax alongside cable TV, it's shared medium.

I'm not in the UK or US though, so regardless of their name, every ISP is equal and is indeed a bog standard ISP from my vantage point.

Re:Old TV caused village broadband outages for 18 months by iso1631 on Tuesday September 22, 2020 @04:49AM (#24552321)
Cable TV? Coax?

Most of the UK, especially outside of large towns, is

1) Twisted pair to the cabinet then Twisted pair to the exchange ("Normal DSL", ADSL)

2) Twisted pair to the cabinet then Fibre to the exchange ("Fibre broadband", FTTC)

3) Fibre to the cabinet then Fibre to the exchange (FTTP)

Cable TV is a minority thing in the UK, maybe 20% of TV owning households, compared with 50% on satelite of some sort, and 95% on terrestial.

Re:Old TV caused village broadband outages for 18 months by tialaramex on Tuesday September 22, 2020 @10:41AM (#24554928)
> 3) Fibre to the cabinet then Fibre to the exchange (FTTP)

FTTP doesn't bother going to a cabinet at all since there would be nothing interesting for it to do there. GPON splitters are low maintenance and so you can just throw one up on a telegraph pole or bury it.

Yes this means that an era of street cabinets gives way to an era in which it's just telegraph poles or the network infrastructure disappears entirely, buried in the street.

> Cable TV is a minority thing in the UK, maybe 20% of TV owning households.

This may well be, however over 50% of UK households could buy their Internet service from Virgin (the Cable TV company) and get DOCSIS over that coaxial cable, they needn't choose to buy television service (although of course Virgin would like them to do so)

Re:Old TV caused village broadband outages for 18 months by milankragujevic on Tuesday September 22, 2020 @04:36AM (#24552252)
BT openreach mainly does DSL and FTTH, AFAIK.

edit:

Just to clarify, FTTH is by definition a shared medium, but the huge capacity in practice means that there is little congestion usually.

Also, since glass fiber is a dielectric, it is difficult to get noise in the signal, and it is immune to EMI.

DSL is by definition a dedicated medium, having a twisted pair wire from the DSLAM at the central office or street cabinet to the user premises into the [splitter and] modem. One user per twisted pair only.

However, the wires are not shielded and thus are susceptible to both crosstalk from other pairs and also EMI given that it's using electrical signals to transfer data.

And of course, the speeds are low for most users due to the local loop length and in general VDSL2 35b maxes out at 300/100 Mbps at around ~80 meters, and VDSL2 17a maxes out at 100/50 Mbps at around 100 meters.

Re:Old TV caused village broadband outages for 18 months by LargoLasskhyfv on Tuesday September 22, 2020 @01:56PM (#24557710)
By that you mean GPON, I guess? Not so here, where it is PTP. And an ISP with excellent peering.
Re:Old TV caused village broadband outages for 18 months by milankragujevic on Tuesday September 22, 2020 @02:09PM (#24557901)
Edit: disregard my comment, I mistook PTP to mean Ethernet while in reality it's Point to point Fiber. In any case my point about noise still stands. But PTP Fiber is really rare in the world, given how many people are stuck on coax.

Yes, with FTTH (or FTTP) I meant GPON. Refarding PTP, do you mean a residential ethernet network? (Technically FTTB, copper / ethernet to the premises)

Branding anything other than FTTH as "Fiber" is misleading.

It's like saying DOCSIS 3.0 is Fiber because its a Hybrid Fiber-Coaxial network, while in reality all cable internet networks are HFC, the difference is the number of nodes serviced by a CMTS, which when reduced by a node split will result in higher speeds and more capacity.

Or saying DSL is "Fiber" because uplink to the DSLAM is Fiber. All DSLAM uplinks are Fiber, difference is how many street cabinets are there because DSL speed is dictated by the local loop length, and with more street cabinets there is higher chance one gets built closer to you enabling you faster speeds.

Re:Old TV caused village broadband outages for 18 months by LargoLasskhyfv on Wednesday September 23, 2020 @08:48PM (#24574137)
I meant a residential fiber network, where the fiber really goes into the home, and the topology of that network is point-to-point. As in from some POP(point-of-presence) from ISP into the home, not over shared channels like in GPON.

My ISP is rolling that out, though with delays. They are a spin-off of a public utility.

Already rolled out the CPE looks like this:

https://genexis.eu/product/fibertwist/

and in the user-forum of that ISP there are people describing how to directly interface to fiber with a pre-fabricated cable and SFP-transceiver with the right wavelengths, from a self-owned router with SFP-cage.

But you still have to connect over PPPoE with a MTU of 1492 bytes. Multiple times actually, on different VLANs for management, VoIP, and public dynamic IPv4(CGNAT)/IPv6(native/64)

Re:Old TV caused village broadband outages for 18 months by milankragujevic on Sunday September 27, 2020 @12:16PM (#24607753)
Understood. Sorry for the confusion.

RE: PPPoE - in general most AAA systems already work with PPP and since ISPs generally have to support legacy infrastructure too, and multiple access technologies, they "force" all users to connect with PPPoE for billing and data collection for identification.

For example, a national ISP in my country uses PPP for xDSL (which is a dedicated medium) and GPON (which is a shared medium, however the authorization and speed limiting is done by the ONT S/N), none of which "need it". You can also mix GPON and DSL logins and it will work, even between users.

The only logical reason for this, from what I can see, is that they are forced to sell access to their infrastructure (being an ILEC) to other ISPs, so they use PPP to route traffic from users of other ISPs from the BRAS to the corresponding network gateway using a MPLS tunnel.

For business fiber, not based on xPON, there is no requirement for a PPP session, and instead you get assigned an IP address range that you router should use, and the fiber is connected with a SFP+ module to your router. On the ISP's side, your "port" is limited to accessing their intranet for management and remote control, and your assigned IP address range.

Re:Old TV caused village broadband outages for 18 months by C1sc0cat on Tuesday September 22, 2020 @05:31AM (#24552574)
coax ? how quaint - there may be a small number of customers on actual coax but this is about Openreach part of BT's plant which will be POTS copper.

Unless they actually mean that tv blew away everyone's wifi, which is impressive but unlikely.

Re:Old TV caused village broadband outages for 18 months by lima on Tuesday September 22, 2020 @06:39AM (#24552940)
> Ah, no. SNR problems in the shared medium is not something wathced or logged.

A good ISP definitely does monitor SNR. It's a basic metric available from each individual customer device.

Re:Old TV caused village broadband outages for 18 months by bayindirh on Tuesday September 22, 2020 @09:45AM (#24554309)
> It's a basic metric available from each individual customer device.

Not mine. Good luck accessing to the modem you provided which I reconfigured to be a dumb bridge and terminated the ADSL connection on my router. Oh, the router neither see the POTS line (it only gets Ethernet) nor have remove management capabilities.

Where endpoints are heterogeneous, there's guaranteed, real fun (Even more so if you make assumptions).

Re:Old TV caused village broadband outages for 18 months by lima on Tuesday September 22, 2020 @04:23PM (#24559407)
They still have SNR metrics on their end of the connection.

And this is why providers hate heterogeneous networks :-) Mine politely asks you to plug in the original modem before they'll help you debug a connectivity issue. But even then, there are sometimes standards like TR-069 that apply across vendors.

Re:Old TV caused village broadband outages for 18 months by JosephRedfern on Tuesday September 22, 2020 @08:02AM (#24553487)
> Ah, no. SNR problems in the shared medium is not something wathced or logged. I remember losing internet connection on heavy rains sometimes because an odd cable shorted at an underground gallery. It wasn't FTTX then, so it was not easy to find.

Even if SNR itself isn't logged, dynamic line management would surely result in dropped speeds, which would be logged, and would stand out as anomalous?