Ah, no. SNR problems in the shared medium is not something wathced or logged. I remember losing internet connection on heavy rains sometimes because an odd cable shorted at an underground gallery. It wasn't FTTX then, so it was not easy to find.
> You have a problem that's repeatable right on schedule.
Similar problems rooted to bandwidth scarcity happened all the time, so in a village I'm not sure there's a lot of people who can discern the difference between, ah it's slow again and Oh, I lost connection, look at that SNR value. I better write a technical report to ISP.
> It should be a no-brainer to send someone with test equipment out there to measure things in the field while the problem was happening.
Even the simplest problem, if it's happening 1 in 10.000 is not a no-brainer because, you won't think that'd happen. Especially because of an old TV.
This is a problem then and should be addressed.
> problems rooted to bandwidth scarcity happened all the time
But the ISP should be able to tell whether their uplink is saturated (and actually upgrade it if it happens too often), so if they're consistently getting complaints around a certain timeframe and the uplink is not saturated during those timeframes then it must be something else.
> Especially because of an old TV
But the diagnostic procedure had nothing to do whether it's an old TV or something else. It was caused by EMI and is detectable with readily-available test equipment which indeed detected it immediately.
Furthermore monitoring packet loss at the PPP level can be done (in fact, Andrews & Arnold - a small ISP in the UK - does it) so if an ISP doesn't have this in place to detect widespread faults then something is also wrong there - why are they operating with zero visibility on their network?
They concluded that something is fishy because normal procedures didn't fix anything. I'm not sure that this is a problem they diagnose 5 times a week.
> But the ISP should be able to tell whether their uplink is saturated (and actually upgrade it if it happens too often), so if they're consistently getting complaints around a certain timeframe and the uplink is not saturated during those timeframes then it must be something else.
Good luck escalating this problem from a village to correct people in a big-shiny-ISP(TM). They're too busy(!) to look for 10 customers' problem. I accept that 18 months in unacceptable but, this is the reality we're in. Support and Ops teams are always overworked because management thinks that "the load just doesn't justify more hands in that department".
> This is a problem then and should be addressed.
The backbone operators and ISPs doesn't monitor SNR levels because they're transient and doesn't provide meaningful data. If you want to do it, it's millions of connections. Also, Not everyone use the standard equipment provided by the ISP so, they cannot get info from all devices (e.g.: from mine). So, endpoint ops are complex if you're working on a WAN with random equipment.
Instead, using bandwidth graphs to find problems is more practical and reliable. Since the problem starts at 07:00, it shows like "They're not using their connections much, eh?" in the said graphs.
Disclosure: My workplace also happens to be the national academic network operator. I'm two hops away from a country-wide core router.
But why did it take 18 months to realize that? You've got an issue that occurs consistently and is preventing customers from getting the service they're paying for (and are presumably in contracts so they can't even cancel without a major fight).
> They're too busy(!) to look for 10 customers' problem
So we're in agreement that this is unacceptable and resolving this problem has been grossly mismanaged and someone should be held accountable?
> doesn't monitor SNR levels because they're transient and doesn't provide meaningful data
This seems weird to me. You've got a nationwide network designed to run on old, poor quality wiring (the article doesn't specify whether it's DSL or coax, but it would apply to both). Wiring issues should be expected and automated processes in place to monitor it and quickly detect issues like this one.
> Not everyone use the standard equipment provided by the ISP
Can't this be collected at the DSLAM or CMTS level? Or in case of DSL, even at the PPP level? (Andrews & Arnold - a small DSL ISP in the UK - measures packet loss at that level and while it's not true SNR, it would be enough for these purposes as localized packet loss would pretty much always mean bad SNR or failing equipment)
Unless you had 50+% of the village calling up every morning for a week, it would not have been identified as a cluster event (and I suspect many ops orgs don't have the means to identify clusters anyway).
“The mechanic told me it would cost $5000(!) to fix, so I decided to scrap the car instead.”
That being said impulse noise is one of the harder things to track down. If the modern tools don’t work most MSOs would end up doing the “unplug a leg and see if the SNR gets better” method until they found the offending house.
I do know people who worked in radio interference and they told me one time they traced a fault to a Budgie cage.
The Budgie had a swing and when it was in use it caused a short which effected the subs (customer) line.
I'm not in the UK or US though, so regardless of their name, every ISP is equal and is indeed a bog standard ISP from my vantage point.
Most of the UK, especially outside of large towns, is
1) Twisted pair to the cabinet then Twisted pair to the exchange ("Normal DSL", ADSL)
2) Twisted pair to the cabinet then Fibre to the exchange ("Fibre broadband", FTTC)
3) Fibre to the cabinet then Fibre to the exchange (FTTP)
Cable TV is a minority thing in the UK, maybe 20% of TV owning households, compared with 50% on satelite of some sort, and 95% on terrestial.
FTTP doesn't bother going to a cabinet at all since there would be nothing interesting for it to do there. GPON splitters are low maintenance and so you can just throw one up on a telegraph pole or bury it.
Yes this means that an era of street cabinets gives way to an era in which it's just telegraph poles or the network infrastructure disappears entirely, buried in the street.
> Cable TV is a minority thing in the UK, maybe 20% of TV owning households.
This may well be, however over 50% of UK households could buy their Internet service from Virgin (the Cable TV company) and get DOCSIS over that coaxial cable, they needn't choose to buy television service (although of course Virgin would like them to do so)
edit:
Just to clarify, FTTH is by definition a shared medium, but the huge capacity in practice means that there is little congestion usually.
Also, since glass fiber is a dielectric, it is difficult to get noise in the signal, and it is immune to EMI.
DSL is by definition a dedicated medium, having a twisted pair wire from the DSLAM at the central office or street cabinet to the user premises into the [splitter and] modem. One user per twisted pair only.
However, the wires are not shielded and thus are susceptible to both crosstalk from other pairs and also EMI given that it's using electrical signals to transfer data.
And of course, the speeds are low for most users due to the local loop length and in general VDSL2 35b maxes out at 300/100 Mbps at around ~80 meters, and VDSL2 17a maxes out at 100/50 Mbps at around 100 meters.
Yes, with FTTH (or FTTP) I meant GPON. Refarding PTP, do you mean a residential ethernet network? (Technically FTTB, copper / ethernet to the premises)
Branding anything other than FTTH as "Fiber" is misleading.
It's like saying DOCSIS 3.0 is Fiber because its a Hybrid Fiber-Coaxial network, while in reality all cable internet networks are HFC, the difference is the number of nodes serviced by a CMTS, which when reduced by a node split will result in higher speeds and more capacity.
Or saying DSL is "Fiber" because uplink to the DSLAM is Fiber. All DSLAM uplinks are Fiber, difference is how many street cabinets are there because DSL speed is dictated by the local loop length, and with more street cabinets there is higher chance one gets built closer to you enabling you faster speeds.
My ISP is rolling that out, though with delays. They are a spin-off of a public utility.
Already rolled out the CPE looks like this:
https://genexis.eu/product/fibertwist/
and in the user-forum of that ISP there are people describing how to directly interface to fiber with a pre-fabricated cable and SFP-transceiver with the right wavelengths, from a self-owned router with SFP-cage.
But you still have to connect over PPPoE with a MTU of 1492 bytes. Multiple times actually, on different VLANs for management, VoIP, and public dynamic IPv4(CGNAT)/IPv6(native/64)
RE: PPPoE - in general most AAA systems already work with PPP and since ISPs generally have to support legacy infrastructure too, and multiple access technologies, they "force" all users to connect with PPPoE for billing and data collection for identification.
For example, a national ISP in my country uses PPP for xDSL (which is a dedicated medium) and GPON (which is a shared medium, however the authorization and speed limiting is done by the ONT S/N), none of which "need it". You can also mix GPON and DSL logins and it will work, even between users.
The only logical reason for this, from what I can see, is that they are forced to sell access to their infrastructure (being an ILEC) to other ISPs, so they use PPP to route traffic from users of other ISPs from the BRAS to the corresponding network gateway using a MPLS tunnel.
For business fiber, not based on xPON, there is no requirement for a PPP session, and instead you get assigned an IP address range that you router should use, and the fiber is connected with a SFP+ module to your router. On the ISP's side, your "port" is limited to accessing their intranet for management and remote control, and your assigned IP address range.
Unless they actually mean that tv blew away everyone's wifi, which is impressive but unlikely.
A good ISP definitely does monitor SNR. It's a basic metric available from each individual customer device.
Not mine. Good luck accessing to the modem you provided which I reconfigured to be a dumb bridge and terminated the ADSL connection on my router. Oh, the router neither see the POTS line (it only gets Ethernet) nor have remove management capabilities.
Where endpoints are heterogeneous, there's guaranteed, real fun (Even more so if you make assumptions).
And this is why providers hate heterogeneous networks :-) Mine politely asks you to plug in the original modem before they'll help you debug a connectivity issue. But even then, there are sometimes standards like TR-069 that apply across vendors.
Even if SNR itself isn't logged, dynamic line management would surely result in dropped speeds, which would be logged, and would stand out as anomalous?
I worked at a UK ISP a couple years back and lets just say the 'monitoring' that ISPS have is extremely limited. Openreach (the company that manages the phone lines, which ISPs lease) almost certainly has more monitoring but the problem is the customer calls the ISP, and the ISP then tells Openreach - which means there's a fairly huge disconnect between the customer and the engineers fixing the line.
That's a choice. Presumably a cost conscious choice, not a limit imposed by Openreach.
A&A presents each customer with a live updated chart of bandwidth usage, latency, and low-level polling of the DSL modem in the customer's home. If your line drops even if you aren't "using" it for anything, you can opt to get a SMS about that, and choose whether this applies at weekends or overnight.
They even plot the rate of "login"/ "logout" events for the modems, both as a whole and mapped onto exchange locations so that you can see if there's a problem beyond just you got unlucky this time.
Since then the firmware on devices seems to have either dropped those types of features (cost reduction) or restricted them to an ISP-accessible interface ("security"). :(
From personal experience it probably took most of that time to escalate the issue high enough for someone to take it seriously, then a few visits to get the right people onsite.
It would have probably been quicker to migrate the village to FTTP
Relevant story - we moved in to a new build and waited 3 months to get ADSL. Not anything fancy, just plain old ADSL. The provider we went with was Talk Talk, and they basically had to go to Openreach and ask them to sort out the issue (turned out to be something to do with over capacity at the exchange.) This dragged on for months. I spent hours on the phone talking to someone in India, who then talked to someone in the UK who then got on to Openreach. It was a wholly terrible experience. I finally lost my proverbial s??t and complained directly to Openreach (and believe it or not actually got a response with the real reason for the delay - some infrastructure through which caballing went through had collapsed and they needed to schedule it to be fixed, but as it was under a dual carriage way, that was taking some time. Still possibly a lie, but better than the nonsense I was getting from the guy in India.) So as they had been lying to me constantly, I gave up and cancelled the contract, and we went with BT instead because we could then use the FON wireless of our neighbours in the interim. Well, one week later the ADSL came on. This was at least 2 weeks before it was scheduled by Openreach and BT, who were aware of the issue. So all we can assume is someone magically fixed it. To this day I don't know if BT and Openreach had some special relationship or if we were just lucky and BT juggled something in the exchange, but - yeah, 4 years later we have cruddy ADSL still, and no Fibre or Cable.
Sky and BT retail definitely pay for the top options of services.
What was the TV interfering with exactly? The ISP's equipment coming into the village?
I don't understand how you know it was simple. It's not clear to me what happened in the BBC article.
> You have a problem that's repeatable right on schedule
This is incorrect, the TV had to be turned on. Hyperbolic quotes on BBC like "It turned out that at 7am every morning " are not actually going to be true in the field.
> After 18 months engineers began an investigation after a cable replacement programme failed to fix the issue.
considering all that, why do they need to explain something took 18 months to fix that affected a few people again? Because I'm starting to think you must live in some sort of paradise I have been cast out of, and it's not fair!
You say that, but the San Antonio Caltrain to California Avenue Caltrain stretch still has an enormous T-Mobile deadzone, even after I twice reported it to them and both times they said "they'd send someone to take a look."
The first customer complaint was likely just one customer to begin with and could have reported that the issue was occurring at a different time of day depending on when they used the Internet.
It wouldn't have been until they had multiple complaints and eventually a user who could pinpoint losing connection at 7:00am that they looked at it being something external to the individual customers connection.
1. You have an issue with your broadband in a rural area. You call up your ISP. You get some handwavy answer about contention and they ask you to restart your router and call them back if it happens again.
2. The next time they agree to have an engineer take a look. You hear nothing back. You contact them again, they say their engineer looked at the exchange and found no issues.
3. The next time you ring them and they agree to send an engineer to your house. The engineer (contracted out) arrives at 2pm, they check your modem, rant a bit off the record about how they don't fund shit, then determines that everything's working fine at 2pm, and it's the ISP's problem that their network collapses at 6pm because they oversold the rate of your village.
4. You contact the ISP. They read the engineer's report, saw it was working fine at 2pm, and decided they didn't need to take action. They suggest using the internet at another time of day, such as when you and most of your neighbours are away from the house and at work.
5. If enough people in the area decide they complain they might finally deign to do something about their service in a few months. More likely is some other ISP comes around and signs up all the customers so pissed off with the last one, until most of the village is now on this ISP and the same problems reocur.
6. After like 4 or so years you've looped through all the ISPs, maybe there's a new generation of tech which promises higher speeds but they can't be bothered funding anything locally so your actual speed and reliability is unchanged.
You have a problem that's repeatable right on schedule, that presumably shows up in the logs and affects multiple customers. It should be a no-brainer to send someone with test equipment out there to measure things in the field while the problem was happening.