Moisture in copper cables is what slowed me down too. It was in a section up the road from me. However now that fibre is installed, it’s glorious and works in the rain.

Having had to debug many of such cable issues in the past, it's baffling to me that cable companies aren't proactively monitoring for things like this.

They have all the data available on their end, as far as I can tell! (Unless DOCSIS modems somehow don't have a standard "signal receive report" functionality?)

Telcos used to monitor their copper outside plant for moisture. This was called Automatic Line Insulation Testing in the Bell System. The ALIT system ran in the hours before dawn. It would connect to each idle line, and apply, for tens of milliseconds, about 400 volts limited to very low current between the two wires, and between each wire and ground, measuring the leakage current. This would detect moisture in the cable. This was dealt with by hooking up a tank of dry nitrogen to the cable to dry it out.

Here's a 1960s vintage Automatic Electric line insulation test system at work in a step-by-step central ofice. [1] Here's the manual for automatic line insulation testing in a 5ESS switch.[2] 5ESS is still the major AT&T switch for copper analog phone lines. After that, it's all packet switching.

For fiber, of course, moisture doesn't affect the signal.

This led to an urban legend: "bell tap". While Western Electric phones were designed to not react to the ALIT test signal, many cheap phones would emit some sound from the "ringer" when the 400V pulses came through, some time before dawn.

[1] https://www.youtube.com/watch?v=Wt1GGdDa5jQ

[2] https://www.manualslib.com/manual/2755956/Lucent-Technologie...

Great comment, thanks!

(I've sent a quick email suggesting it be added to https://news.ycombinator.com/highlights :)

If you're really into telephony history, the Internet Archive has "The History Of Engineering and Science in the Bell System" (3 volumes) online.

If you have to build reliable distributed systems, it's worth understanding how this was done in the electromechanical era of telephony, where the component reliability was much worse than the system reliability. "Number 5 Crossbar"[1] is worth reading, but hard to follow if you have no idea how telephone switching worked and are unfamiliar with the terminology.

Number 5 Crossbar, in current terms, was a collection of microservices. There was a big, dumb switch fabric, and "markers" which told it what to connect. Other microservices included trunks, originating registers (which listen to incoming dial digits), senders (which sent dial digits to the next switch), billing punches (which recorded toll call data for later billing), translators (which held routing tables), and trouble recorders (which logged errors.) Central offices had at least two of each resource, for redundancy. Resources were "seized" as needed from resource pools, with a hardware timeout and alarms to prevent resource lockup. If something went wrong in setting up a call, it was retried once, using different resources. If it failed on the second try, the caller got a fast busy and there was an alarm and a trouble recorder dropped a trouble card. Markers did not have persistent state. They started each call with a reset. So they could not get stuck in a bad state.

In the entire history of the Bell System, no electromechanical switching office was ever down for more than 30 minutes for any reason other than a natural disaster or a fire. It's worth understanding how they did that.

[1] https://telephoneworld.org/mdocs-posts/number-5-crossbar-sys...

Not truly related to the post content, but there is something about the way these old manuals are formatted/printed that immediately inspires confidence in the contents.

Maybe because you know that someone spent a lot of time on it before it was published since no adjustments could be made after the fact.

> trouble recorders

This feels like a term a sci-fi author would invent in an alternate history setting to replace "error log" and I find it very humorous.

No, just practical.

The previous version was a panel of blinking lights called the "trouble indicator". When an alarm sounded, someone had to go to the panel and record by hand which lights were on. There were about 200 lights. So the trouble recorder, which recorded that info automatically, was added in larger central offices as an upgrade.[1]

[1] https://hackaday.com/2022/12/02/stack-trace-from-the-1950s-p...

> Wait! Was that an old adding machine?

At 02:40.

And yes, it is an adding machine.

Ah, so that's why there were always nitrogen tanks on NYC sidewalks.

Yup, here's a tom scott video on the very same: https://www.youtube.com/watch?v=juZqGU9iuq0

We still have a land line. When a call comes through the phone often gives a gentle "peep", then a pause, then goes full-on ring. I've started to react to the "peep".

But every evening, mostly around 21:00 or so, the phone gives a gentle "peep" without then ringing.

I wonder if it's a line test?

Crap electronic ringer, probably. If you put a scope on the line, you should be able to see what's happening. Remember to be prepared for higher voltages, up to 400V.

There are various weird, obsolete signals in analog phones. Ring pulse alerting signal. ALIT test. Polarity reversal. Ring to ground. Ground start. Caller ID (1200 baud FSK between the first and second rings) DSL. Basic talk and ring was standardized around 1900, and everything else is backwards compatible. Ringers are supposed to ignore all that stuff. People who implement Asterisk PBXs are into this.

Here are some actual waveforms, if anybody cares.[1]

[1] http://www.adventinstruments.com/Products/AI-5120/Screenshot...

In my observation, to a first approximation, cable operators take off-the-shelf equipment, connect it, power it on, and bill customers for it. They don't really have the r&d capability to innovate and create new monitoring solutions quickly.

It might happen that an equipment manufacturer sees an opportunity and builds something, but then they have to go into a long sales cycles to convince operators to use it. Operators are in a duopoly situation in most places, so quality of service is kind of a secondary concern for them - customers may get annoyed, but as long as the competition is not vastly superior, few actually switch. It is not a market prone to innovation.

on dslreports or broadbandreports there's at least two instances of me complaining about two cable companies because, at last, it was figured out there was moisture ingress in the LE (line extender, usually on cable lines on poles). The only common denominator was it happened during prime time, every night, and went away around midnight.

The other common denominator was the cable company refusing to believe it was an issue with their equipment; this meant it took a couple of months of calling them every night until they finally sent a technician and a manager to my house to verify that I wasn't wrong, leaving my house, coming back 15 minutes later to say "it'll be fixed tomorrow, there's a problem with the LE balance up the road" - and then the issue is resolved.

Now this doesn't sound so bad, until you learn that the first time this happened to me, i had only VoIP - so the internet would start to foul, i'd call the cable company, and the tier 1 would reset my modem at some point, and then i wouldn't be able to call back until after midnight (or whatever), when there was no longer a problem. So after a week of this, i would walk 30 minutes - one way - to a pay phone (remember those?) once the internet slowed, call them, explain that i couldn't do anything they wanted me to do physically, since they disconnected my phone line every time i called.

This is what happens with a de facto monopoly.

I will never pay suddenlink another dime, even if they're the only terrestrial provider, for whatever reason.

That time is an intersection of heavy home use and when the dew hits.

I think you’d get this problem, monopoly or not, whenever cost saving measures are in place (and they always are, for good reason) at the customer-interface level.

Maybe there should always be a hidden option that only people that meet a certain troubleshooting ability threshold get access to when calling in for tech support….

SHIBBOLEET

https://xkcd.com/806/

> I think you’d get this problem, monopoly or not, whenever cost saving measures are in place (and they always are, for good reason) at the customer-interface level.

I'm guessing that these scripts that we're all complaining about solve 95% of problems customers call in about. Sure makes things painful for the 5% of cases, though.

I've been a (grudging) Comcast customer for ~17 years, and I have been impressed by how their monitoring has improved over that time. It's been quite a number of years since I've had to convince them that I had an actual problem that their systems didn't automatically detect.

There is a hidden option - you can call it "proof of work", or "proof of determination". You keep calling, and trying ways to escalate, maybe even send a paper letter; eventually, something in the customer "support" process will yield and you'll get through to someone who can actually help you.

Interesting, I wonder if I’m experiencing something a little bit similar that Comcast can’t seem to debug.

Almost every day, in the heat of summer, I get one to five 10-minute outages as soon as the temp gets over about 80F. More when it’s hotter, usually. Usually it results in a modem reset, so it’s hard to tell how long the actual outage is.

Been happening for going on 5 years. They replaced the under-street cable from our house to the junction box across the street to no effect. I suspect it’s that junction box, but afaict, none of my neighbors that share that junction box have the same issue. Not very fun to have your WFH day collapse unexpectedly in the middle of the afternoon.

Strangely, for the last month we’ve had several days of 80+ temps with no sign of outage. So fun.

Yeah, probably very similar thing if that pattern is true. It's the shift forcing your modem to change speeds, but neither side being willing to accept it.

If you can, try forcing a level at/below the speed you get during the breakage and see if it just rides it out. If it does, shift it back up and plan your coffee breaks around it. Or don't, I'm not your mother

Several day's of 80+ temps, meaning it hasn't dropped below 80? Possibly it is just getting above 80 before you start your work day. And not dropping below until after your work day has ended?

I've experienced something similar except for temperatures below ~32-36 degrees. At this particular location it would result in a ~1hr outage going below that temperature, but not when it went back above it for some reason.

We had the same problem at an old house. There was a cable splitter in the attic that was expanding in the heat and losing connection to the cable. We bought a heftier one and moved it under the insulation in the attic.

Interesting, I removed a splitter from the attic many years ago and replaced it with an F-F coupler.

I wonder if the coupler’s center conductor contacts could be expanding just enough to break the connection?

I wouldn’t be surprised if that was happening.

I ran into a similar issue, except internet and phone would get really bad on a cold morning.

Tech showed up around noon, saw I was indeed having a bad connection, went and checked the signal at the junction box for the street (can't remember what you call these) and everything was normal there, so he closed it back up again and double checks the signal at the house again, but it was fine. He walks the lines to double check but everything looked normal.

His best guess was that moisture was condensing ever so slightly inside the junction box that morning, and was let out as soon as he opened it at around noon, which fixed the problem.

Common issue in Ireland for DSL customers. Damaged copper cabling would leak water when it rained causing dropouts and lower speeds. Telecom engineers would call out on days when the copper had dried out and be unable to find any fault. Turns out correlating such reports with weather reports is hard. :/

I’d suspected that, but kept it to myself because it sounded a bit mad

I had a similar problem, due to an old line running to my house; liquid getting in, etc. And when it acted up, I'd call the cable company and be like "look, I can show you I'm losing packets right now... I need you to run tests on your end to confirm". And every time, they'd tell me they could schedule a tech to come out and take a look at it. Only, I couldn't "schedule" the problem to occur when the tech came out.. so they'd come out, declare all was fine, and leave. It was infuriating.

Eventually I called so many times and had so many appointments, that the tech lead gave me his direct number and told me to call him directly the next time it happened. When it did, I did, and he ran some tests, and confirmed there was a problem. I don't know that we ever got it sorted out (it was a while ago), but just getting them to agree there was an issue took a very long process.

A customer's DSL connection dysfunctioned every evening during December - but worked fine the rest of the year... Culprit: interference from nearby Christmas decorations leaking EM all over the place.

A customer's DSL connection dysfunction's frequency increased mornings and evenings. Culprit: the lift's electric motor leaking EM all over the place.

A bunch of DSL connections degrade when traffic increase... Crosstalk in big cables of course !

The sort of fun incidents that take a good while to troubleshoot... I'm glad we are migrating away from DSL to fiber: either it works or not !

It's not like fiber doesn't have its own weird failure modes. Favourite one I heard was shoddy belowground work while crossing a street. No problem with ordinary car traffic, but heavy trash haul trucks could interrupt the link.

An ISP my friend worked at was having weird outages in one area, and it turned out that they had an apartment block built right in the way of their free-space optical link. Surprisingly, it was fine at first because the link went straight through it without obstructions, window to window. But when they started to add window panes, finishing the construction, the link became spotty, and adding the doors blocked the signal completely.

How does one even attempt to troubleshoot that, without resorting to a questionably legal drone or chopper flight?!

Line of sight, easy - just aim a sight and find that you are seeing a building rather than the opposite antenna.