The problem is that they then assume that all cases are one of those 95% in order to solve the 95% as quickly as possible, which probably looks good to whatever metrics they're tracking.
But if you're one of the 5% you're fucked.
If there's anyone out there designing tech support procedures, you should add an "is this a 5% problem?" question to whatever checklist you give to support staff.
When I was the engineer customer service escalated to, I was damn sure to thank them every time they escalated something. Even the one guy who escalated all the things I'd roll my eyes about in private. At least he was making sure the escalation path worked.
Someone who has taken the time to report an issue is probably one of hundreds or thousands who had an issue and didn't think it could be fixed and shrugged it off. We certainly can't fix everything, but weird network shit like this can be fixed, and it's worth escalating, because when you get it fixed, you can also figure out (hopefully) how to monitor for it, so it doesn't happen again.
OTOH, I didn't work for the phone company. We don't care, we don't have to, we're the phone company. https://vimeo.com/355556831 (sorry about the quality, I guess internet video was pretty lowdef in the 70s :P)
The most memorable one:
Customer service had been getting calls all morning with a peculiar complaint: A customer's phone would ring, and when they answered, the party on the other end didn't seem to hear them. They seemed to be talking to _someone_, but not the party they were connected to. Eventually they hung up. Sometimes, a customer would place a call, and be on the other end of the same situation -- whoever answered would say hello, but the two parties didn't seem to be talking to each other. Off into the void. They'd try again, and it would work, usually, but repeats weren't uncommon.
So everyone's looking at system logs and status alarms and stuff, and what else changed? There were two new racks of echo-cancellers placed in service last night, could that cause this? Not by any obvious means, I mean e-cans are symmetrical and they were all tested ahead of time. There was a fiber cut out by the railroad but everything switched over to the protect side of the ring OK, didn't it? Let's check on that. Everyone's checking into whatever hunch they can synthesize, and turning up bupkus.
Finally around lunchtime, one of the techs bursts into the ops center, going "TIM! I GOT ONE I GOT ONE IT'S HAPPENING TO ME, PATH ME! okay look I don't know if you can hear me, but please don't hang up, I work for the phone company and we've got a problem with the network and I need you to stick on the line for a few minutes while we diagnose this. I know I'm not who you expected to be talking to, and if you're saying anything right now, someone else might be hearing it, but that's why this is so weird and why it's so important YEAH IT CAME INTO MY PERSONAL LINE and that's why it's so important that you don't hang up okay? I really appreciate it, just hang out for a few, we'll get this figured out..."
Office chairs whiz up to terminals and in moments, they've looked up his DN and resolved it to a call path display, including all the ephemera that would be forgotten when the call disconnects. Sure enough, it's going over one of the new e-cans. Okay, that's a smoking gun!
So they place the whole set of new equipment, two whole racks of 672 channels each, out-of-service. What happens when you do that is the calls-in-process remain up, but new calls aren't established across the OOS element. Then you watch as those standing calls run their course and disconnect, and finally when the count is zero, you can work on it. (If you're doing work during the overnight maintenance window, you're allowed to forcibly terminate calls that don't wrap up after a few minutes, but that's verboten for daytime work. A single long ragchew is the bane of many a network tech!) The second rack was empty of calls in _seconds_, and everyone quickly pieced together what that implied -- every single call that had been thus routed was one of these problem calls where people hang up very quickly. This thing had been frustrating hundreds of callers a minute, all morning.
With the focus thus narrowed, the investigation proceeded furiously. Finally someone pulls up the individual crossconnects in the DACS (a sort of automated patch panel, not entirely unlike VLANs) where the switch itself is connected to the echo-cancellation equipment. And there it is. (It's been too long since I spoke TL1 so I won't attempt to fake a message here, but it goes something like this:) Circuit 1-1 transmit is connected to circuit 29-1 receive, 29-1 transmit isn't connected to anything at all. 1-2 transmit to 29-2 receive, 29-2 transmit to 1-1 receive. Alright, we've got our lopsided connection, and we can fix it, but how did it happen in the first place?
If all those lines had been hand-entered, the tech would've used 2-way crossconnects, which by their nature are symmetrical. A 2-way is logically equivalent to a pair of 1-ways though, and apparently this was built by a script which found it easier to think in 1-ways. Furthermore, for a reason I don't remember the specifics of, it was using some sort of automatic "first available" numbering. There'd been a hiccup early on in the process, where one of the entries failed, but the script didn't trap it and proceeded merrily along. From that point on, the "next available" was off by one, in one direction.
Rebuilding it was super simple, but this time they did it all by hand, and double-checked it. Then force-routed a few test calls over it, just to be sure. And in a very rare move, placed it back into service during the day. Because, you see, without those racks of hastily-installed hardware, the network was bumping up against capacity limits, and customers were getting "all circuits busy" instead. (Apparently minutes had just gotten cheaper or something, and customers quickly took advantage of it!)
He loved all of that stuff, absolutely hated when everything went to computers. Quit and became a maintenance man at a nursing home, commercial laundry repair guy then finally retired this year in his late 70’s (due to Covid) after working maintenance at a local jail.
I believe the #5 ESS machine itself is always in closed cabinets, so it's likely that what you're remembering was the toll/transport equipment, or ancillary frames. Gray 23-inch racks as far as the eye can see!
Depending on how old that part of the office was, they were likely either 14' or 11'6" tall with rolling ladders in the aisles, or 7' tall and the only place they'd have laddertrack was in front of the main distributing frame.
As for magnetic core, if you could see it mounted in a frame, what you probably saw was a remreed switching grid, which is a sort of magnetic core with reed-relay contacts at each bit, so writing a bit pattern into it establishes a connection path through a crosspoint matrix. It's not used as storage but as a switching peripheral that requires no power to hold up its connections. (Contrast with crossbar, which relaxes as soon as the solenoids de-energize.)
Remreed was used in the #1 ESS (and the #1A, I believe), and is extensively documented in BSTJ volume 55: https://archive.org/details/bstj-archives?&and[]=year%3A%221...
https://upload.wikimedia.org/wikipedia/commons/7/7e/1A_Frame...
I just remember thinking it looked awkward getting to the equipment under them.
I don’t know if the ‘5E’ as he called it was actually in operation yet, he ended up moving us all out of state to take a job developing and delivering training material for it...I think that’s what finally broke him lol. Hands on kinda dude.
I’ll have to hit him up later today to see if he remembers ‘remreed’ (he will). Thanks for the info!
At night, traffic was often low enough that you could hear individual call setups and teardowns, each a cascade of relay actuations rippling from one part of the floor to another. The junctor relays in particular were oddly hefty and made a solid clack, twice per call setup if I recall correctly, once to prove the path by swinging it over it to a test point of some sort, and then again to actually connect it through. On rare occasion, you'd hear a triple-clack as the first path tested bad, an alternate was set up and tested good, and then connected through.
Moments after such a triple-clack, one of the teleprinters would spring to life, spitting out a trouble ticket indicating the failed circuit.
The #5, on the other hand, was completely electronic, time-division switching in the core. The only clicks were the individual line relays responsible for ringing and talk battery, and these were almost silent in comparison. You couldn't learn anything about the health of the machine by just standing in the middle of it and listening, and anyone in possession of a relay contact burnishing tool will tell you in no uncertain terms, that the #5 has no soul.
> including all the ephemera that would be forgotten when the call disconnects
Interesting to know there is information which is not logged. I’m guessing keeping this info, even for a day, would have helped isolate the issue?
How did the echo cancellers pass testing?
All that was true, the failure happened when it was being taken out of testing config and into operational config. Either nobody considered that that portion could fail, or the urgency to add capacity to a suddenly-overloaded network meant that some corners were cut. (Marketing moves faster than purchasing-engineering-installation...)
Oh, and as to the point about keeping the call path ephemera. Yeah probably, but in a server context, that'd be akin to logging the loader and MMU details of how every process executable is loaded and mapped. Sure, it might help you narrow down a failing SDRAM chip, but the other 99.99999% of the time when that's not the problem, it's just an extra deluge of data.
The funny thing is that if everyone played along they could have had a mean game of telephone going.
Have you heard any of Evan Doorbell's telephone tapes[1]? It's a series of recordings mostly from the 1970s, but with much more recent narration, exploring and sort-of documenting the various phone systems from the outside in. Might be interesting to see what they figured out, and what they didn't :)
I used to play counter strike/Starcraft in my middle school years. I pretty much figured out I had consistent packet loss with a simple ping test. I was on the phone with Time Warner every other day for months. They kept sending the regular technicians, at one point ripping out all of my cable wires in the house to see if it fixed the problem. Nothing worked, I kept calling, at this point I had the direct number to level 3 support. They saw the packet loss too. Finally, after two or three months they send out a Chief Engineer. Guy says I’ll look at the ‘box’ on one of the cable poles down the block. He confirmed something was wrong at that source for the whole area. Then it finally got fixed.
Took forever dealing with level 1 support, and lots of karenesque ‘can I talk to your supervisor please’, but that’s literally what it took.
So yeah, if you want stuff like this fixed, stay professional, never ever curse, consistently ask to speak to the supervisor, keep records, and keep calling.
Small shout out to the old http://www.dslreports.com/ for being a great support community during the early days of broadband for consumer activism in terms of making sure you got legit good broadband.
I ended up just cancelling the service and signing up for one of their competitors instead.
It had happened before and then magically fixed itself a few days later.
One time I had a week of outage with AT&T basically said the problem was on my side. They could ping the modem, and then punted. I had several truck rolls. The techs were really nice guys, but were basically cabling guys, better for finding a bad cable than debugging a packet loss. The problem for me was that my ipv4 static ip addresses would not receive traffic.
I was at wit's end after a week and I debugged the thing myself. By looking at EVERY bit of data on the router, I found mention of the blocked packets in the firewall log. I would clear all the logs, and found even with the firewall DISABLED, the firewall log would see and block incoming packets I was sending using my neighbor's comcast connection.
I called AT&T, but this time mentioning "firewall is completely off, but packets are blocked by the router and showing up in the log" was concrete enough for them to look up a (known) solution.
The fix was to disable the firewall, but to enable stealth mode. wtf?
To be clear, this was a firmware bug, and caused dozens of calls to AT&T, lots of heartache and finger pointing always in my direction.
I should also mention at the start of this fiasco, I checked the system log and noticed they pushed a firmware update to the modem at the time the problem started. Strangely after one call to the agent, that specific line disappeared out of the log file, but other log entries remained. hmmm.
since then, they basically screw up my modem every month or two - they push new firmware and new "features" appear (like the one that sniffs and categorizes application traffic like "youtube" and "github"). It also helpfully turns wifi BACK ON when I had disabled it. I immediately go turn if back off and then they immediately send me a big warning email that my DSL settings have been changed.
This is all because AT&T believes that the edge of their network is not the PON ONT/OLT, but rather, the router they issue you. If you want to be on their network, you have to use their router as some part of the chain.
My latest discovery is that in doing this, the router can actually get super hot operating at gigabit speeds for extended periods of time. When this happens, it magically starts dropping packets. Solution? Aim a fan at the router so it has "thermal management."
Total. Garbage. I'd switch to Spectrum if they had decent upload speed, but alas, they don't in my building.
[0]: https://particle17.xyz/posts/amplifi-alien-thoughts/#appendi...
[1]: https://www.att.com/support/article/u-verse-high-speed-inter...
Another option is getting the 802.1x certificate out of a hacked router, but it's not possible as far as I know on the 5268ac. You could buy a hackable ATT router but they're not cheap. Some sellers even sell the key by itself.
Mysteriously, doing this fixed an issue I previously had where SSHing into AWS would fail.
I'm currently using eap_proxy with my BGW210, and it's been a huge improvement, but I fear the day the device needs to be replaced with a newer model.
However, it has 5Gbit Ethernet, hasn't re-enabled WiFi on automatic firmware updates, and has only screwed with my IP Passthrough configs once which was resolved with a router reboot. (that was possibly my router's fault, it seemed like it was unable to fetch a new DHCP lease)
All you have to do is answer a form with questions like: Do you know how to plug in a computer? Do you know where the power switches are on your devices? etc.
CSA Pre™ is valid for 5 years; you can initiate the renewal process to do 6 months before expiry.
I've had this exact experience, except that it wasn't a dream.
Back in 2010 I had a weird issue where my cable connection would sometimes completely block the connectoin right after the DHCP response (we had dynamic IPs back then). This would go on for a couple of hours, until the IP lease expired, then my connection would come back. Luckily, I was running an OpenBSD box as my router which allowed me to diagnose the problem. But it was also impossible to explain to the servicedesk employees.
One evening it happened again, and I called the servicedesk, totally prepared to do the 'yes I have turned it off and on again' dance. But to my surprise the employee that I got on the phone was very knowledgable and even said that it was very cool that I had an OpenBSD box as a router. He very quickly diagnosed that someone in my neighbourhood was 'hammering' the DHCP service by not releasing his lease (a common trick to keep your IP address somewhat static). This caused a double IP on the subnet, and the L2 switch to block traffic to my port.
He asked me "do you know how to spoof your MAC with an OpenBSD box?". Then I knew this guy was legit. He instructed me to replace the last 2 bytes of the MAC with AB:BA (named after the music group). They had a separate DHCP pool for MAC addresses in that range. If they ever saw an ABBA mac address on their network, they knew it was someone who had connectivity issues before.
The problem was immediately solved, and I had a rock-solid internet connection for years, with a static IP!
I ended up chatting about networking and OpenBSD a bit, before I (as humble as I could) told the guy I was a bit flabbergasted that someone as knowledgable as him was working on the servicedesk.
It turned out, he was the chief of network operations at the ISP (the biggest ISP in my country). He was just manning the phone while some of his colleagues from the servicedesk were having diner.
Sometimes miracles do happen.
Anyway I was told on more than one occasion by different telcos that the standard operating procedure for many techs was to take the call, do nothing, and call back 20 minutes later and ask if it looked better because ... often enough it did.
It always was.
The solution could be a priority support tier where you pay upfront for an hour of a real network engineer's time (decently compensated so that he actually cares about solving the problem) and the charge is refunded only if the problem indeed ends up being on the ISP's side. This should self-regulate as anyone wasting the engineers' time for a simple problem they could resolve themselves would pay for that time.
About 20 years ago I got escalated to high-tier Comcast support for an issue that turned out to be a little of A, a little of B: Comcast (Might have been @Home, based on the timing) required that your MAC address be whitelisted as part of their onboarding process. Early home routers had a "MAC Address Clone" feature for precisely this reason. At some point, the leased network card got returned to Comcast. Our router continued to work just fine... until about a year later, when the local office put that network card into some other customer's home. We started getting random disconnects in the middle of the day, and it took forever to diagnose, as the other customer was not particularly active with their internet use. Whose fault was it? Ours, for ARP Spoofing? Theirs, for requiring the spoofing?
That makes sense.
When the cable modem issued a DHCP request, the CMTS would have been configured to insert some additional information (a "circuit-id") into the DHCP request as it relayed it to the DHCP server.
The short version is that the "competent tech" looked at the logs from the DHCP server, which would have showed that the "same" cable modem (i.e., MAC address) was physically connected to either 1) two different CMTS boxes or 2) two different interfaces of the same CMTS.
How would one cable modem be physically present in two different locations at the same time? Obviously, it wouldn't.
At that point, either 1) there are two cable modems with the same burned in address or 2) one of the two cable modems is cloning/spoofing its MAC address. Which one of those is more likely?
(If you're interested in the details, try "DHCP Option 82" as your search term.)
For instance: "This might just be a loose cable, can you unplug each end of it, and plug it back in?" invariably elicited an "I've already done that", or a brief pause followed by "okay, there, I did it, do you see it?". Lies, often.
But: "Alright, I want you to try turning the cable around for me, yeah swap it end for end. Sometimes you get a connector that just doesn't fit quite right, but it works the other way around and it's faster than sending a tech to replace it", would often get a startled "Oh! I hadn't thought of that, one moment..." and then the customer actually DOES unplug the thing, and what do you know, they click it in properly this time.
Then one day, she tells me that she tells people to unplug the power cord, cup the end in their hand, and then plug it back in.
Suddenly, I really liked her. It's a genius move that makes them think they've done something obscure, but she really just wanted them to actually check the cable.
Luckily, I already knew how to fix them. I found the job to be a cake walk and quite liked helping people. But I had to listen to people around me fumble through it.
It was frustrating at the time, but my favorite thing that happened was that I was admonished twice for having average call times that were too low. To them, that's a warning that someone is just getting people off the line without fixing their problems.
They monitored calls and they said I'd never received a complaint, but the system would keep flagging me for low call times so I had to artificially raise them. They suggested that I have a conversation with the clients.
I didn't, and I didn't stay there much longer, but it was quite a crazy situation. But I also felt much less pressure to handle calls quickly after that, too, which was nice.
Everything they do is driven by metrics, and their contracts are written around KPIs like maintaining a ludicrously short average time to answer, short handle times, and open/closed ticket ratios. If they do not hit these metrics, then the outsourcing company owes service credits. The incentives do not align with making customers happy and solving their problems. Everything is geared towards deflecting users with self-help options, simple scripts for the agents, and walking the line of hitting those metrics with the fewest number of warm bodies.
It's a pretty hellish business.
For the last several years, I've gotten my mobile service through an MVNO named Ting. On the rare occasion that I've needed to call support, there's no IVR, just a human who typically answers on the second ring. They speak native English, and have never failed to solve my problem either immediately or with one prompt escalation.
They're so jarringly competent I wonder how they still exist, if being obnoxiously incompetent is apparently a business requirement.
... only if you want people to actually be able to escalate. Suspect AT&T are going to lose a lot less money over this issue than hiring an extra high quality support person would cost
For evidence, see: “Retention departments give you the best rates if you threaten to cancel” (which then caused companies to have to rename the retention departments and change the policies.
$ diff example-*
4c4
< <titde>Example Domain</title>
---
> <title>Example Domain</title>
11c11
< backoround-color: #f0f0f2;
---
> background-color: #f0f0f2;
23c23
< box-shadow: 2pp 3px 7px 2px rgba(0,0,0,0.02);
---
> box-shadow: 2px 3px 7px 2px rgba(0,0,0,0.02);
41c41
< <p>This domain is for use in illustrative examples in documents. You may usm this
---
> <p>This domain is for use in illustrative examples in documents. You may use thisSo either the bit corruption is such that is not detected by the checksum
Or ATT is doing something nefarious and touches layer 4 and corrupts the data while doing so
https://www.evanjones.ca/tcp-checksums.html
With enough packets passing the dodgy RAM a noticeable number will manage to get mangled in such a way that the checksum is still correct.
Routers don’t touch the TCP header at all
But in any case it's irrelevant in this context as ATT shouldn't be doing any NAT
Another quality product by Apple ;P
That tweet in particular doesn't show any retransmits.
tcpdump/wireshark gets a little hard to read at times; especially when the packet dump is a lie: all those packets marked red for bad checksums are from the dumping machine, and the checksums are wrong because the NIC is filling them in, and the capture interface doesn't get to see what they are). Perhaps the other people in the thread who said mac os was ignoring bad checksums were also confused; or perhaps it does ignore bad checksums, it's pretty bad at networking (it can't handle a synflood in 2020 because it's got synhandling code from 2000)
I wouldn't have expected Apple to purposefully break the checksum, just like they I don't think they purposefully have no synflood protection because they pulled the TCP/IP stack at the turn of the century and never pulled it to get the many many many upgrades from upstream (although, they did add on MP-TCP, so there's that). I wouldn't be surprised if tcp checksums had stopped working ages ago, possibly because of a aggressive driver, and nobody noticed. Kind of like how if you spawn a few thousand threads that just sit around sleeping, it will delay watchdog kicks and the kernel will panic. (also, from reports on here, not personal experience)
>>> p1 = IP(dst="192.168.mac.ip")/TCP(dport=1984,sport=20001)
>>> p2 = IP(dst="192.168.mac.ip")/TCP(dport=1984,sport=20002)
>>> p2.show2()
###[ IP ]###
version = 4
ihl = 5
tos = 0x0
len = 40
id = 1
flags =
frag = 0
ttl = 64
proto = tcp
chksum = 0xf905
src = 192.168.linux.ip
dst = 192.168.mac.ip
\options \
###[ TCP ]###
sport = commtact_http
dport = bb
seq = 0
ack = 0
dataofs = 5
reserved = 0
flags = S
window = 8192
chksum = 0xb836
urgptr = 0
options = []
>>> p2[TCP].chksum = 0xb836 ^ 0x8 # mangled checksum
>>> sr1(p1, timeout=1)
Begin emission:
Finished sending 1 packets.
....*
Received 5 packets, got 1 answers, remaining 0 packets
<IP version=4 ihl=5 tos=0x0 len=44 id=0 flags=DF frag=0 ttl=64 proto=tcp chksum=0xb902 src=192.168.mac.ip dst=192.168.linux.ip |<TCP sport=bb dport=microsan seq=1671494800 ack=1 dataofs=6 reserved=0 flags=SA window=65535 chksum=0x6039 urgptr=0 options=[('MSS', 1460)] |>>
>>> sr1(p2, timeout=1)
Begin emission:
Finished sending 1 packets.
.....................................................................................................................................................................................
.....................................................................................................................................................................................
....................................................................................................................................
Received 494 packets, got 0 answers, remaining 1 packets
0 and 0 are still 0 and 0,
1 and 1 are still 1 and 1,
but 0 and 1 become 1 and 1 (or 0&0)
and 1 and 0 become 1 and 1 (or 0&0).
I remember seeing this wackiness when I bridged two address lines on an EEPROM with the tiniest amount of solder.
RoHS/tin whiskers strikes again?
Not just in the handshake, TLS moves these things called TLSPlaintext records (about 16kbytes each), not only in the handshake, but also for all the actual data - and they'll always have integrity protection to ensure bad guys can't change anything. TLS can't know the difference between a bad guy tampering with data and your crappy Internet mangling the data in transit, in either case the TLS protocol design says to "alert" bad_record_mac which your browser or similar software will probably treat as a failed connection, even if it happens mid-way through an HTTP transaction.
Because TLS guarantees integrity even if your fiber is a complete shit show, any TLSPlaintext records which do get from one end to the other are guaranteed to be as intended.
https://tools.ietf.org/html/rfc8446
If you aren't interested in why this works just the Introduction to the RFC explains the intent, specifically what we care about here is that TLS delivers:
"Integrity: Data sent over the channel after establishment cannot be modified by attackers without detection."
And further notes "These properties should be true even in the face of an attacker who has complete control of the network".
So yes, it is blowing up all sorts of traffic everywhere. You just don't notice when it is plaintext.
Edit: Actually I take that back, I have seen one of the issues where sometimes web sites will not load at random, and a reload fixes the issue.
What's a "DNS prop"?
Nice find! I had been wondering why I had been seeing odd TLS failure messages recently.
The files differ after anywhere from 1sec to 5sec for me, and it's always the same character positions, and it always seems to be the same lines, and the same number of lines.
My ATT modem (Arris bgw-210-700) may have gotten a firmware upgrade recently as I found some settings I shut off got re-enabled. I use it for VDSL but the same model family is used for ATT fiber.
It has to be a special kind of hardware that does much deeper packet inspection (DPI) to recalculate TCP checksums, usually used for spying, throttling, censorship, injecting ads, injecting exploits, etc., but not merely routing/forwarding packets.
There is no reason for it to touch the TCP header.
And yet, an unfortunate number of L2 switches do exactly that. :(
Can you name any? Just curious
The problem is routers slow down the high speed serial signals from fiber to by splitting the bits over a large number of slower speed signals internally. Often those wider busses are a multiple of 16 bits. For example, one ASIC I know of moves things around in 204 byte chunks. (Might have been 208, been a while.) Anyway, the problem is that if there is a defect in one of those parallel elements it will always flip bits in the same offset position mod 204 bytes, which is the same position mod 16 bits. If the hardware is degraded enough, it can end up flipping two bits in the same position, and that has a fairly good chance of passing the checksum.
Ethernet has proper CRCs on packets, which is a lot less vulnerable to shenanigans like this, but unfortunately those can end up being checked on the way in, discarded, and then re-generated on the way out of a router. If anything is corrupted in the middle of the switch ASIC, nothing notices and it passes along. I once helped troubleshoot an issue in our network where a BGP packet was corrupted in this way. The flipped bits ended up causing a more specific route to be generated, and we had the world weirdest BGP route hijack within the bounds of our own data center.
Only in certain areas! Within San Francisco on overhead-cabled blocks (for example) their lines are their own.
Has anyone claimed to see these flipped bits on a domain other than example.com?
See e.g. https://gist.github.com/bmastenbrook/14c0e22fc02b95d4a48f82d...
Given that (it seems) only AT&T customers are complaining, and (it seems) it only affects servers on EdgeCast.
https://twitter.com/bmastenbrook/status/1335400747794530304
It loads http://example.com and https://example.com and compares the result (should be equal) in a loop, and then reports if it finds a difference. I'm seeing multiple bit flips in the unencrypted version, and having a lot of issues loading web pages, presumably because a corrupted packet in a TLS handshake is an error and the connection dies.
Tech support, even when Twitter accounts with a lot of followers message them, is completely useless, they just say that they don't show any outages at your location. They need to be flooded with complaints for someone to look at this, or maybe someone from AT&T is on here that can get it looked at...