In my case, the TWC guys replaced every piece of coax and splitter from my cable modem and TVs to the pole.
Turned out the issue was the changed out the cabling on the big avenue near my house and used the wrong grade of cable.
I called the TWC state lobbyist and told him I represented 150 homeowners who were filing a complaint with the regulator and the city council just head of their franchise agreement renewal. Trucks were replacing those lines the next morning.
> In my case, the TWC guys replaced every piece of coax and splitter from my cable modem and TVs to the pole.
I am surprised by the coax inside the house. Usually they'll help if they can, but your house wiring is your problem. (Unless their solution is stapling a coax cable to your baseboard.)
The techs were observing erratic behavior, but there wasn't any evidence of bad cabling -- I had actually replaced it about 3 years before. Also, my wiring is low-complexity... I terminate everything in the basement, and have 1 TV. The TV cable ends about 6 feet from the cable box.
Even in other circumstances, the techs are pretty good.
I had an issue with Comcast last year where I would have large latency spikes for ~2 hours every night, and speedtest.net always remained at normal latency: https://i.fluffy.cc/kNCxLMbkF2wx7NHFJxbzcgX35Bsb5nvl.png
I couldn't find any non-speedtest sites where my latency was less than 100ms, but Comcast's own speed test, several other public speed tests, speedtest.net, and even ookla.com (the company behind speedtest.net) were perfect (the ~20ms you see on the graph).
What you describe is probably congestion in the network somewhere that isn't the last mile. Speed tests aren't a good indicator for any issues occurring anywhere but the last mile, as they won't take the same path with fairly high probability.
Really, my point was that congestion is often localized in a network. The fact that it isn't present in a path doesn't mean it's an ISP gaming speed tests - though perhaps they'd be more keen on peering directly and avoiding congested transit if it makes them look better, and I would absolutely believe that would happen.
On a separate note, what tool did you use to create that graph?
[1] https://github.com/cfallin/pingd (in Rust, requires liboping)
I don't know why he would trust customer equipment to run the test. I used one of these.
http://www.viavisolutions.com/en-us/products/network-test-an...
(* )I'm not sure what the english word is for that.
That's only relevant if the complaint is about the connection from your home to the PoP. If the technician was sent out to solve an issue with the Internet being slow in general, he/she needs to actually find what the bottleneck is, not call it a day when some traffic makes it to a PoP.
It takes a lot of time (time = money) to train first tier customer service reps how to recognize what needs to be escalated to the people who have 'enable' on the network equipment 2 to 4 hops upstream of a typical cablemodem/DSL customer. And then the management decision to empower them to do so if necessary. And a method for the NOC of filtering through the crap to discover that "Yes, this customer really has discovered some service impacting issue that has not been automatically reported by our NMS".
Sadly, I was not the one who negotiated with them. Had I been, I would have been fed up well before things became that bad. The $71.48 came after breaking the bundle I favor of an OTA antenna and VoIP. I managed to talk my way out of the ETF by telling a supervisor that the ETF was the same whether I cancelled all services or just 2 and that I was giving Verizon the chance to keep some business, provided that they reduced services to just the Internet connection and waived the ETF. The supervisor waived it.
After breaking the bundle, I took over responsibility for such bills. I am paying Cablevision ~$45 a month now. Bandwidth is lower, but I reason that as long as bandwidth is high enough to deliver traffic with acceptable latencies, being able to burst higher does not matter that much. So far, things have been okay on both pricing and having a usable connection, despite a few initial goofs by Cablevision.
FiOS revenues make up 70% of Verizon wireline's revenue, and the average monthly bill is like $110. But the division's profit margin is 2-5% most years. Let's say generously they're making $7 a month on you (10%). A Verizon tech makes $35/hour+ not including benefits. So a one-hour field call wipes out 5 months of profit.
Google searches show that they have had a special $29.99 reactivation rate for years. Several months ago they made it $39.99 in the second year. A couple years prior, they had also added a $1.99 municipal construction fee. That works out to be $37 a month. I assume that the amortized cost of keeping the fiber lit is below that because I doubt that they would make such an offer at a loss.
Charging a long time customer higher pricing would provide them with money that they can spend on things like Fios network expansion and network upgrades, which are expensive.
Verizon spent about $20 billion on FiOS, and has about 7-8 million customers. So the average customer cost about $2,500 to wire-up. That means it'll take 10-15 years to recoup the initial cost of the build-out.
The special-offer pricing is not meaningful. Most of the cost of building and maintaining fiber is getting to the neighborhood. FIOS penetration is only about 1/3 in the places where service is offered. Which means that Verizon has a lot of incentive to make offers to those 2/3 at a rate below what it could sustain charging everyone.
Again, even though the average FIOS subscriber pays $100+ per month (triple play, etc), the division's net profit margin is under 5%. It would be losing enormous amounts of money if people were paying $30-40/month.
http://www.buffalonews.com/business/verizon-still-cant-justi...
I recall that number was ~$1000 9 years ago. I find it hard to believe that Verizon did not recoup that over 9 years. That is less than $10 a month over that time frame. My household paid $10 to $20 a month more for higher speed tiers over that 9 years. Those cost Verizon almost nothing so it is reasonable to conclude that they recouped the initial installation cost in this instance.
Also, Verizon has two tiers of special rates. One for new sign ups at locations that have already been wired and one for new sign ups at locations that have not. Mine is one that has been wired. Entering into an arrangement where the income from providing service is less than the day to day expenses of providing it before including one time installation costs even with a contract period makes no sense, so I do not think that Verizon is doing that. Since the one time installation costs are paid, they would have been making money even at the lower rate in my case under that assumption. Obviously, it would not be as much, but I am okay with that.
I realize that individuals' calculations often underestimate ISP expenses. However, my case is one where such calculations should apply. I also realize that wireline margins are low in general. I did not mean to state that the profitability of my particular Fios connection was the norm. I only meant to say that there are instances of old lines where the margins far exceed the average. That is why I said the connection was setup "nearly 10 years prior". The profitability of the connection would have been a very different matter otherwise.
>In New York, Verizon said it costs about $750 per house to bring FiOS service to a neighborhood, plus an additional $600 to run the fiber into the home of a consumer who wants to purchase FiOS service.
Assuming 40% penetration (also in the article) and $600 on top, the cost is $2250, which is close to your $2500 calculation. I still think Verizon made that up over the course of 9 years. I just need to include the set top box rentals in the calculation. Assuming that my household paid $15 on average over 9 years more than the base tier price on average for the internet connection and each of the 3x $5 a month set top boxes that we had for 8 years cost Verizon $250 each, I get $2310. This is before considering that those should still be worth something and the principle that Verizon would not offer lower rates than their operating costs. That would be a great way to obtain negative margins.
fastly-debug.com.
The goal of our site is to gather debug information but I use it as a reliable speed-test to a good portion of the internet's websites. I think this is a good tool like fast.com because unlike the 'official' speedtest sites, if you optimize ours or Fast's you're actually making the internet much faster for a lot of sites for users.
So then switched to a cable provider, pay half what was paying with Verizon, get lower advertised speeds yet have a much more reliable and consistent connection.
Also Verizon had free installation at that time when I signed up, so they installed a bunch of their equipment on premises, even a phone (including a battery pack) and now after I dropped them they come to door, in person, every other month begging me to reconsider.
Netflix might have been able to get Verizon to be more reasonable if they modified their streaming clients to upload random numbers back to them in excess of the amount downloaded. Then Verizon's logic would have dictated that Verizon owed Netflix's backbone provider money and Verizon would have wanted to avoid paying rather than be paid. Such a situation should have caused the links that Netflix traffic used to be upgraded without much negotiation. I was surprised that Netflix did not do that as a workaround. That would have made anyone even superficially familiar with game theory smile.
That is one thing to expect that on a low end 2Mbps connection, but I was paying for a 100Mbps wouldn't expect to have any issues at rate I was having them. How they decided to structure their payments downstream is their problem so to speak (putting on my non-technical consumer hat on).
I am glad I had a choice to switch and now with half the price or so (and also half the advertised bandwidth (65Mbps) I seem to get a much better user experience.
I was affected by that peering dispute too. That lead me to research it, understand it and workaround it. Consequently, I use that knowledge to implement workaround whenever a peering issue affects me.
To name one, I observed 1.3 second average RTTs between Shanghai and New York when I was visiting family there in 2014. My research into the Verizon peering disputes prepare me fairly well for dealing with it. I initially used OpenVPN over TCP and my pings were running inside the VPN tunnel. Packet loss was typically in the range of 10% to 20%. There were occasional latency spikes to 30 seconds. Switching to UDP lowered latencies to ~300 ms in either case, but bandwidth was rather low. I forget how bad it was in absolute terms, but I can say that it was too bad to use YouTube or even VoIP. What helped was routing through servers in Japan.
The trick was to do a perverse variation on a NAT where the LAN and WLAN interfaces are the same and the NAT is effectively a proxy for a port on a remote system. It works well for routing around peering disputes when you can get a VM in a place where there are not peering disputes between the local ISP and the VM provider and the VM provider and the destination. Trace route results at the time suggested that Japanese ISPs appeared to maintain both sides of the underwater link between them and China while US ISPs only control the network sockets at the Internet exchanges in the US. Routing through the Japanese internet backbone presumably eliminated the congested US-Chinese peering links from the path, which showed because packet loss went into the single digits. I do not remember the exact numbers, but it also put speed between 1Mbps and 10Mbps. That was enough for me to make VoIP calls to NY while I was visiting family in Shanghai and YouTube was usable and even load YouTube videos.
However, the connection was still bad enough to make Verizon's peering dispute with Netflix look mild in comparison. There were occasional latency spikes to 30 seconds, which appeared to be from buffer bloat in the great firewall. Google search results on the Chinese backbone at the time suggested that the entire county also appeared to have a star-like network topology for its backbone, with the exception being that Hong Kong was on the outside of that. My trace route results from Shanghai then were consistent with it.
Another OSS developer in Shanghai informed me at the time that the three major ISPs in China have peering disputes among themselves despite the Chinese government owning all three. He claimed that communications over his 100Mbps last mile connection at his house with his employer's office operated at dial up speeds because of that. I did not confirm that, but I believed him. What I saw of Chinese Internet connectivity while I was ther struck me as being incredibly terrible.
I still remember how bad things were with Verizon and Netflix. However, the reality is that things could have been many times worse like they were in China when I was there in 2014. At least with Verizon and Netflix, you had the option of the HE tunnel broker. In China, I saw ISPs using carrier grade NAT, such that using IPv6 via the tunnel broker is impossible. That is not to say that IPv6 via the tunnel broker is some sort of magic cure for issues. The only reason IPv6 via the tunnel worked during that dispute was because there was no congestion between Hurricane Electric and Verizon as well as between Hurricane Electric and Netflix. If many people had tried using it, it would likely have degraded too.
I have 0 interest in buying a fast line to comcasts internal services, they are selling me a way to connect to the rest of the internet.
Whether it's what they intended to sell or not, ISPs, through their own marketing material, sell a connection to "the Internet", and unfortunately for them that term cannot fairly be applied to just Comcast's network anymore (though it would make for a comical court case).
If they can't provide what is advertised, specifically "Internet access", then it's false advertising and I expect, and will fight for, my money back.
A workaround would be to rely on an IPv6 tunnel broker or a VPN tunnel to a VM in some datacenter. As long as the tunnel is unaffected by peering disputes, your traffic over it will live at the other side of the divide between content producers and consumers. That gives a far better experience than relying on a last mile operator to give you access to the broader Internet.
Speaking of which, does anyone know when hacker news will support IPv6?
How would breaking up Verizon solve a problem caused by Verizon's internal divisions being separate?
I never have buffering and I use it a TON.... I'm convinced that they want my data consumption over anything
http://arstechnica.com/business/2016/03/netflix-throttles-vi...
I think what was happening is that the VPN service had servers in the NYC area that had great connections to the local ISP network, and they also had a great connection to the Netflix servers, vs Verizon was deliberately keeping their pipe to the Netflix servers small so that there was limited Netflix bandwidth entering the local ISP network.
Why improve business fundamentals when it's cheaper to manipulate human emotion through marketing?
It's cheap to have someone pester you and 100+ other people that day to see if you'll budge, but it's far from a personal effort to win you back.
This is important because unlike your average Internet speed test (which ISPs take pains to optimize), there's a very real possibility that your ISP is happy to let your Netflix experience suffer - assuming they don't throttle it outright - as previously mentioned on HN:
https://news.ycombinator.com/item?id=7183682
https://news.ycombinator.com/item?id=7853603