News for hackers. Stuff that matters.
Re:Fast.com: Netflix internet connection speed test by Vraxx on Wednesday May 18, 2016 @12:24PM (#11723069)
I've had a tech who came to deal with a reported issue I was having tell me to test on a specific Time Warner Cable speed test. He then got upset/flustered when I used a different speed test to demonstrate that the issue was still present. I had to explain that I use the internet connection I was paying for to do things on domains that aren't owned and optimized by TWC.
Re:Fast.com: Netflix internet connection speed test by jmtulloss on Wednesday May 18, 2016 @01:27PM (#11723757)
To give the tech the benefit of the doubt, it's possible that he just isn't certain he can trust other speed tests. They may point to issues beyond his control.
Re:Fast.com: Netflix internet connection speed test by Spooky23 on Wednesday May 18, 2016 @11:38PM (#11727782)
Absolutely. The tech has a limited scope and typically doesn't have the ability to escalate engineering issues. You need to keep complaining and Twittering until you get a senior guy.

In my case, the TWC guys replaced every piece of coax and splitter from my cable modem and TVs to the pole.

Turned out the issue was the changed out the cabling on the big avenue near my house and used the wrong grade of cable.

I called the TWC state lobbyist and told him I represented 150 homeowners who were filing a complaint with the regulator and the city council just head of their franchise agreement renewal. Trucks were replacing those lines the next morning.

Re:Fast.com: Netflix internet connection speed test by basseq on Thursday May 19, 2016 @05:08PM (#11733852)
You are a hero!

> In my case, the TWC guys replaced every piece of coax and splitter from my cable modem and TVs to the pole.

I am surprised by the coax inside the house. Usually they'll help if they can, but your house wiring is your problem. (Unless their solution is stapling a coax cable to your baseboard.)

Re:Fast.com: Netflix internet connection speed test by Spooky23 on Friday May 20, 2016 @02:36PM (#11740348)
Thanks! :)

The techs were observing erratic behavior, but there wasn't any evidence of bad cabling -- I had actually replaced it about 3 years before. Also, my wiring is low-complexity... I terminate everything in the basement, and have 1 TV. The TV cable ends about 6 feet from the cable box.

Even in other circumstances, the techs are pretty good.

Re:Fast.com: Netflix internet connection speed test by Vraxx on Wednesday May 18, 2016 @01:30PM (#11723789)
Yeah, it's tough to find the line of when to stop giving the benefit of the doubt though. I was using speedtest.net at the time, which is pretty well known and widely used. I had also had multiple visits recently regarding the issue that still had not been resolved, and so my impatience made the decision for me.
Re:Fast.com: Netflix internet connection speed test by ckuehl on Wednesday May 18, 2016 @02:36PM (#11724519)
For what it's worth, ISPs are definitely shaping traffic to try to game results on public speed tests.

I had an issue with Comcast last year where I would have large latency spikes for ~2 hours every night, and speedtest.net always remained at normal latency: https://i.fluffy.cc/kNCxLMbkF2wx7NHFJxbzcgX35Bsb5nvl.png

I couldn't find any non-speedtest sites where my latency was less than 100ms, but Comcast's own speed test, several other public speed tests, speedtest.net, and even ookla.com (the company behind speedtest.net) were perfect (the ~20ms you see on the graph).

Re:Fast.com: Netflix internet connection speed test by mcpherrinm on Wednesday May 18, 2016 @08:31PM (#11727121)
Speedtest.net and other speedtests often choose the speedtest server that's close to you and has low ping.

What you describe is probably congestion in the network somewhere that isn't the last mile. Speed tests aren't a good indicator for any issues occurring anywhere but the last mile, as they won't take the same path with fairly high probability.

Re:Fast.com: Netflix internet connection speed test by gruez on Wednesday May 18, 2016 @11:37PM (#11727781)
Given that he has a nice graph with hundreds of data points, I'm guessing he got the ping results from the ping command, not by running the speedtest.net test(which would actually select the lowest ping server)
Re:Fast.com: Netflix internet connection speed test by mcpherrinm on Thursday May 19, 2016 @12:18AM (#11727931)
You're right, though I had assumed he was pinging whatever server speedtest had selected, which may be erronous.

Really, my point was that congestion is often localized in a network. The fact that it isn't present in a path doesn't mean it's an ISP gaming speed tests - though perhaps they'd be more keen on peering directly and avoiding congested transit if it makes them look better, and I would absolutely believe that would happen.

Re:Fast.com: Netflix internet connection speed test by 67726e on Thursday May 19, 2016 @07:57AM (#11729490)
I think the argument is that the speed tests are testing ideal/non real-world usage. That seems an awful lot like gaming the results. A more realistic, indicative speed test might hit several servers at points around the country, or even world. Most of my internet traffic is not hitting a server on the outskirts of town or in the next small city over.
Re:Fast.com: Netflix internet connection speed test by Vraxx on Wednesday May 18, 2016 @02:45PM (#11724607)
Ugh. That's terrible. I kinda figured that it's an eventual conclusion though, so when speedtest.net starts giving me contradictory results to what I am experiencing I have a pretty good idea of what the check first.

On a separate note, what tool did you use to create that graph?

Re:Fast.com: Netflix internet connection speed test by icelancer on Wednesday May 18, 2016 @03:54PM (#11725311)
Looks like just redirecting a recurring ping command output to a text file then graphing the results in google docs or any other spreadsheet. I've done the same to shame Comcast.
Re:Fast.com: Netflix internet connection speed test by cfallin on Wednesday May 18, 2016 @06:39PM (#11726598)
Ooh, a chance to plug a toy side-project! I wrote "pingd" [1] to ping an address continuously and dump (timestamp, latency, dropped) tuples to a SQLite database, and also report results (tables only, no plots) via a web interface. Slightly less hacky than "ping host > logfile". Also to shame Comcast.

[1] https://github.com/cfallin/pingd (in Rust, requires liboping)

Re:Fast.com: Netflix internet connection speed test by Steko on Wednesday May 18, 2016 @07:02PM (#11726717)
FWIW I've had TWC phone techs ask me to use speedtest.net by name on a couple occasions.
Re:Fast.com: Netflix internet connection speed test by hackney on Friday May 20, 2016 @01:17AM (#11735682)
Just today I switched back to TW from Frontier(FIOS). Using fast gave me more than double the speeds of fios. The tech was super cool and he normally uses speedtest but I told 'em, try fast. Plus now I finally get to use my brandnew surfboard modem and brandnew RT-N66U router.
Re:Fast.com: Netflix internet connection speed test by lstyls on Wednesday May 18, 2016 @10:18PM (#11727490)
I have TWC in Brooklyn and for a while was on the phone with Level 3 support often. Every time they had me use speedtest.net. So even their network engineers trust that site more than their in-house one.
Re:Fast.com: Netflix internet connection speed test by eeeeeeeeeeeee on Wednesday May 18, 2016 @10:29PM (#11727538)
I agree to a point. If you use speedtest.net and show several different testing sites and the results are consistent, that should be good evidence. And you need to use a hard-wired connection because a lot of peoples wifi isn't very good.
Re:Fast.com: Netflix internet connection speed test by JensRex on Wednesday May 18, 2016 @04:27PM (#11725606)
Former telephone technician(* ) here. What I'm interested in is the connection from your home to our PoP. Anything beyond that is irelevant to me.

I don't know why he would trust customer equipment to run the test. I used one of these.

http://www.viavisolutions.com/en-us/products/network-test-an...

(* )I'm not sure what the english word is for that.

Re:Fast.com: Netflix internet connection speed test by hueving on Wednesday May 18, 2016 @04:35PM (#11725683)
>Former telephone technician(* ) here. What I'm interested in is the connection from your home to our PoP.

That's only relevant if the complaint is about the connection from your home to the PoP. If the technician was sent out to solve an issue with the Internet being slow in general, he/she needs to actually find what the bottleneck is, not call it a day when some traffic makes it to a PoP.

Re:Fast.com: Netflix internet connection speed test by walrus01 on Wednesday May 18, 2016 @05:04PM (#11725952)
It's a stupid idea to do a truck roll to resolve upstream peering/transit/congestion issues with the IP network. The technician's job is to diagnose OSI layer 1/2 issues to your local DSLAM or CMTS.
Re:Fast.com: Netflix internet connection speed test by superuser2 on Thursday May 19, 2016 @12:49PM (#11731553)
With Time Warner, there is an absolutely impenetrable firewall between customer service and backend/upstream network engineering. Rolling a truck appears to be the absolute ceiling on what a call center employee can do.
Re:Fast.com: Netflix internet connection speed test by walrus01 on Thursday May 19, 2016 @02:56PM (#11732731)
This is unfortunate, but I can sort of understand why. Customers are frequently mistaken on what's wrong with their connection, example, the customer thinks they're seeing 0.5% packet loss and high jitter consistently to something that's 3 hops upstream of their default gateway, when in fact there's something messed up in their home Wifi which is on channel 1 the same as their four neighbors.

It takes a lot of time (time = money) to train first tier customer service reps how to recognize what needs to be escalated to the people who have 'enable' on the network equipment 2 to 4 hops upstream of a typical cablemodem/DSL customer. And then the management decision to empower them to do so if necessary. And a method for the NOC of filtering through the crap to discover that "Yes, this customer really has discovered some service impacting issue that has not been automatically reported by our NMS".

Re:Fast.com: Netflix internet connection speed test by dsl on Wednesday May 18, 2016 @05:15PM (#11726036)
Yet broadband providers continue to do it.
Re:Fast.com: Netflix internet connection speed test by walrus01 on Wednesday May 18, 2016 @05:18PM (#11726060)
Usually because a customer can't properly articulate on the phone what's wrong with their connection. Last mile broadband providers want to do as much as possible to avoid a site visit to a home. Unless they're charging $95 per visit one technician visit for 30 minutes can eat up all the profit from that customer for an entire year.
Re:Fast.com: Netflix internet connection speed test by ryao on Wednesday May 18, 2016 @07:05PM (#11726730)
How much does your ISP charge you? Verizon managed to get $71.48 a month out of me on a connection that had been set up by a technician nearly 10 years prior before I got fed up. A technician visit at that rate would probably eat just 1 month of profit on a connection whose initial installation cost had been paid many times over. The reason that I got fed up was that they kept trying to get more. Nearly all of those years had involved a triple play, for which they had raised rates until they charged ~$162 a month.

Sadly, I was not the one who negotiated with them. Had I been, I would have been fed up well before things became that bad. The $71.48 came after breaking the bundle I favor of an OTA antenna and VoIP. I managed to talk my way out of the ETF by telling a supervisor that the ETF was the same whether I cancelled all services or just 2 and that I was giving Verizon the chance to keep some business, provided that they reduced services to just the Internet connection and waived the ETF. The supervisor waived it.

After breaking the bundle, I took over responsibility for such bills. I am paying Cablevision ~$45 a month now. Bandwidth is lower, but I reason that as long as bandwidth is high enough to deliver traffic with acceptable latencies, being able to burst higher does not matter that much. So far, things have been okay on both pricing and having a usable connection, despite a few initial goofs by Cablevision.

Re:Fast.com: Netflix internet connection speed test by rayiner on Wednesday May 18, 2016 @10:44PM (#11727598)
How much profit do you think Verizon actually makes from your $70/month connection?

FiOS revenues make up 70% of Verizon wireline's revenue, and the average monthly bill is like $110. But the division's profit margin is 2-5% most years. Let's say generously they're making $7 a month on you (10%). A Verizon tech makes $35/hour+ not including benefits. So a one-hour field call wipes out 5 months of profit.

Re:Fast.com: Netflix internet connection speed test by ryao on Thursday May 19, 2016 @11:08AM (#11730798)
They make nothing off it because I cancelled service.

Google searches show that they have had a special $29.99 reactivation rate for years. Several months ago they made it $39.99 in the second year. A couple years prior, they had also added a $1.99 municipal construction fee. That works out to be $37 a month. I assume that the amortized cost of keeping the fiber lit is below that because I doubt that they would make such an offer at a loss.

Charging a long time customer higher pricing would provide them with money that they can spend on things like Fios network expansion and network upgrades, which are expensive.

Re:Fast.com: Netflix internet connection speed test by rayiner on Thursday May 19, 2016 @11:36AM (#11730991)
Your analysis is wrong. When Verizon spends a bunch of money to wire up a neighborhood, that doesn't come out of profits. Instead, it books a capital asset that is depreciated over time. Verizon wireline's EBITDA margin these days--five or six years after the end of significant network expansion--is about 22%. That's what's left after paying for ongoing maintenance, support, and bandwidth, but before accounting for recouping the cost of the original build. After accounting for depreciation, interest, and taxes, it's under 5%. That means the up-front cost of your connection is being recouped at $10-20 per month at most.

Verizon spent about $20 billion on FiOS, and has about 7-8 million customers. So the average customer cost about $2,500 to wire-up. That means it'll take 10-15 years to recoup the initial cost of the build-out.

The special-offer pricing is not meaningful. Most of the cost of building and maintaining fiber is getting to the neighborhood. FIOS penetration is only about 1/3 in the places where service is offered. Which means that Verizon has a lot of incentive to make offers to those 2/3 at a rate below what it could sustain charging everyone.

Again, even though the average FIOS subscriber pays $100+ per month (triple play, etc), the division's net profit margin is under 5%. It would be losing enormous amounts of money if people were paying $30-40/month.

Re:Fast.com: Netflix internet connection speed test by ryao on Thursday May 19, 2016 @12:08PM (#11731246)
Your analysis is wrong because the subscriber count excludes Fios subscribers in areas that they sold to Frontier while the $20 billion includes them. The costs also vary depending on subscriber density. Furthermore, the actual installation cost is $750 in my state according to Verizon:

http://www.buffalonews.com/business/verizon-still-cant-justi...

I recall that number was ~$1000 9 years ago. I find it hard to believe that Verizon did not recoup that over 9 years. That is less than $10 a month over that time frame. My household paid $10 to $20 a month more for higher speed tiers over that 9 years. Those cost Verizon almost nothing so it is reasonable to conclude that they recouped the initial installation cost in this instance.

Also, Verizon has two tiers of special rates. One for new sign ups at locations that have already been wired and one for new sign ups at locations that have not. Mine is one that has been wired. Entering into an arrangement where the income from providing service is less than the day to day expenses of providing it before including one time installation costs even with a contract period makes no sense, so I do not think that Verizon is doing that. Since the one time installation costs are paid, they would have been making money even at the lower rate in my case under that assumption. Obviously, it would not be as much, but I am okay with that.

I realize that individuals' calculations often underestimate ISP expenses. However, my case is one where such calculations should apply. I also realize that wireline margins are low in general. I did not mean to state that the profitability of my particular Fios connection was the norm. I only meant to say that there are instances of old lines where the margins far exceed the average. That is why I said the connection was setup "nearly 10 years prior". The profitability of the connection would have been a very different matter otherwise.

Re:Fast.com: Netflix internet connection speed test by ryao on Thursday May 19, 2016 @04:11PM (#11733385)
I took another look at this. The $750 per house figure was actually for wiring the neighborhood without going into any homes, which costs an additional $600:

>In New York, Verizon said it costs about $750 per house to bring FiOS service to a neighborhood, plus an additional $600 to run the fiber into the home of a consumer who wants to purchase FiOS service.

Assuming 40% penetration (also in the article) and $600 on top, the cost is $2250, which is close to your $2500 calculation. I still think Verizon made that up over the course of 9 years. I just need to include the set top box rentals in the calculation. Assuming that my household paid $15 on average over 9 years more than the base tier price on average for the internet connection and each of the 3x $5 a month set top boxes that we had for 8 years cost Verizon $250 each, I get $2310. This is before considering that those should still be worth something and the principle that Verizon would not offer lower rates than their operating costs. That would be a great way to obtain negative margins.

Re:Fast.com: Netflix internet connection speed test by dsl on Wednesday May 18, 2016 @06:01PM (#11726338)
Again, you seem to be confusing what they should be doing with what they are actually doing.
Re:Fast.com: Netflix internet connection speed test by walrus01 on Wednesday May 18, 2016 @06:05PM (#11726383)
sadly, I am... I work for an ISP that generally doesn't make these mistakes, our network engineering/operations/construction team enjoys observing all the stupid things Centurylink and Comcast do in our local market.
Re:Fast.com: Netflix internet connection speed test by toast0 on Wednesday May 18, 2016 @04:50PM (#11725830)
OTOH - the technician doesn't need to be at your house if you have a good connection to the PoP. Field technicians may not be trained in troubleshooting network beyond the PoP and likely don't have the right access to tools to make the changes that need to be made.
Re:Fast.com: Netflix internet connection speed test by xanderstrike on Wednesday May 18, 2016 @04:22PM (#11725559)
On the other hand, when I worked at Verizon we used SpeedTest.net because our in-house tool often reported speeds slower than users would experience/