Netflix, YouTube Streaming Gets Worse on Major ISPs, Who In Turn Blame the Deep, Dark Mysteries of the Internet

from the what-the-hell-is-going-on dept

Tue, Feb 11th 2014 2:24pm — Karl Bode

Last week this blog entry by iScan developer David Raphael saw a lot of attention as proof positive that Verizon was violating network neutrality. Specifically, Raphael observed that services using Amazon Web Services accessed from his home FiOS connection (including Netflix) delivered bits more slowly than services accessed on his business FiOS connection. As his supposed nail in the neutrality-violation coffin, Raphael provided a chat transcript from a Verizon rep supposedly "confirming" that the company was throttling select services. This "proof" then bumbled and stumbled its way around the Internet as evidence neutrality had been trampled.

Except as anybody who has spent any time on the phone with a support rep knows, what comes out of their mouths may or may not have any direct tie to this particular plane of reality. It's also worth noting that the kind of Netflix stuttering Raphael saw has been going on for much of the last year across numerous ISPs, long before Verizon's recent court victory over the FCC's neutrality rules. Verizon also denied they were manipulating traffic in a statement sent to all media outlets:

"We treat all traffic equally, and that has not changed. Many factors can affect the speed a customer's experiences for a specific site, including, that site’s servers, the way the traffic is routed over the Internet, and other considerations. We are looking into this specific matter, but the company representative was mistaken. We are going to redouble our representative education efforts on this topic."

While Raphael may have stumbled into Verizon shenanigans, what he saw could also be explained by normal peak network or peering congestion, and we've long discussed the problems with crying net neutrality wolf. Peering relationships are indeed complicated, with ISPs, content companies, and Internet backbone operators all engaged in an endless tug of war over video profits and peering imbalances (I recommend this piece last summer from Jon Brodkin at Ars Technica or this piece from last week by GigaOM's Stacey Higginbotham).

With that as background, fast forward to this week when Netflix released their latest ISP streaming rankings, which show that most of the nation's biggest ISPs have been dropping in regards to their Netflix streaming performance. ISPs that partner with Netflix's Open Connect CDN do better in the rankings, and major ISPs have tried somewhat feebly to claim Netflix's attempt to name and shame them into using their CDN is a sort of reverse network neutrality.

Still, Verizon FiOS (and DSL), Comcast, and AT&T's U-Verse services have seen major drops in the rankings not entirely explained by their refusal to use Netflix's CDN. Being large network operators, you would imagine that they'd be able to explain what exactly is happening to stink up millions of broadband users' Netflix experiences, though Comcast, AT&T and Time Warner Cable refuse to comment to the media on their collective performance nose dive. Netflix also refuses to comment. Verizon, with all of their network expertise, continues to blame the problems on either unknown Internet bogeymen or Netflix:

"We state unequivocally that Verizon’s broadband Internet access services deliver a pristine user experience to our customers at any time of day on every day of the week. This has been repeatedly proven through independent testing by the Federal Communications Commission (FCC), which has conclusively demonstrated that FiOS Internet consistently delivers both download and upload speeds in excess of what we advertise. In short, our Internet customers often get more than they pay for. How the Internet works can be complicated, and consumers should be aware of the fact that the integrity of their home Internet connection is only a portion of the streaming video quality equation. If their broadband connection is functioning correctly, the source of their frustration and the content they wish to see may be one in the same."

In summation, the only major ISP wiling to comment on why millions of people are having YouTube and Netflix issues on fast connections insists "it's complicated." So what's really happening? One theory du jour is that companies like Verizon are abusing their gatekeeper position and letting their peering points to companies like Netflix saturate in order to gain leverage in negotiations and rhetoric, while giving a leg up to their own CDN or services (like Red Box Instant by Verizon). Verizon's certainly no stranger to these kinds of tactics, but again, there's not enough data to prove that hypothesis, and while neutrality rules are dead, big ISPs aren't exactly eager to do anything to invite new neutrality regulations.

The reality is that despite the best guesses of some very smart people, the jury is out because we lack enough data to make full conclusions just yet. The good news is we should see some more robust data in the coming months as data from collection operations like MLAB comes to light in looming FCC ISP performance reports. In addition, YouTube has started ranking ISPs based on YouTube streaming performance, which will also hopefully shed a little more light on precisely which Internet bogeymen are to blame for your stuttering video feed.

Thank you for reading this Techdirt post. With so many things competing for everyone’s attention these days, we really appreciate you giving us your time. We work hard every day to put quality content out there for our community.

Techdirt is one of the few remaining truly independent media outlets. We do not have a giant corporation behind us, and we rely heavily on our community to support us, in an age when advertisers are increasingly uninterested in sponsoring small, independent sites — especially a site like ours that is unwilling to pull punches in its reporting and analysis.

While other websites have resorted to paywalls, registration requirements, and increasingly annoying/intrusive advertising, we have always kept Techdirt open and available to anyone. But in order to continue doing so, we need your support. We offer a variety of ways for our readers to support us, from direct donations to special subscriptions and cool merchandise — and every little bit helps. Thank you.

–The Techdirt Team

Filed Under: broadband, david raphael, fcc, net neutrality
Companies: at&t, comcast, netflix, verizon, youtube

43 Comments

If you liked this post, you may also be interested in...

Reader Comments

Subscribe: RSS

View by: Time | Thread

Anonymous Coward, 11 Feb 2014 @ 2:30pm

did you expect them to own up and openly say that they are not throttling, they have just reduced the available bandwidth allotted to customers using Netflix!
[ link to this | view in thread ]
Anonymous Coward, 11 Feb 2014 @ 2:34pm

I'd be willing to bet its some kind of routing issue.

I use Verizon and have seen absolutely ludicrous IP routing that often leads to drastically reduced speeds.

Once while transferring a file between my house and a friend about 30 miles across town, I noticed extremely low speeds (we both have 50+ Mbps normally). I tried a trace-route and saw packets being routed through Atlanta, Washington D.C. and Chicago... and we are both in Florida... data was being sent over thousands of miles to travel between two computers in the same city.
[ link to this | view in thread ]
TKnarr (profile), 11 Feb 2014 @ 2:39pm

This may backfire on the ISPs
Users don't buy Internet service for the ISP. They buy it for everything else out there. The ISP's service is just the pipe as far as most people are concerned. If the ISPs degrade service too far, people will start looking for another ISP to get their pipe from.

The major choices for broadband here in San Diego are Cox, Time Warner and AT&T. TW and AT&T ought to worry that, when looking at houses I could possibly buy, one of my criteria is "located in an area serviced by Cox" because I just don't want to deal with the ongoing headaches I'm sure to have with the other two.
[ link to this | view in thread ]
Karl Bode (profile), 11 Feb 2014 @ 2:40pm

Netflix has commented
Worth adding this investor analyst note that claims Netflix doesn't believe Verizon is intentionally hampering their streams:

http://recode.net/2014/02/11/netflix-says-verizon-isnt-slowing-down-its-streams/

so far, that�s not happening, Netflix says.

That update comes to us via a note from J.P. Morgan analyst Doug Anmuth, who says he has been talking to Netflix CEO Reed Hastings and CFO David Wells, and they told him they don�t think cable and telco companies are hampering the company�s video streams.

Anmuth doesn�t have much to report on the topic, so here are his comments in their entirety: �Netflix does not seem overly concerned regarding Net Neutrality, and continues to believe that violations would be escalated quickly. Netflix also indicated that it has no evidence or belief that its service is being throttled.�
[ link to this | view in thread ]
DannyB (profile), 11 Feb 2014 @ 2:44pm

Re:
It may be a routing issue.

But that doesn't mean it isn't deliberate.
[ link to this | view in thread ]
Baron von Robber, 11 Feb 2014 @ 2:44pm

There is a way to tell...
Get a VPN service.
Try it with and without.
If it's bout the same, they are not degrading Netflix/Youtube.

If there is a difference, they are.
[ link to this | view in thread ]
Anonymous Coward, 11 Feb 2014 @ 2:56pm

Re: Re:
Makes sense...

Verizon's routers likely automatically send "Low priority" traffic on the least efficient route.

Their packet inspector probably mistook my ssh session for bittorrent or something.
[ link to this | view in thread ]
Anonymous Coward, 11 Feb 2014 @ 2:57pm

Re: This may backfire on the ISPs
You have one maybe two ISPs in most areas and with places like Kentucky pushing legislation to make city/county based ISPs illegal, or extremely difficult to establish or continue on, there really isn't anywhere else to go. Its like politics, you got a red pile of crap and a blue pile of crap, and no matter the color you have a pile of crap.
[ link to this | view in thread ]
Anonymous Coward, 11 Feb 2014 @ 2:58pm

Re: There is a way to tell...
What if they also degrade VPN connections?
[ link to this | view in thread ]
Baron von Robber, 11 Feb 2014 @ 3:02pm

Re: Re: There is a way to tell...
That would be messed up. VPN does have overhead so all things being equal, there would be a little slowdown.
But to throttle that as well?

Time to go back to dialup =b
[ link to this | view in thread ]
Baron von Robber, 11 Feb 2014 @ 3:04pm

Re: Re: Re: There is a way to tell...
Forgot to mention, over on Ars in the comment section on the same topic, some have tried this and miracously, with VPN, their Netflix/Youtube got faster. :/
[ link to this | view in thread ]
Anonymous Coward, 11 Feb 2014 @ 3:10pm

Re: Re: Re: Re: There is a way to tell...
If VPN traffic goes up they will start forwarding it through the NSA super computer to decrypt the packets and determine if you're using in to get Netflix and then throttle all of your connections to 128kbps
[ link to this | view in thread ]
Anonymous Coward, 11 Feb 2014 @ 3:13pm

Re: Re: Re: Re: There is a way to tell...
Honestly, I would not be surprised if that is due to routing.

As per my previous post above, bad routing can introduce serious connection slowdown. By going through a VPN, assuming you have a decent connection to it, you can bypass misconfigured or malicious routers ISP side.

Its not entirely unreasonable for an ISP to use bad routing as a way of throttling. That way they can claim to not be "throttling" a connection, and they are technically not... they just route your packets to mars...
[ link to this | view in thread ]
Anonymous Coward, 11 Feb 2014 @ 3:13pm

Re: Re: Re: Re: Re: There is a way to tell...
To fight terrorisms and such
[ link to this | view in thread ]
Anonymous Coward, 11 Feb 2014 @ 3:14pm

bullshit. the ISPs are deliberately throttling youtube. here in germany it is known fact. Telekom, the biggest provider does deliberately reduce the bandwith because some "issues" with peering agreements, meaning telekom wants to be paid extra for that stuff on top of the regular agreements. Of course that doesn't fly and the customers have to suffer for it.

so yeah, no mystery here, just plain greed.
[ link to this | view in thread ]
Mark Gisleson (profile), 11 Feb 2014 @ 3:19pm

More anecdotal evidence
I moved to a remote rural location with no cable or dish options available. I had to set up a Verizon Home Fusion device. It is amazing. It can download the Library of Congress in about half an hour. OK, 30mbps isn't that fast, but at $5 a GB I had to dump my Netflix account because I couldn't afford the bandwidth it used.

That and even at 30 mbps, Netflix was choppy and unwatchable. All other streaming video was better, including illegal NBA feeds.

State of the art wireless technology, but it couldn't stream Netflix worth a crap.

There's something else going on here.
[ link to this | view in thread ]
RonKaminsky (profile), 11 Feb 2014 @ 3:20pm

Re: Re: Re:
Hmm, "ssh"... "Washington D.C."...

Brings to mind: NSA. You probably share an ISP and similar location to someone who's connected to someone who's suspicious, eh?
[ link to this | view in thread ]
Karl Bode (profile), 11 Feb 2014 @ 3:24pm

Re: More anecdotal evidence
I've been on FiOS for five years, and about 50% of the Time YouTube streams are entirely unwatchable.
[ link to this | view in thread ]
Gabriel J. Michael (profile), 11 Feb 2014 @ 3:24pm

Disingenuous
I also love the bit in Verizon's response about how since speed tests show people getting higher than advertised speeds, their customers are getting "more than they paid for" and presumably should STFU.

It's like saying that since you can go 80 mph on the Beltway at 3 AM, you shouldn't complain about traffic.
[ link to this | view in thread ]
Anonymous Coward, 11 Feb 2014 @ 3:26pm

Re: Re: Re: Re:
Oh noes!
[ link to this | view in thread ]
Anonymous Coward, 11 Feb 2014 @ 3:28pm

It would seem that much of Netflix's success was a result of the actions of ISPs' -- by throttling Bittorrent through both active packet-shaping and by extremely restricting the upstream bandwidth.

If restricting Bittorrent traffic was justified by the ISPs for business reasons, why should those same reasons not apply to Netflix, which is a much bigger bandwidth hog?
[ link to this | view in thread ]
John Fenderson (profile), 11 Feb 2014 @ 3:36pm

You have choices?
At my place, my choices for broadband ISPs consist of Comcast, Comcast, and Comcast.

Consider yourself very lucky indeed!
[ link to this | view in thread ]
Mark Milliman, 11 Feb 2014 @ 3:38pm

Traffic Equality
The article states the problem very clearly. The problem is that all traffic is treated equal. For better or for worse, all traffic IS NOT EQUAL. Voice is more sensitive than video that is more sensitive than all else. Latency and jitter is increased in port queues that will impact the end performance of a service. By treating all traffic the same your voice and video is bound to be impaired no matter how much bandwidth is thrown at it. As long as you people insist that Net Neutrality means treating all traffic the same, you will be eternally searching for the boogie man.
[ link to this | view in thread ]
Beech, 11 Feb 2014 @ 4:46pm

Really,
If Verizon didn't want people thinking they were violating net neutrality, they shouldn't have fought a lawsuit over their rights to violate net neutrality. They opened Pandora's Box, and now every time one of their customers has a hiccup in internet speed they're going to blame it on throttling. If Verizon really cared so much about "treating all traffic equally" they should have just settled the lawsuit by saying "okay". Problem solved.
[ link to this | view in thread ]
AnonCow, 11 Feb 2014 @ 5:13pm

I trust Verizon and Comcast with packets like I trust Jamie Dimon with my grandmother's life savings.
[ link to this | view in thread ]
Anonymous Coward, 11 Feb 2014 @ 5:22pm

Re: Re: Re: There is a way to tell...
Take a close look at the terms of service your ISP gives you. I don't think it's been enforced all that often, but some of them don't allow VPN connections on residential accounts.

So if that becomes a common solution to video streaming, degrading them might not be necessary.
[ link to this | view in thread ]
Anon77, 11 Feb 2014 @ 5:24pm

Re: Re: Re: Re: Re: There is a way to tell...
Route all traffic over a peering point that cant handle that much traffic and you have effectively throttled the traffic. Yet you can say it's not throttled as you haven't used your throttling specific hardware.

They are being crafty.

virgin media in the UK had terrible youtube streaming for a long time and blamed 3rd party suppliers. These were in fact a local youtube CDN attached to their own network yet they still didn't give it enough bandwidth to run correctly.

bottom line is never trust an isp, they will screw customers over any way they can
[ link to this | view in thread ]
rapnel, 11 Feb 2014 @ 5:44pm

Re: Re: Re: Re: There is a way to tell...
Not allowing VPN connections on residential accounts would be preposterous. That's a direct attack on your ability to communicate privately and in private. Unacceptable and not only with regards to any supposed terms of service.
[ link to this | view in thread ]
Anonymous Coward, 11 Feb 2014 @ 6:11pm

Re: There is a way to tell...
That's incorrect. Let's say Verizon uses Cogent to get to AWS, and your VPN uses Level3 to get to the VPN point A and it's bandwidth to get to AWS. If Verizon is jamming the link to Cogent, your VPN could have less latency to AWS and better performance.
The sad fact is, most CDNs like Netflix would gladly peer with Verizon for Free, but it's in the dual providers best interest not to, ie Verizon and RedBox.
Peering agreements are a lag on the internet, and sadly most network engineers understand this, that is why it's important to peer with as many providers as possible and keep those links from being saturated.

If you really want to test this out, I would suggest first using mtr with both the vpn and without and seeing where the loss of packets is occuring and high latency points. It's not full proof but with enough data points and also using looking glass sites, you could narrow down the link with the congestion. Given if you are a residential user not announcing BGP with multiple providers, good luck finding someone to listen.
[ link to this | view in thread ]
Chronno S. Trigger (profile), 11 Feb 2014 @ 6:11pm

The problem with residential connections
This guy states a few facts that suggest a vary possible reason for the discrepancy.

Business connections are dedicated connections, you don't share the bandwidth. With a residential connection, you do.

Then we can probably safely assume that his business testing was done during business hours and home testing was done after. I have found that during the day my connection to Youtube is amazing, 1080p video, but right around 3:30pm every single day I can't watch the 144p videos. It picks back up again after 11:30pm. Took me a little while to figure it out, but it turns out that the connection would start sucking right as the kids started getting home from school. Weekends, summer, and the random snow days didn't have this problem. The kids would come home, play games, watch videos, and kill the internet connection for the entire neighborhood.

The "Deep, dark mysteries of the Internet" are a damn good excuse because they're true. We're going to need something much more then random ping tests to prove Verizon is throttling.
[ link to this | view in thread ]
Anonymous Coward, 11 Feb 2014 @ 6:42pm

Latency
A few people have touched on this already, latency is the real killer here not your Mbps.

Q. Whats faster?
A) 100Mbps connection with 1.0 Second of latency
B) 50Mbps connection with 0.5 seconds of latency?

A. Both are the same.

How the data is routed will have a huge impact on latency.
From the data center at work my latency is minimal to nearly everywhere, this is because we are only one hop away from several major providers. (Level3, TW Telecom, XO, Time Warner Cable, AT&T, Hurricane and many more.)

From home on my cable modem the latency on just my ISP's routers is terrible but oddly the latency to their speed test server is minimal so there is "nothing wrong"
[ link to this | view in thread ]
MikeC (profile), 11 Feb 2014 @ 8:30pm

Ping of NetFlix.Com -- via VPN/Corporate Network vs Via Comcast Direct within 2 minutes of each other

[VPN]

7 59 ms 60 ms 60 ms 206.111.0.65.ptr.us.xo.net [206.111.0.65]
8 135 ms 125 ms 132 ms vb2000d2.rar3.washington-dc.us.xo.net [207.88.13.66]
9 135 ms 141 ms 140 ms te-11-0-0.rar3.sanjose-ca.us.xo.net [207.88.12.69]
10 149 ms 136 ms 131 ms 207.88.14.226.ptr.us.xo.net [207.88.14.226]
11 126 ms 132 ms 129 ms 216.156.84.6.ptr.us.xo.net [216.156.84.6]
12 142 ms 141 ms 138 ms xe-2-2-0-955.jnrt-edge02.prod1.netflix.com [69.53.225.30]
13 140 ms 136 ms 133 ms te1-8.csrt-agg02.prod1.netflix.com [69.53.225.10]
14 136 ms 138 ms 141 ms netflix.co.uk [69.53.236.17]

[DIRECT]

7 28 ms 30 ms 30 ms he-3-2-0-0-cr01.ashburn.va.ibone.comcast.net [68.86.90.177]
8 27 ms 28 ms 26 ms he-0-12-0-0-pe07.ashburn.va.ibone.comcast.net [68.86.83.82]
9 84 ms 32 ms 34 ms dcx2-edge-01.inet.qwest.net [65.120.84.65]
10 * * * Request timed out.
11 104 ms 102 ms 102 ms 63.145.225.58
12 102 ms 103 ms 101 ms te1-8.csrt-agg01.prod1.netflix.com [69.53.225.6]
13 102 ms 102 ms 102 ms netflixinc.com [69.53.236.17]

First 6 entries almost identical just local hops, but when it get's handed off to the backbone it gets interesting. Note the different destinations depending on the backbone when it get to the same IP address but different domains resolved based on the backbone it rode? Not all that surprising but also when I route local via comcast - look at hop 10 when it's handed off to Qwest (not surprising in my experience with Qwest networks).

The VPN definitely increases the latency across the board as expected. But the fact that netflix is resolving different FQDN's to the same IP from different aggregate ports in their network, a touch weird but then I don't work much with networks with that kind of capacity.

Ever wonder why they call it a CLOUD?? Fully opaque, easy to get lost, only god/buddha/allah know how it works and they aren't really sure. So as much as I hate big ISP's I never ascribe malice to what most likely is based in incompetence.
[ link to this | view in thread ]
fred, 11 Feb 2014 @ 9:05pm

maybe
Has anyone considered that perhaps the demand for those services has grown too quickly? Peering is not unlimited, and the amount of total bandwidth cannot increase endlessly, at least not without adding more connections, which costs.

Also, it depends on every step from provider to destination, and the increases in demand now are likely very much like the early days or torrents. Isps cannot add connectivity that quickly to meet a sudden spike in user demand.
[ link to this | view in thread ]
techflaws (profile), 11 Feb 2014 @ 10:39pm

Re:
Was it justfied? What do consumers like to pay for? Restricted Internet or free?
[ link to this | view in thread ]
DannyB (profile), 12 Feb 2014 @ 6:36am

Re: Re: Re:
Maybe it's not a least efficient route. Maybe it's a route that passes through an NSA inspection checkpoint.

Hey that Netflix traffic is encrypted! Could be terrorists.
[ link to this | view in thread ]
DannyB (profile), 12 Feb 2014 @ 6:40am

Re: Traffic Equality
Yes, latency matters. For interactive protocols.

But good bandwidth is sufficient for video streaming applications like Netflix, YouTube, etc. It doesn't matter if your video stream is seconds behind the packets leaving the server, as long as there is bandwidth for a continuous flow of packets to your screen and speakers.
[ link to this | view in thread ]
John Fenderson (profile), 12 Feb 2014 @ 9:12am

Re: The problem with residential connections
Well said.

There's a million "gotchas" like this. It's premature to accuse anybody of being a bad actor without more comprehensive analysis and testing.
[ link to this | view in thread ]
zip, 12 Feb 2014 @ 10:34am

Re: Latency
Latency means little to downloading/uploading applications that use multiple connections/streams, because when one connection is idly waiting for a packet acknowledgment, the other connections are busy sucking down maximum bandwidth.
[ link to this | view in thread ]
Anonymous Coward, 12 Feb 2014 @ 1:48pm

its not about a "major" isp's, .....what do "major" anythings usually have?

Alot of customers, .....thats why they are most likely the first to do something not in favour of the people, why would the "pressures that be" pressure a ISP with 500 users, when they want to "affect", the 2 million market, the bigger a WELL INTENTIONED company gets, the bigger the pressures they recieve, the harder they have to keep their original course.........and this is no excuse for them to FAIL this, because they have 2million damn customers, that alone means any bad decisions is MAGNIFIED.......i dont just want to support a top quality service, i want to support a MORAL one, .......you can go on and assume that is my damn fucking line in the sand........no morals, then piss off with whatever it is your shoveling

Not had me coffee
[ link to this | view in thread ]
John85851 (profile), 12 Feb 2014 @ 2:50pm

But they don't say they're not doing it
Some commenters at Ars make a good point: the statement from Verizon basically says their network is operating as expected, but they don't specifically say they aren't throttling Netflix. Why don't they "state unequivocally" that they're not doing it?
[ link to this | view in thread ]
Anonymous Coward, 12 Feb 2014 @ 10:53pm

Re: Re: Re: Re:
Wow, a lot of people share my IP, because of the way my shared vps provider works. I can confirm we're all evil because of that.
[ link to this | view in thread ]
The Secret Tor Files, 11 Dec 2014 @ 8:00am

The Truth
The sad truth is if your running Netflix or YouTube, you are getting throttled, which you are you can barely get 320p video quality. Yet somehow if you suddenly set up a vpn and run YouTube you can get 1080 without a problem, its now a sad reality that these companies disregard net neutrality for profit.
[ link to this | view in thread ]
Annie lobo, 28 Nov 2016 @ 1:53am

Netflix
I use to stream movies via NetFlix, the video quality is really good but it's pretty expensive for me. Thus, I just went through your write up, it's really useful. I have tried few websites given, the stock of content is amazing to watch on them. Thank you for the help!
[ link to this | view in thread ]