Netflix, YouTube Streaming Gets Worse on Major ISPs, Who In Turn Blame the Deep, Dark Mysteries of the Internet
from the what-the-hell-is-going-on dept
Last week this blog entry by iScan developer David Raphael saw a lot of attention as proof positive that Verizon was violating network neutrality. Specifically, Raphael observed that services using Amazon Web Services accessed from his home FiOS connection (including Netflix) delivered bits more slowly than services accessed on his business FiOS connection. As his supposed nail in the neutrality-violation coffin, Raphael provided a chat transcript from a Verizon rep supposedly "confirming" that the company was throttling select services. This "proof" then bumbled and stumbled its way around the Internet as evidence neutrality had been trampled.Except as anybody who has spent any time on the phone with a support rep knows, what comes out of their mouths may or may not have any direct tie to this particular plane of reality. It's also worth noting that the kind of Netflix stuttering Raphael saw has been going on for much of the last year across numerous ISPs, long before Verizon's recent court victory over the FCC's neutrality rules. Verizon also denied they were manipulating traffic in a statement sent to all media outlets:
"We treat all traffic equally, and that has not changed. Many factors can affect the speed a customer's experiences for a specific site, including, that site’s servers, the way the traffic is routed over the Internet, and other considerations. We are looking into this specific matter, but the company representative was mistaken. We are going to redouble our representative education efforts on this topic."While Raphael may have stumbled into Verizon shenanigans, what he saw could also be explained by normal peak network or peering congestion, and we've long discussed the problems with crying net neutrality wolf. Peering relationships are indeed complicated, with ISPs, content companies, and Internet backbone operators all engaged in an endless tug of war over video profits and peering imbalances (I recommend this piece last summer from Jon Brodkin at Ars Technica or this piece from last week by GigaOM's Stacey Higginbotham).
With that as background, fast forward to this week when Netflix released their latest ISP streaming rankings, which show that most of the nation's biggest ISPs have been dropping in regards to their Netflix streaming performance. ISPs that partner with Netflix's Open Connect CDN do better in the rankings, and major ISPs have tried somewhat feebly to claim Netflix's attempt to name and shame them into using their CDN is a sort of reverse network neutrality.
Still, Verizon FiOS (and DSL), Comcast, and AT&T's U-Verse services have seen major drops in the rankings not entirely explained by their refusal to use Netflix's CDN. Being large network operators, you would imagine that they'd be able to explain what exactly is happening to stink up millions of broadband users' Netflix experiences, though Comcast, AT&T and Time Warner Cable refuse to comment to the media on their collective performance nose dive. Netflix also refuses to comment. Verizon, with all of their network expertise, continues to blame the problems on either unknown Internet bogeymen or Netflix:
"We state unequivocally that Verizon’s broadband Internet access services deliver a pristine user experience to our customers at any time of day on every day of the week. This has been repeatedly proven through independent testing by the Federal Communications Commission (FCC), which has conclusively demonstrated that FiOS Internet consistently delivers both download and upload speeds in excess of what we advertise. In short, our Internet customers often get more than they pay for. How the Internet works can be complicated, and consumers should be aware of the fact that the integrity of their home Internet connection is only a portion of the streaming video quality equation. If their broadband connection is functioning correctly, the source of their frustration and the content they wish to see may be one in the same."In summation, the only major ISP wiling to comment on why millions of people are having YouTube and Netflix issues on fast connections insists "it's complicated." So what's really happening? One theory du jour is that companies like Verizon are abusing their gatekeeper position and letting their peering points to companies like Netflix saturate in order to gain leverage in negotiations and rhetoric, while giving a leg up to their own CDN or services (like Red Box Instant by Verizon). Verizon's certainly no stranger to these kinds of tactics, but again, there's not enough data to prove that hypothesis, and while neutrality rules are dead, big ISPs aren't exactly eager to do anything to invite new neutrality regulations.
The reality is that despite the best guesses of some very smart people, the jury is out because we lack enough data to make full conclusions just yet. The good news is we should see some more robust data in the coming months as data from collection operations like MLAB comes to light in looming FCC ISP performance reports. In addition, YouTube has started ranking ISPs based on YouTube streaming performance, which will also hopefully shed a little more light on precisely which Internet bogeymen are to blame for your stuttering video feed.
Thank you for reading this Techdirt post. With so many things competing for everyone’s attention these days, we really appreciate you giving us your time. We work hard every day to put quality content out there for our community.
Techdirt is one of the few remaining truly independent media outlets. We do not have a giant corporation behind us, and we rely heavily on our community to support us, in an age when advertisers are increasingly uninterested in sponsoring small, independent sites — especially a site like ours that is unwilling to pull punches in its reporting and analysis.
While other websites have resorted to paywalls, registration requirements, and increasingly annoying/intrusive advertising, we have always kept Techdirt open and available to anyone. But in order to continue doing so, we need your support. We offer a variety of ways for our readers to support us, from direct donations to special subscriptions and cool merchandise — and every little bit helps. Thank you.
–The Techdirt Team
Filed Under: broadband, david raphael, fcc, net neutrality
Companies: at&t, comcast, netflix, verizon, youtube
Reader Comments
Subscribe: RSS
View by: Time | Thread
[ link to this | view in chronology ]
I use Verizon and have seen absolutely ludicrous IP routing that often leads to drastically reduced speeds.
Once while transferring a file between my house and a friend about 30 miles across town, I noticed extremely low speeds (we both have 50+ Mbps normally). I tried a trace-route and saw packets being routed through Atlanta, Washington D.C. and Chicago... and we are both in Florida... data was being sent over thousands of miles to travel between two computers in the same city.
[ link to this | view in chronology ]
Re:
But that doesn't mean it isn't deliberate.
[ link to this | view in chronology ]
Re: Re:
Verizon's routers likely automatically send "Low priority" traffic on the least efficient route.
Their packet inspector probably mistook my ssh session for bittorrent or something.
[ link to this | view in chronology ]
Re: Re: Re:
Brings to mind: NSA. You probably share an ISP and similar location to someone who's connected to someone who's suspicious, eh?
[ link to this | view in chronology ]
Re: Re: Re: Re:
[ link to this | view in chronology ]
Re: Re: Re: Re:
[ link to this | view in chronology ]
Re: Re: Re:
Hey that Netflix traffic is encrypted! Could be terrorists.
[ link to this | view in chronology ]
This may backfire on the ISPs
The major choices for broadband here in San Diego are Cox, Time Warner and AT&T. TW and AT&T ought to worry that, when looking at houses I could possibly buy, one of my criteria is "located in an area serviced by Cox" because I just don't want to deal with the ongoing headaches I'm sure to have with the other two.
[ link to this | view in chronology ]
Re: This may backfire on the ISPs
[ link to this | view in chronology ]
You have choices?
Consider yourself very lucky indeed!
[ link to this | view in chronology ]
Netflix has commented
http://recode.net/2014/02/11/netflix-says-verizon-isnt-slowing-down-its-streams/
[ link to this | view in chronology ]
There is a way to tell...
Try it with and without.
If it's bout the same, they are not degrading Netflix/Youtube.
If there is a difference, they are.
[ link to this | view in chronology ]
Re: There is a way to tell...
[ link to this | view in chronology ]
Re: Re: There is a way to tell...
But to throttle that as well?
Time to go back to dialup =b
[ link to this | view in chronology ]
Re: Re: Re: There is a way to tell...
[ link to this | view in chronology ]
Re: Re: Re: Re: There is a way to tell...
[ link to this | view in chronology ]
Re: Re: Re: Re: Re: There is a way to tell...
[ link to this | view in chronology ]
Re: Re: Re: Re: There is a way to tell...
As per my previous post above, bad routing can introduce serious connection slowdown. By going through a VPN, assuming you have a decent connection to it, you can bypass misconfigured or malicious routers ISP side.
Its not entirely unreasonable for an ISP to use bad routing as a way of throttling. That way they can claim to not be "throttling" a connection, and they are technically not... they just route your packets to mars...
[ link to this | view in chronology ]
Re: Re: Re: Re: Re: There is a way to tell...
They are being crafty.
virgin media in the UK had terrible youtube streaming for a long time and blamed 3rd party suppliers. These were in fact a local youtube CDN attached to their own network yet they still didn't give it enough bandwidth to run correctly.
bottom line is never trust an isp, they will screw customers over any way they can
[ link to this | view in chronology ]
Re: Re: Re: There is a way to tell...
So if that becomes a common solution to video streaming, degrading them might not be necessary.
[ link to this | view in chronology ]
Re: Re: Re: Re: There is a way to tell...
[ link to this | view in chronology ]
Re: There is a way to tell...
The sad fact is, most CDNs like Netflix would gladly peer with Verizon for Free, but it's in the dual providers best interest not to, ie Verizon and RedBox.
Peering agreements are a lag on the internet, and sadly most network engineers understand this, that is why it's important to peer with as many providers as possible and keep those links from being saturated.
If you really want to test this out, I would suggest first using mtr with both the vpn and without and seeing where the loss of packets is occuring and high latency points. It's not full proof but with enough data points and also using looking glass sites, you could narrow down the link with the congestion. Given if you are a residential user not announcing BGP with multiple providers, good luck finding someone to listen.
[ link to this | view in chronology ]
so yeah, no mystery here, just plain greed.
[ link to this | view in chronology ]
More anecdotal evidence
That and even at 30 mbps, Netflix was choppy and unwatchable. All other streaming video was better, including illegal NBA feeds.
State of the art wireless technology, but it couldn't stream Netflix worth a crap.
There's something else going on here.
[ link to this | view in chronology ]
Re: More anecdotal evidence
[ link to this | view in chronology ]
Disingenuous
It's like saying that since you can go 80 mph on the Beltway at 3 AM, you shouldn't complain about traffic.
[ link to this | view in chronology ]
If restricting Bittorrent traffic was justified by the ISPs for business reasons, why should those same reasons not apply to Netflix, which is a much bigger bandwidth hog?
[ link to this | view in chronology ]
Re:
[ link to this | view in chronology ]
Traffic Equality
[ link to this | view in chronology ]
Re: Traffic Equality
But good bandwidth is sufficient for video streaming applications like Netflix, YouTube, etc. It doesn't matter if your video stream is seconds behind the packets leaving the server, as long as there is bandwidth for a continuous flow of packets to your screen and speakers.
[ link to this | view in chronology ]
Really,
[ link to this | view in chronology ]
[ link to this | view in chronology ]
The problem with residential connections
Business connections are dedicated connections, you don't share the bandwidth. With a residential connection, you do.
Then we can probably safely assume that his business testing was done during business hours and home testing was done after. I have found that during the day my connection to Youtube is amazing, 1080p video, but right around 3:30pm every single day I can't watch the 144p videos. It picks back up again after 11:30pm. Took me a little while to figure it out, but it turns out that the connection would start sucking right as the kids started getting home from school. Weekends, summer, and the random snow days didn't have this problem. The kids would come home, play games, watch videos, and kill the internet connection for the entire neighborhood.
The "Deep, dark mysteries of the Internet" are a damn good excuse because they're true. We're going to need something much more then random ping tests to prove Verizon is throttling.
[ link to this | view in chronology ]
Re: The problem with residential connections
There's a million "gotchas" like this. It's premature to accuse anybody of being a bad actor without more comprehensive analysis and testing.
[ link to this | view in chronology ]
Latency
Q. Whats faster?
A) 100Mbps connection with 1.0 Second of latency
B) 50Mbps connection with 0.5 seconds of latency?
A. Both are the same.
How the data is routed will have a huge impact on latency.
From the data center at work my latency is minimal to nearly everywhere, this is because we are only one hop away from several major providers. (Level3, TW Telecom, XO, Time Warner Cable, AT&T, Hurricane and many more.)
From home on my cable modem the latency on just my ISP's routers is terrible but oddly the latency to their speed test server is minimal so there is "nothing wrong"
[ link to this | view in chronology ]
Re: Latency
[ link to this | view in chronology ]
[VPN]
7 59 ms 60 ms 60 ms 206.111.0.65.ptr.us.xo.net [206.111.0.65]
8 135 ms 125 ms 132 ms vb2000d2.rar3.washington-dc.us.xo.net [207.88.13.66]
9 135 ms 141 ms 140 ms te-11-0-0.rar3.sanjose-ca.us.xo.net [207.88.12.69]
10 149 ms 136 ms 131 ms 207.88.14.226.ptr.us.xo.net [207.88.14.226]
11 126 ms 132 ms 129 ms 216.156.84.6.ptr.us.xo.net [216.156.84.6]
12 142 ms 141 ms 138 ms xe-2-2-0-955.jnrt-edge02.prod1.netflix.com [69.53.225.30]
13 140 ms 136 ms 133 ms te1-8.csrt-agg02.prod1.netflix.com [69.53.225.10]
14 136 ms 138 ms 141 ms netflix.co.uk [69.53.236.17]
[DIRECT]
7 28 ms 30 ms 30 ms he-3-2-0-0-cr01.ashburn.va.ibone.comcast.net [68.86.90.177]
8 27 ms 28 ms 26 ms he-0-12-0-0-pe07.ashburn.va.ibone.comcast.net [68.86.83.82]
9 84 ms 32 ms 34 ms dcx2-edge-01.inet.qwest.net [65.120.84.65]
10 * * * Request timed out.
11 104 ms 102 ms 102 ms 63.145.225.58
12 102 ms 103 ms 101 ms te1-8.csrt-agg01.prod1.netflix.com [69.53.225.6]
13 102 ms 102 ms 102 ms netflixinc.com [69.53.236.17]
First 6 entries almost identical just local hops, but when it get's handed off to the backbone it gets interesting. Note the different destinations depending on the backbone when it get to the same IP address but different domains resolved based on the backbone it rode? Not all that surprising but also when I route local via comcast - look at hop 10 when it's handed off to Qwest (not surprising in my experience with Qwest networks).
The VPN definitely increases the latency across the board as expected. But the fact that netflix is resolving different FQDN's to the same IP from different aggregate ports in their network, a touch weird but then I don't work much with networks with that kind of capacity.
Ever wonder why they call it a CLOUD?? Fully opaque, easy to get lost, only god/buddha/allah know how it works and they aren't really sure. So as much as I hate big ISP's I never ascribe malice to what most likely is based in incompetence.
[ link to this | view in chronology ]
maybe
Also, it depends on every step from provider to destination, and the increases in demand now are likely very much like the early days or torrents. Isps cannot add connectivity that quickly to meet a sudden spike in user demand.
[ link to this | view in chronology ]
Alot of customers, .....thats why they are most likely the first to do something not in favour of the people, why would the "pressures that be" pressure a ISP with 500 users, when they want to "affect", the 2 million market, the bigger a WELL INTENTIONED company gets, the bigger the pressures they recieve, the harder they have to keep their original course.........and this is no excuse for them to FAIL this, because they have 2million damn customers, that alone means any bad decisions is MAGNIFIED.......i dont just want to support a top quality service, i want to support a MORAL one, .......you can go on and assume that is my damn fucking line in the sand........no morals, then piss off with whatever it is your shoveling
Not had me coffee
[ link to this | view in chronology ]
But they don't say they're not doing it
[ link to this | view in chronology ]
The Truth
[ link to this | view in chronology ]
Netflix
[ link to this | view in chronology ]