Without Greater Transparency And Meaningful Net Neutrality Rules, The Netflix Interconnection War Will Get Much, Much Uglier

from the sunlight-is-the-best-disinfectant dept

Fri, Nov 7th 2014 11:13am — Karl Bode

If you've followed the ongoing feud between Netflix and the nation's biggest ISPs, you'll recall that streaming performance on the nation's four biggest ISPs (AT&T, Verizon, Comcast and Time Warner Cable) mysteriously started to go to hell earlier this year, and was only resolved once Netflix acquiesced to ISP demands to bypass transit partners and pay carriers for direct interconnection. Though the FCC has refused to include interconnection in their consideration of new net neutrality rules, we've noted how it's really the edges of these networks where the biggest neutrality battles are now being waged.

While Netflix's deals did resolve the congestion issues for users on these ISPs this past fall, the debate over this practice has since broken down into two distinct camps. One is Netflix, its transit partners, and consumer advocacy groups, who claim that the nation's biggest ISPs have intentionally let key transit points degrade, intentionally creating significant congestion and forcing Netflix to pay them directly if they want their video services to work. On the other side is the mega-ISPs, their PR folks and a smattering of industry consultants, who argue that Netflix is to blame by choosing Cogent as a cheaper transit link to these carriers.

The problem is that because so much of this data is confidential and ISPs aggressively prevent any meaningful access to raw network data (in part because congestion has always been used as a policy bogeyman to justify anti-competitive behavior and usage caps), clearly illustrating who is at blame here can be (quite intentionally) difficult.

However, as we noted late last month, a new study from M-Labs (an organization ISPs have historically tried to hamstring) clarified at least a few key points in the debate. One, the report put to bed (or should have) the mega-ISP claim that Netflix was intentionally choosing congested partners like Cogent to save a buck, noting that other mid-sized ISPs not pushing for direct interconnection deals (like Cablevision) were seeing no congestion over Cogent connections to Netflix. It also made something very clear: natural congestion wasn't responsible for the degrading of user Netflix traffic: it was a conscious ISP business decision for users to have degraded experiences.

Now to be clear there are no saints here, as Netflix, transit partners and last mile ISPs are all running face-first at the video cash trough. Also to be clear, M-Lab went out of its way to avoid naming ISPs specifically in the report, in part to avoid liability (remember Verizon threatened to sue Netflix for even suggesting the problem could be on Verizon's end), but also because the data proving culpability isn't yet complete.

Now if you've followed the adventures of AT&T, Comcast and Verizon over the years, it's very easy to come to some conclusions based on their long, proud history of anti-competitive behavior. Since most of the data is obscured, intentionally degrading transit connection points is a brilliant way to grab the extra cash from content companies through the kind of troll tolls they've long talked about. The FCC has requested confidential data from all parties and is investigating Netflix's claims, but it's not yet possible to completely prove shenanigans without stepping foot in the very rooms that provide interconnection and examining both the hardware and the data.

If the M-Lab study pushes the needle in any direction, it's indisputably toward the ISPs being at fault. Still, it has been entertaining to see some folks on the ISP side of the argument continue to declare Netflix the villain. Though that M-Lab study pretty clearly illustrates that Netflix's choice of Cogent wasn't the core problem (again because Cablevision saw no Cogent congestion), Frost and Sullivan analyst Dan Rayburn magically comes to the complete opposite conclusion. When Cogent this week acknowledged that the company had utilized some traffic management to try and improve Netflix streaming issues earlier this year, that was enough evidence for Rayburn to again declare Netflix the bad guy:

"This morning, Cogent admitted that in February and March of this year the company put in place a procedure that favored traffic on their network, putting a QoS structure in place, based on the type of content being delivered. Without telling anyone, Cogent created at least two priority levels (a ‘fast lane’ and ‘slow lane’), and possibly more, and implemented them at scale in February of this year. What Cogent did is considered a form of network management and was done without them disclosing it, even though it was the direct cause of many of the earlier published congestion charts and all the current debates.

In short, everything is Cogent and Netflix's fault. Again, nobody disagrees that there's too much secrecy in these agreements. But to hear Cogent tell Ars Technica their side of the story, the QOS (quality of service) implemented was only necessary because of intentional ISP failures to upgrade network hardware on their end of the equation:

"The system was put in place only because Internet service providers refused to upgrade connections to Cogent to meet new capacity needs, Cogent Chief Legal Officer Bob Beury told Ars. "The problem was, after years of upgrading connections as necessary to accommodate the flow of traffic as necessary at these peering points, Comcast, Time Warner Cable, Verizon, and AT&T refused to do so and let their customers be hurt,” he said.

It's worth noting that Cogent has been in more than a few major disputes over the years when it comes to settlement-free peering, and hasn't made a whole lot of friends in the process. It's also worth noting Cogent does appear to have been caught in a lie, its website proudly proclaiming it doesn't engage in prioritization of any kind. Since then, both Rayburn and a number of Comcast employees have been making the rounds, proclaiming this is, again, proof positive that Cogent is the bad guy in this entire affair. Except it's not.

The problem is that the idea that ISPs are neglecting basic upgrades to create a problem isn't just the opinion of Cogent, it's also the suspicion of nearly every consumer group in existence, smaller independent ISPs (like Sonic.net) as well as Cogent competitors like Level3. Level3 VP Mark Taylor made the compelling case back in July that Verizon was to blame for the Netflix streaming slowdown on Verizon's network:

"We could fix this congestion in about five minutes simply by connecting up more 10Gbps ports on (Verizon's) routers. Simple. Something we’ve been asking Verizon to do for many, many months, and something other providers regularly do in similar circumstances. But Verizon has refused. So Verizon, not Level 3 or Netflix, causes the congestion. Why is that? Maybe they can’t afford a new port card because they've run out – even though these cards are very cheap, just a few thousand dollars for each 10 Gbps card which could support 5,000 streams or more. If that’s the case, we’ll buy one for them. Maybe they can’t afford the small piece of cable between our two ports. If that’s the case, we’ll provide it. Heck, we’ll even install it."

Another, earlier Level3 blog post claimed that all of the biggest ISPs were engaged in this behavior. Netflix too has numerous times now claimed that while the big ISPs insist these are all just run of the mill transit debates, what's happening with this latest debate is very, very different. Netflix has repeatedly insisted that ISPs are using their mono/duopoly control over the last mile to effectively charge for functional access to their customers, imposing arbitrary tolls to offload network operation costs to other companies.

Now it's possible that Netflix, Cogent and Level3 are all lying (something Comcast engineers have been telling telecom reporters), but you can't ignore historical context. If you've been paying attention, using monopoly power to impose arbitrary and unnecessary new tolls on content companies is precisely what the biggest ISPs have spent the last decade promising they would do. AT&T, Verizon, Comcast and Time Warner Cable have spent the last decade engaging in some of the most anti-competitive behavior in the history of any technology market, whether that's writing laws prohibiting cities and towns from deploying their own broadband, to using astroturf and propaganda to push anti-consumer policies. The discussion of their current behavior can't simply exist in a context-free vacuum.

This week, Rayburn's contrarian, vacuum-sealed analysis of the issue resulted in an interesting Twitter exchange between himself and Netflix's David Tempkin. While Rayburn is busy claiming he has confidential but unpublishable knowledge from the industry that somehow proves Netflix is the villain in this equation, Temkin, in turn, accuses Rayburn of engaging in "biased conjecture," arguing that only the FCC currently has all of the data they need to validate Netflix's claims that ISPs are up to no good:

And there's the rub. As somebody who has written about and studied Verizon, AT&T, Comcast and Time Warner Cable behavior for fifteen years, my gut tells me that yes -- they probably are intentionally degrading transit connection points, knowing that the resulting cacophony of often dull engineering analysis would act as cover for their bad behavior. Hiding anti-competitive shenanigans under intentionally over-complicated technical analysis is status quo for incumbent ISPs (again, see the usage cap debate or the wireless carrier blocking of disruptive tech for just two examples). But even if Level3 and and M-Lab's data confirms part of my suspicions, I can't claim to fully prove it without direct access to transit connection points and a close look at ISP hardware and traffic data -- no matter how many compelling charts I post. Neither can Dan Rayburn, Susan Crawford, or anybody else. Not quite yet.

That means there are two things we need here if we care at all about network neutrality and the future of Internet video. One is significantly more transparency across the board, from Netflix, transit partners, and last-mile ISPs, something Rayburn is spot on about. To that end, Jon Brodkin at Ars Technica filed a FOIA request with the FCC earlier this year in an attempt to to get a better look at the data behind this debate, and ran into a brick wall thanks to all of the companies involved in the fight:

"Verizon, Netflix, and Comcast filed requests for confidential treatment of the agreements in their entirety," the FCC's response to Ars said. "In support of its request for confidential treatment, Comcast asserts that, if its agreement were disclosed, competitors would gain valuable insight into the parties' business practices, internal business operations, technical processes and procedures, and information regarding highly confidential pricing and sensitive internal business matters to which competitors otherwise would not have access."

In other words, Comcast wants to quietly accuse everybody of lying, but neither they (or Netflix) want to release the data that proves it. Hiding ISP data from the public has also long been a favorite FCC hobby. Take for example the FCC's $300 million broadband coverage map, which took ISP claims of speeds and locations as gospel, but also completely omits any data on pricing at ISP behest. That lack of data pricing has allowed ISPs for years to claim that the broadband market is more competitive than it actually is. Without transparent access to hard data, most telecom policy discussions wind up being little more than verbose games of patty cake.

The other thing desperately needed is for the FCC to wake up and acknowledge that the debate over net neutrality has very intentionally been immeasurably more complicated, shifting from concerns about throttling or simple service blocking, to issues like interconnection and usage caps (and all the dangerous business models under consideration therein). If the FCC's new rules (hybrid or otherwise) don't incorporate both of these concerns (and so far Wheeler has stated they won't), they're going to fall well short of protecting consumers from the mega-ISPs with a noncompetitive stranglehold over the last mile.

Thank you for reading this Techdirt post. With so many things competing for everyone’s attention these days, we really appreciate you giving us your time. We work hard every day to put quality content out there for our community.

Techdirt is one of the few remaining truly independent media outlets. We do not have a giant corporation behind us, and we rely heavily on our community to support us, in an age when advertisers are increasingly uninterested in sponsoring small, independent sites — especially a site like ours that is unwilling to pull punches in its reporting and analysis.

While other websites have resorted to paywalls, registration requirements, and increasingly annoying/intrusive advertising, we have always kept Techdirt open and available to anyone. But in order to continue doing so, we need your support. We offer a variety of ways for our readers to support us, from direct donations to special subscriptions and cool merchandise — and every little bit helps. Thank you.

–The Techdirt Team

Filed Under: business models, congestion, dan rayburn, fcc, interconnection, transit
Companies: at&t, cablevision, cogent, comcast, level 3, m-lab, netflix, time warner cable, verizon

26 Comments

If you liked this post, you may also be interested in...

Reader Comments

Subscribe: RSS

View by: Time | Thread

That One Guy (profile), 7 Nov 2014 @ 11:00am

Where have I heard that before...
"This morning, Cogent admitted that in February and March of this year the company put in place a procedure that favored traffic on their network, putting a QoS structure in place, based on the type of content being delivered. Without telling anyone, Cogent created at least two priority levels (a ‘fast lane’ and ‘slow lane’), and possibly more, and implemented them at scale in February of this year. What Cogent did is considered a form of network management and was done without them disclosing it, even though it was the direct cause of many of the earlier published congestion charts and all the current debates.

Why does that sound familiar...

Oh, right...

"This morning, [AT&T/Verizon/Comcast/Time Warner Cable] admitted that in February and March of this year the company put in place a procedure that favored traffic on their network, putting a QoS structure in place, based on the type of content being delivered. Without telling anyone, [AT&T/Verizon/Comcast/Time Warner Cable] created at least two priority levels (a ‘fast lane’ and ‘slow lane’), and possibly more, and implemented them at scale in February of this year. What [AT&T/Verizon/Comcast/Time Warner Cable] did is considered a form of network management and was done without them disclosing it, even though it was the direct cause of many of the earlier published congestion charts and all the current debates.

For someone attempting to defend the major ISP's, talking about fast-lane/slow-lanes, and 'prioritized traffic' probably isn't the best tactic to use.
[ link to this | view in thread ]
Anonymous Coward, 7 Nov 2014 @ 11:38am

I thought Comcast was still under a net neutrality agreement after their last merger.
[ link to this | view in thread ]
Designerfx (profile), 7 Nov 2014 @ 11:39am

Re: Where have I heard that before...
Cogent is not letting BGP connections saturate. That's not QOS.

QOS is crappy in general and poorly understood by many, but not the same thing.
[ link to this | view in thread ]
Karl Bode (profile), 7 Nov 2014 @ 11:43am

Re:
They only agreed to adhere to the old FCC neutrality rules that didn't do much of anything in the first place. All they really did is prevent outright ham-fisted blocking or heavy throttling of websites and services. Everything else can be wiggled around if you make some half-assed claims it's for the security and integrity of the network.
[ link to this | view in thread ]
Anonymous Coward, 7 Nov 2014 @ 11:43am

So when is the FCC going to start issuing million dollar fines for customers not getting what they paid for? For that matter, why hasn't the FCC already done this for the advertised speeds to potential customers compared to what they actually get?

Couldn't possibly be regulatory capture could it? /S
[ link to this | view in thread ]
Karl Bode (profile), 7 Nov 2014 @ 11:45am

Re:
So when is the FCC going to start issuing million dollar fines for customers not getting what they paid for?
Right around the same time they start fining ISPs for putting entirely nonsensical fees below the line to misleadingly jack up the advertised price.
[ link to this | view in thread ]
John Fenderson (profile), 7 Nov 2014 @ 11:54am

Re: Where have I heard that before...
This is also deliberately misleading in another way.

In the net neutrality context, nobody objects to QoS differentiation as long as it's done in a neutral way. This is a subtle point that the anti-neutrality people intentionally confuse.

Prioritizing certain types of traffic is a good and acceptable thing. Not all traffic is equal in terms of delivery requirements: video, online gaming, etc., is much more sensitive to lag and jitter than email, for example, and prioritizing video over email delivery is something that is clearly beneficial for everyone. That's not what net neutrality is about, and is not what is meant by "fast lanes" and "slow lanes".

What would not be OK is to prioritize traffic based on who sending or receiving it. That's what net neutrality is about.

In other words, it's fine to to all the QoS prioritization you want, so long as the prioritization applies to everyone in the same way. A video stream I send should be of equal priority as a video stream that Comcast sends, but not necessarily the same priority as an email that I send.
[ link to this | view in thread ]
That One Guy (profile), 7 Nov 2014 @ 12:00pm

Re: Re: Where have I heard that before...
True, treating traffic based upon the type of traffic is one thing, doing so based upon the source is quite another.

Still, I found it rather funny that someone defending the big ISP's would use an argument that with only a little tweaking could be turned against them.
[ link to this | view in thread ]
PRMan, 7 Nov 2014 @ 12:20pm

Sorry, Karl
But Netflix is the innocent party here.

They offered to provide, for free, servers housing all of their most popular content that would reduce Netflix downloads by a significant margin. And most of the ISPs refused.

That lets Netflix completely off the hook in my book.
[ link to this | view in thread ]
Karl Bode (profile), 7 Nov 2014 @ 12:28pm

Re: Sorry, Karl
As the piece notes I THINK Netflix is the "innocent" party here as well, but the fact ISPs didn't sign up for Netflix's free Open Connect CDN doesn't somehow magically settle the issue of whether or not ISPs intentionally caused a logjam. Only seeing the raw data will finally settle the issue.
[ link to this | view in thread ]
Anonymous Coward, 7 Nov 2014 @ 12:35pm

Re: Re: Sorry, Karl
"Now it's possible that Netflix, Cogent and Level3 are all lying..."

Whats in it for netflix? What do they get by lying, after paying for better service?
[ link to this | view in thread ]
Karl Bode (profile), 7 Nov 2014 @ 12:45pm

Re: Re: Re: Sorry, Karl
My point is made repeatedly that it's unlikely Netflix is lying, but nobody can that that definitively yet.
[ link to this | view in thread ]
John Fenderson (profile), 7 Nov 2014 @ 12:48pm

Re: Re: Re: Sorry, Karl
I think that what Karl is doing here is good journalism: sticking to the available facts. He's not saying that Netflix is doing right or wrong, he's saying that the available evidence cannot support either assertion. We just don't have enough hard data.

Kudos to him for also mentioning that, if we go be the past behavior of everyone involved, it's not unreasonable to assume that the major ISPs are engaging in bad behavior. But, as likely as that is to be true (and I think it's very likely), it's still an assumption until we have actual facts.
[ link to this | view in thread ]
Karl Bode (profile), 7 Nov 2014 @ 1:11pm

Re: Re: Re: Re: Sorry, Karl
err, say that...
[ link to this | view in thread ]
Karl Bode (profile), 7 Nov 2014 @ 1:12pm

Re: Re: Re: Re: Sorry, Karl
Thanks.

Unless you're somebody like Brett Swanson, who is known to take money from the telecom industry to push inaccurate facts and anti-consumer policy. (he coined the "Exaflood" term and also helped push "intelligent design"): http://www.techpolicydaily.com/internet/caused-web-slow-down-comcast-twc-verizon/
[ link to this | view in thread ]
Mason Wheeler (profile), 7 Nov 2014 @ 1:19pm

But even if Level3 and and M-Lab's data confirms part of my suspicions, I can't claim to fully prove it without direct access to transit connection points and a close look at ISP hardware and traffic data -- no matter how many compelling charts I post. Neither can Dan Rayburn, Susan Crawford, or anybody else. Not quite yet.

But you don't need to. We're talking about streaming video over the Internet, right?

Well, the Supreme Court just recently established a new standard of evidence for video on the internet, and if this doesn't "look like a duck" I don't know what does! So the ISPs are guilty as charged. ;)
[ link to this | view in thread ]
Aerilus, 7 Nov 2014 @ 1:37pm

Just some numbers for core networking equipment

I just purchased some layer three fiber switches made by smc who supplies a lot of the isps with their equipment. its about 8k$ for a 48port sfp+(10gbps) with 4x40gbps uplinks. 40km 10gbps sfp adapters/cards are about 1000$ each you would need one for every port you wanted to activate. this isn't rocket science. I know for a fact in NC there are known congestion points that drop packets and we have some serious internet infrastructure.
[ link to this | view in thread ]
Anonymous Coward, 7 Nov 2014 @ 2:20pm

best line:
"most telecom policy discussions wind up being little more than verbose games of patty cake."
[ link to this | view in thread ]
Anonymous Coward, 7 Nov 2014 @ 2:46pm

Re: best line:
I'll just leave this here. It seems so...relevant.

https://www.youtube.com/watch?v=fdsZCxXV8TE#t=30
[ link to this | view in thread ]
Andrew D. Todd, 7 Nov 2014 @ 3:06pm

That Works Out to Eleven Cents per Month per Broadband Customer. (to Aerilus, #17)
Let's do the numbers. So, the main machine costs $8000 for 160 Gibits uplink. Assume that each broadband customer gets 10 Mbits, which is likely to be the ultimate capacity of their copper wireline subscriber loops. The switch is capable of supporting 16,000 customers, assuming that they are all talking at once. The 10 Gbit downstream cards at $1000 are capable of supporting 1000 such customers each. In the worst case, geographical constraints might mean that you have to use all 48 downstream ports, instead of just 16, so that is 333 customers per downlink card. That works out to fifty cents per customer for the uplink, and $3.33 for the downlink, or $3.83. Assume three years amortization, and that is about eleven cents a month. And, I repeat, that is for a dedicated ten megabits.
[ link to this | view in thread ]
aerilus, 7 Nov 2014 @ 3:44pm

Re: That Works Out to Eleven Cents per Month per Broadband Customer. (to Aerilus, #17)
yes when looked at like that its a very negligible and reasonable cost to keep your network upgraded. instead of giving money to every election campaign and spending hundreds of million in marketing and lobbying.
[ link to this | view in thread ]
Anonymous Coward, 7 Nov 2014 @ 5:41pm

Re: Re: Re: Re: Sorry, Karl
I understand that. I was asking a legitimate question. If they are lying, and are believed, what is to be gained? If nothing, then it just means that they likely aren't lying.
[ link to this | view in thread ]
Anonymous Coward, 7 Nov 2014 @ 6:00pm

Re: Re: Re: Re: Re: Sorry, Karl
I see the confusion comes from replying in the thread. I should have just posted my question. I am not arguing anything.

I can see why the ISP's may be lying, I am just wondering what is in it for netflix to lie(conspiracy theories welcome too)? Nothing isn't proof that they aren't lying, just less likely that they are lying. If the article says what they have to gain by lying or that they have nothing to gain by lying, I missed it, sorry.
[ link to this | view in thread ]
Andrew D. Todd, 7 Nov 2014 @ 9:05pm

Re: Re: Re: Re: Re: Re: Sorry, Karl
Those of us who are engineers have access to a source of facts without reference to ComCast. We can design systems similar to those ComCast builds, work out what components and labor are required to build them (otherwise known as a "bill of materials"), price everything, and work out a total. This is called a "costing-out" exercise. We can make the simplifying assumption that all the users talk at once. We don't know how good a job ComCast is able to do in negotiating volume discounts, but we do know that our price estimates form an upper bound. We may have difficulty figuring out how often people talk at the same time, but we do know that it is not greater than all the time. If we make those two simplifying assumptions, of publicly available component prices, and everyone talking at once, all the time, and still get a given price, we know that ComCast's costs must be lower. What we in fact find is that most of the costs of a communications network are in the subscriber loops, not in the switches and trunk lines. Communication networks are essentially fractal-shaped. It is not very expensive to overbuild the network center around the worst-case assumption. The foregoing is true, whether ComCast CEO Brian Roberts says it or not. If Brian Roberts insists that two equals three, that does not make two equal to three-- it merely makes Brian Roberts a self-confessed liar.

We don't ask whether there might be some motive for ComCast to lie about technical matters, and attempt to assess its veracity. We start from the fact of the lie, and ask which evil purpose led ComCast to tell lies which are so easily found out. If the kinds of answers we get sound like a conspiracy theory, then so be it. The obvious point is that Brian Roberts wants to preserve a top-down telecommunications system, such as cable television, and finds the internet threatening. He wants a system in which people consume what content their lords and masters tell them to consume, and do not choose what to consume, and most certainly do not produce their own content and redistribute it. However, this last is what the internet is all about.

The man who runs the city waterworks does not get to say whether people drink coffee or tea. For that matter, he has essentially no power to prevent the distribution of imported bottled water. His truly essential function has to do with flush toilets. In the recent events in Charleston, WV, when poisonous chemicals were fed into the water system, the system could not supply water which was safe to drink, or to use to prepare food, or even to bathe in, but it at least continued flushing the toilets.
[ link to this | view in thread ]
Gerald Robinson (profile), 8 Nov 2014 @ 9:28am

Net neutrality and Netflix
It's past the time that the FTC did it's job and broke up the nasty anti-competative oligopolies of: TW, COMCAST, AT&T, Verizon and Disney. They have an interlocking set on non-compete agreements which lock out competition in most of the US.
[ link to this | view in thread ]
Spike (profile), 8 Nov 2014 @ 6:12pm

These same ISP's demanding payment are the same ones that have refused to install Netflix caching servers into their network. Gee I wonder.... not.
[ link to this | view in thread ]