Interconnection: Or How Big Broadband Kills Net Neutrality Without Violating 'Net Neutrality'

from the routing-around-the-problem dept

Mon, May 5th 2014 12:07pm — Mike Masnick

For years now, every time the net neutrality debate starts getting really confusing, Tim Lee comes along and puts it all into useful perspective. Six years ago, there was his exceptionally useful position paper on net neutrality for the Cato Institute. A couple years ago, he wrote another great piece for National Affairs magazine that deftly explained why the internet wasn't competitive and why that's a problem. Now working for Vox, he's put together a great piece that explains the technical difference between the interconnection fights and the net neutrality battle -- but also explains how the end result is basically the same.

There has certainly been plenty of discussion about Netflix's recent interconnection deals -- starting with Comcast back in February, and Verizon just recently. As we noted back in February about the Comcast/Netflix deal, it wasn't really a net neutrality issue, because it was about interconnection, but we were equally worried about how this really demonstrated the monopsony power of major broadband providers. Lee's latest Vox piece dives much deeper into that distinction, and highlights some key insights:

These are not technically "net neutrality" violations. The animated graphics in the Vox piece do a great job explaining why. Just go look.
But that might not matter, because the end result is basically the same.
Because of that, this is a really effective way for the big broadband providers to pay lip service to net neutrality, while getting exactly what they want anyway.
The real problem, as always, is too much market power by the big broadband players, allowing them to do an "evil" thing almost no one thought they would do.

To explain the first point, you basically have to recognize that with the interconnection issue, Comcast and others aren't technically "favoring" Netflix (or anyone) on their own network, but are rather setting up deals that favor that content off their network, but right up until it hits the last mile. And they're doing this by purposely letting their connection to other transit providers pile up. And that's why the end result is the same, but allows Comcast to technically be correct that they're not violating net neutrality, while creating the identical end result for most of the major players:

Comcast says it wasn't deliberately degrading the performance of Netflix traffic, which would have been a violation of network neutrality. But that implies that every website that used the same transit provider as Netflix was experiencing similar performance problems. That's obviously bad for Comcast customers, and it's also bad for startups that are trying to become the next YouTube or Netflix. If the only way to get excellent service on America's largest broadband networks is to negotiate a private connection directly to those networks, smaller companies with less cash and fewer lawyers are going to be at a competitive disadvantage.

In other words, those public transit links that were providing Netflix with subpar service could become the de facto slow lanes on Comcast's network, while private, direct connections could become the fast lane.

It's a neat trick. Comcast doesn't have to favor or disfavor any particular traffic. It just conveniently neglects to do the most basic things to make sure traffic flows smoothly from the various transit players -- and then offers to negotiate direct deals with big players to bypass third-party transit providers. Thus, it's not "favoring" anyone on its network. It's just creating a situation that favors those who pay up to access its network.

And, again, the FCC doesn't seem to recognize this at all. Tom Wheeler has been fairly explicit that he doesn't view the interconnection issue as relevant to this discussion yet. And the various rules that are being fought over won't have any impact one way or the other on interconnection. And, as Lee's article describes, part of the reason all the commotion has been focused "over there" on "net neutrality" rather than "over here" on "interconnection" is because the very idea of Comcast doing what Comcast has been doing these past few years just seemed impossible to consider not too long ago:

One reason for this is that Comcast's hardball tactics are relatively new. According to Wu, "we never thought much about" interconnection issues when the concept of network neutrality was developed in 2002. That was "mainly because it seemed like, 'Why would anyone do that?'" It was generally assumed that consumer ISPs would always buy enough transit to serve their customers' needs.

Wu worked in Silicon Valley prior to 2002. And he says he remembers that "solving a joint technical challenge" — that is, making the internet work as well as possible — "was the predominant motivation of the engineers." That's still true in most parts of the internet. But some of the largest ISPs now seem to view declining network performance not as a technical problem to be solved so much as a source of leverage in business negotiations.

That last line is worth repeating: Big ISPs view declining network performance as a source of leverage, rather than a technical problem.

And the only reason they can do this is because of the tremendous market power they have. And, once again, as we've argued for over a decade, that demonstrates the real heart of the problem: the lack of real competition in the broadband space. Either way, it's yet another must read if you're trying to understand the state of play today.

Thank you for reading this Techdirt post. With so many things competing for everyone’s attention these days, we really appreciate you giving us your time. We work hard every day to put quality content out there for our community.

Techdirt is one of the few remaining truly independent media outlets. We do not have a giant corporation behind us, and we rely heavily on our community to support us, in an age when advertisers are increasingly uninterested in sponsoring small, independent sites — especially a site like ours that is unwilling to pull punches in its reporting and analysis.

While other websites have resorted to paywalls, registration requirements, and increasingly annoying/intrusive advertising, we have always kept Techdirt open and available to anyone. But in order to continue doing so, we need your support. We offer a variety of ways for our readers to support us, from direct donations to special subscriptions and cool merchandise — and every little bit helps. Thank you.

–The Techdirt Team

Filed Under: broadband, fast lanes, interconnection, monopoly power, net neutrality, power
Companies: cogent, comcast, level 3, netflix

23 Comments

If you liked this post, you may also be interested in...

Reader Comments

Subscribe: RSS

View by: Time | Thread

Ninja (profile), 5 May 2014 @ 12:00pm

Infra-structure is a tricky area. It doesn't make much sense to build several parallel networks for competing ISPs. Brazil is trying to deal with it by throwing in the giant network Telebras (Govt company) has all over the country specially where the regular isps simply don't want to go (costs, smaller user base etc) letting them provide services in those places using the state telecom links. Truth be said it's still crawling in its early states but it may be an idea. There's the issue of having the Govt handling the infra-structure which might not be so wise in this specific case but it is an idea. Or you could split the ISP from the infra-structure provider while restricting regions where a determined company can build their network and making them practice the same pricing for all ISPs?

I wonder where the balance is...
[ link to this | view in thread ]
limbodog (profile), 5 May 2014 @ 12:53pm

There's plenty of competition! All you have to do to get it is move to South Korea.
[ link to this | view in thread ]
Andypandy, 5 May 2014 @ 1:01pm

ISP'S
They were given billions in support of them building stronger networks that could handle the increase in use of the internet and they abused the option to do so by using the money to buy competition and politicians etc etc.
Once the internet is classified as an essential service and the ISP's are demoted to salesmen only then will things start to improve for consumers, but the lobbyists are using a lot of money to try to ensure it does not happen , happily the consumer had woken up or partly woken up and maybe just maybe things will get better.
[ link to this | view in thread ]
jupiterkansas (profile), 5 May 2014 @ 1:01pm

Re:
or Kansas City.
[ link to this | view in thread ]
My Name Here, 5 May 2014 @ 1:04pm

Not So Simple
It's a great story, but it really isn't as simple an issue as it is painted.

One of the things you have to deal with here is that net neutrality doesn't really exist to start with. The basis of the internet is peering and exchange, and not all ISPs exchange directly with everyone else on the net at the exactly same level. Some companies use one provider more than the other, and larger companies even run their own networks and transit in some cases. Some companies peer with every provider available, others rely on a more limited number of peers.

Since not all content has exactly the same connectivity to your ISP, everyone has a different experience online. My ISP might support more of HE.NET, yours might use above.net, and a third might get most of it's connectivity from Level3. If the server(s) you are trying to reach peer more with Level3 and your ISP more with HE.NET, then you may find yourself getting slower response than someone on an ISP that peers directly onto level3.

What Netflix is doing, albeit grudgingly, is paying for the transit and paying the ISP to accept to peer with them. They have reached a point where the amount of traffic they are trying to push through the regular ISP peering is more than the ISPs are willing to pay for or are able to obtain, and in fact are degrading the performance of all sites using the same peering. Yes, Netflix is buying a faster connection, but it is the same result as the ISP buying more connectivity to support them. The only question is who is paying - either you pay with higher ISP bills, or Netflix pays (and charges you at some point) to get better connectivity.

The plus of all of this is that the ISP's existing peering arrangements end up working better, sites that were slower because Netflix was overloading a peer point get better response, and everything gets better. There is no loss of service, just an improvement.

No traffic is blocked. No services are lost. A non-Netflix subscriber actually benefits. A Netflix subscriber benefits. It's pretty much all pluses, because it's the original service plus more, not less. You don't lose anything, as a consumer you only gain.

The biggest long term issue faced by ISPs is the overload of their internal network. That is not such a big issue for the moment, but if more streaming services or IPtv type services come online, they may find themselves unable to accept any more peering because they cannot handle the traffic internally. Perhaps this will be a driver for higher speed networking at the ISP level.
[ link to this | view in thread ]
John Fenderson (profile), 5 May 2014 @ 1:22pm

Re: Not So Simple
"The plus of all of this is that the ISP's existing peering arrangements end up working better"

I have two main reasons I doubt this: 1) We're talking about Comcast, and 2) Netflix already had a solution in hand, available for free to Comcast (and any ISP), to correct this by using their own network. Concast said "no". Comcast wasn't interested in solving any bandwidth problems. They were only interested in extracting unearned income from Netflix.

That Netflix caved on this, and that they're doing similar deals with other major ISPs, is disastrous and caused me to cancel my Netflix subscription.

"Perhaps this will be a driver for higher speed networking at the ISP level."

Only for connections with major players who can afford to pay an extra tribute to the ISP. For everyone else, the result that seems more likely is worse performance as the ISPs allow their non-premium peering points to continue to degrade over time.
[ link to this | view in thread ]
Anonymous Coward, 5 May 2014 @ 1:52pm

Re:
Hell, at this point, Bin Laden's bunker is more neutral than Comcast et al.
[ link to this | view in thread ]
That One Guy (profile), 5 May 2014 @ 2:30pm

Re:
Just tossing the idea out, feel free to pick it apart and re-work any parts that don't make sense, but how about the government stops giving the ISP's billion dollar handouts/tax breaks, which are supposed to be used to 'upgrade' their networks(and which instead just go to pad out their profit reports), and instead puts it towards building public networks of their own, networks which private companies may use, but would not own.
[ link to this | view in thread ]
John Thacker, 5 May 2014 @ 2:52pm

Re: Re: Not So Simple
Netflix already had a solution in hand, available for free to Comcast (and any ISP), to correct this by using their own network.

True, they offered to operate their own CDN for free inside Comcast's network. The ISPs that regularly charge CDNs for operating inside turned Netflix down rather than accept it for free. (Smaller ISPs, that have to pay more for Internet transit, eagerly accepted a price of zero.)

But I presume that Netflix wasn't offering to freely carry the traffic of any other video content provider, including startups, were they? Just their own? Competitors would still have to pay a CDN to carry their traffic, so Netflix was angling for an advantage over all the other competitors out there? I mean, sure, they could say that they'd fully support anybody else's effort to put together their own CDN, but a small startup company wouldn't have the ability to do that like Netflix, they'd have to go with one of the competing ones.

Kind of seems like Netflix was searching for a cheaper fast lane for themselves than all the other CDNs out there. Not that I entirely blame them for trying to throw their weight around to get a better deal, but not exactly "net neutrality" on their part either.
[ link to this | view in thread ]
John Thacker, 5 May 2014 @ 2:59pm

But that implies that every website that used the same transit provider as Netflix was experiencing similar performance problems. That's obviously bad for Comcast customers, and it's also bad for startups that are trying to become the next YouTube or Netflix.

Actually, sort of not. The only reason this works against Netflix is because Netflix is singlehandedly responsible for such an enormous fraction of all Internet traffic at once. Any startup would be invisible in the midst of all that network traffic-- all they'd have to do is avoid picking the transit provider (Level 3) that serves the behemoth Netflix and as a result constantly promises bandwidth it can't deliver.

What actually hurts startups is that Internet transit tends to be priced metered / by actual sustained throughput (95th percentile), but in a way that the unit price decreases the more you guarantee that you'll use. So the big guys pay a smaller rate as their volume increases.

Now, consumer ISP access is priced very differently, which is another part of the problem. Unlike Internet transit, it's not priced at actual sustained throughput, but on the basis of a theoretical maximum throughput, which might be achieved to some sites, but many sites it can't be achieved-- and it couldn't be achieved regardless of what your ISP does, and you wouldn't even necessarily want to achieve your max throughput anyway. Unfortunately, though, that also means that if your service degrades to certain sites, there's no way to tell nor does it immediately cost your ISP any more. But any way of measuring actual throughput or sampling it (which is what happens in Internet transit) is functionally equivalent to metering, which everyone, including me, hates.
[ link to this | view in thread ]
Andrew D. Todd, 5 May 2014 @ 5:05pm

Food Is The Endgame
The posts of "My Name Here" and "John Thacker" are collections of Comcast talking points, which are without technical merit. At this point, I don't think they need further discussion. Here is a more interesting issue, raised in a recent Business Week article:

Venessa Wong, "Soon Hulu Will Take Your Pizza Hut Order, Too," Business Week, May 05, 2014

http://www.businessweek.com/articles/2014-05-05/soon-hulu-will-take-your-pizza-hut-order-too#r=h pt-fs

Hulu, the internet television provider, is launching a service to take orders for delivered pizza. I do not know whether this will be successful, but they have convinced themselves that it will, and the conviction will spread rapidly through the television, music, and ISP businesses. In practice, Hulu seem to be making deals with the kind of national pizza chains which specialize in soggy pizza, There's always someone local, as within a mile or less, who is better. But, as Ms. Wong hints, the most desirable customer, from Hulu's point of view, is someone who will happily eat dogfood from a bowl on the floor...

People spend much more money on food than they do on video. United States Box Office expenditure is on the order of ten billion dollars. I don't know just what video for home consumption is, but it will be on the same order of magnitude. Food is on the order of a trillion dollars, a hundred times greater, more or less. Sooner or later, the greater movie-ISP industry will want to get its claws in that sum. Bandwidth issues may become irrelevant, because the messages necessary to communicate food orders are so short. When I made myself a checklist for going grocery shopping, about ten years ago, it came to 1920 bytes. The ISP's will be committing overt acts of sabotage, trying to block particular telephone numbers.

Perhaps Mr. Thacker should explain to us why Comcast pizza is better than any other kind of pizza?
[ link to this | view in thread ]
lucidrenegade (profile), 5 May 2014 @ 6:47pm

What is "monopsony power"? :0)
[ link to this | view in thread ]
My Name here, 5 May 2014 @ 8:09pm

Re: Food Is The Endgame
You scored well here. You not only dismissed a long post as "monopolistic bastard's talking points", you also managed to insult the average consumer at the same time.

Perhaps you can explain how pizza would harm net neutrality, and how applying the old Techdirt saw of using free content to sell other stuff is somehow bad if it's used by a bigger company.

Mike will spank you and put you in the corner with the dunce hat on for such nonsense!
[ link to this | view in thread ]
Andrew D. Todd, 5 May 2014 @ 10:04pm

Re: Re: Food Is The Endgame
Well, not to repeat myself needlessly:

http://www.techdirt.com/articles/20140206/06594726111/comcast-backed-lobbyist-insists-sea ttle-doesnt-want-faster-cheaper-broadband.shtml#c481

http://www.techdirt.com/articles/20140218/080305 26262/apparently-we-are-all-confused-killing-net-neutrality-will-be-just-great-startups.shtml#c208
ht tp://www.techdirt.com/articles/20140218/08030526262/apparently-we-are-all-confused-killing-net-neutr ality-will-be-just-great-startups.shtml#c373
http://www.techdirt.com/articles/20140218/08030526262/ap parently-we-are-all-confused-killing-net-neutrality-will-be-just-great-startups.shtml#c406
http://www .techdirt.com/articles/20140218/08030526262/apparently-we-are-all-confused-killing-net-neutrality-wi ll-be-just-great-startups.shtml#c461
http://www.techdirt.com/articles/20140218/08030526262/apparently -we-are-all-confused-killing-net-neutrality-will-be-just-great-startups.shtml#c526
http://www.techdir t.com/articles/20140218/08030526262/apparently-we-are-all-confused-killing-net-neutrality-will-be-ju st-great-startups.shtml#c612
http://www.techdirt.com/articles/20140218/08030526262/apparently-we-are- all-confused-killing-net-neutrality-will-be-just-great-startups.shtml#c653

http://www.techdirt.com/ar ticles/20140430/00090727072/what-inefficient-airline-boarding-procedures-have-to-do-with-net-neutral ity.shtml#c3767
http://www.techdirt.com/articles/20140430/00090727072/what-inefficient-airline-boardi ng-procedures-have-to-do-with-net-neutrality.shtml#c4352

Of course, you weren't here when we talked about all this previously, were you?
[ link to this | view in thread ]
Anonymous Coward, 6 May 2014 @ 2:40am

interconnection business isn't anything new
Some ISPs/carriers are running an interconnection business for over 15 years now. It very simple. Since you're a big player you tell others that you'll peer just with tier 1 providers. Also you stay away from local public IXs. That way you force any smaller ISPs to buy transit from you. Take AS3320 for example. They got a huge subscriber network and should be concerned about optimal routing to make their customers happy. Instead, they're staying away from decix (very large publix IX), do private peerings with tier 1 carriers and try to force smaller providers to buy transit for horrendous rates. AS3320 is member of some IXs, but only outside it's own main market. And guess what, the lines to those IXs are congested from time to time. The smaller ISPs would love to get their traffic to AS3320's customers as directly as possible, but they buy transit from tier 1 carriers with reasonable rates and accept that the AS path for AS3320 becomes a little bit longer.

I know that DSL business doesn't create much profit and providers would be more than happy if they're paid twice for the same traffic (content provider in and customer out), but it's the wrong way to force smaller providers into being transit customers. From a business point of view it's good to do that to make more money and to keep the competitors small. We all know how that will develop into a market with a few giants and horrendous internet rates for customers. Unfortunately the regulators like FCC don't know how the internet works. And net neutrality isn't just something for customers and content providers, it's should be standard for the whole internet.
[ link to this | view in thread ]
Allen (profile), 6 May 2014 @ 5:11am

Network Neutrality is a distraction
I won't say what I think of Network Neutrality because frankly it's a distraction. Regulations to overcome the lack of competition can never be as effective as competition itself. An isp that is forced to be concerned with customer retention is not going to play power games with content providers (or at least not to the point where customers might notice).
[ link to this | view in thread ]
Skippy, 6 May 2014 @ 7:00am

Re: Re: Re: Not So Simple
You are only partially right in this.

First off lets get something straight as so many people imply this with their post. Netflix does NOT just randomly place packets on any ISP's network. It is 100% the traffic of the ISP's paying consumers.

Secondly, the ISP is responsible for maintaining a suitable network for their users that are paying them to do such. Upgrading links because they have become saturated is one of those responsibilities.

Third, who, what and where those packets causing #2 contain or are from is not at all relevant. The ISP's consumers have requested data on the internet, thus it is the responsibility of the ISP to see that they get that data in the most efficient manner. Purposely allowing a point to saturate to "extort" money out of a company is 100% against the interest of their own paying subscribers and I am sure there is a crafty lawyer out there that can make a class action out of it against them based on their own TOS.

Lastly, regardless of size the ISP benefits just as much (if not more) than Netflix for entering into their OpenConnect program with no money exchanging hands. Being that ISP traffic is very one sided (they request a lot more traffic from the internet then they put on it) they are already behind the 8 ball when it comes to peering agreements and thus pay transit providers to even be a part of the internet. Though some of them have gotten quite large and now also have transit capacity to help offset this, the fact still remains that ISP traffic is very one sided.
[ link to this | view in thread ]
John Fenderson (profile), 6 May 2014 @ 8:52am

Re: Network Neutrality is a distraction
I agree. The trouble is that there appears to be no realistic path to increasing competition. The anticompetitive nature of the market is by design and by law, and the existing giants will fight like crazy to make sure it stays that way. Solving the competition problem requires us to fix a lot of the political corruption problem first. It's a tremendously heavy lift.

Network neutrality laws, on the other hand, are achievable. They're far from ideal (which, ironically, is the very thing that makes them achievable), but better than what we have right now.

Personally, the answer I prefer is reclassify ISPs as common carriers like they should have been from day 1. This would open up the transmission channels and allow real competition.
[ link to this | view in thread ]
lobbyistRcriminals, 6 May 2014 @ 9:10am

Re: ISP'S
Except we have a lobbyist running the FCC...
[ link to this | view in thread ]
NaBUru38 (profile), 6 May 2014 @ 9:37am

Isn't this issue called "backbone fracture"?
[ link to this | view in thread ]
fgoodwin (profile), 6 May 2014 @ 12:25pm

Re: Re:
But the cable companies don't get any gov't handouts.
[ link to this | view in thread ]
Andrew D. Todd, 6 May 2014 @ 12:39pm

Your Definition of an ISP is Mailman-Think (to: Skippy, #17)
Comcast does not know how to be a mailman. A mailman takes letters from the collection box, and delivers them faithfully. From time to time, when circumstances warrant, the Post Office issues the mailman a weapon (classically a long-barreled Colt 45 revolver) so that he can shoot "road agents" who are attempting to rob the United States Mails. A mailman wears a blue uniform, and got his job by virtue of passing a civil service exam, and gets paid much more than a FedEx courier, or a bicycle messenger. And he gets a pension. The Post Office recruits from the same basic labor pool as the Police Department and the Fire Department. This sort of tradition of public service is profoundly alien to Comcast.

Comcast is basically an organization of "Carnies," that is, the people who work in traveling carnivals, and fair midways. Their thinking is "Carnie-Think" If you go to the multiplex theater in the mall, they have perhaps half a dozen movies showing, all more or less geared to adolescents. They don't respond to requests. If you ask to see, say, Alexander Nevsky, they will look baffled, and a more or less illiterate teenage employee will assure you that there isn't any such film. Similarly, suppose you've decided that fried food is hard on your stomach. So if you go to McDonald's, and ask for meat loaf and mashed potatoes instead of burger and fries, they are going to laugh at you. In a company like this, the ordinary employee, the foot soldier or spear-carrier, is never anything to be proud of, he's just minimum-wage, and expendable. Comcast comes out of this tradition. They think they ought to be able to tell you what to eat, and what to drink, and what to watch, and what to read, and what to think, and how to vote, and so on and so forth.
[ link to this | view in thread ]
nasch (profile), 7 May 2014 @ 12:45pm

Re: Network Neutrality is a distraction
Regulations to overcome the lack of competition can never be as effective as competition itself.

Are there any serious broadband problems in the US that wouldn't be solved by healthy competition?
[ link to this | view in thread ]