Level3 Proves That Verizon Is Absolutely To Blame For Netflix Congestion... Using Verizon's Own Blog Post

from the ouch dept

Fri, Jul 18th 2014 11:10am — Mike Masnick

As you know, despite Netflix having already agreed to pay Verizon's shake down fee to avoid interconnection congestion for its streaming, Netflix and Verizon are fighting a very public fight about who's to "blame" for the network congestion. Netflix took a public shot at Verizon by publicly highlighting Verizon's congested network to users, leading Verizon to send a cease-and-desist letter, claiming it was misleading. As we noted, there's no doubt that it's actually Verizon at fault, because it's not delivering exactly what it sold customers (and we wonder why the FTC is still not getting involved yet). If Verizon's network is getting congested, that's on Verizon to fix, since it sold its users a promise that they could reach anywhere on the internet, including Netflix.

Verizon tried to spin the story back in its favor last week, with a blog post about "the congestion myth," in which it claimed that the real problem was how Netflix chose to route its traffic to Verizon. It presented the following nifty chart, claiming that there was no congestion at all on Verizon's network, and saying that it was all about how Netflix was choosing to deliver its traffic to Verizon's network:

As you can see in the "red" arrow, it's showing that there's 100% utilization at the interconnection link. And Verizon claims this is all on Netflix. Here's Verizon's explanation:

One might wonder why Netflix and its transit providers were the only ones that ran into congestion issues. What it boils down to is this: these other transit and content providers took steps to ensure that there was adequate capacity for their traffic to enter our network. In some cases, these are settlement-free peering arrangements, where the relative traffic flows between an IP network provider and us remain roughly equal, and both parties invest in sufficient facilities to match these roughly equal flows. That is the traditional basis for such deals. In other cases there may be traffic imbalances, but the networks or content providers have entered into paid arrangements with us to ensure connections and capacity to meet their needs for their out-of-balance traffic.

Some reporters took this at face value, but it never made any sense at all. The chart above pretty clearly shows that the congestion point is actually Verizon's border router. And if it just made a basic upgrade to accept the traffic that it has promised to consumers, there would be no problem at all.

Level3 has now jumped into this debate as well, with even more data showing that Verizon is the real culprit here. Level3 is carrying a bunch of that Netflix traffic, and notes that it has more than enough bandwidth to carry it. It says the only problem is Verizon refusing to take 5 minutes to upgrade its system:

Verizon has confirmed that everything between that router in their network and their subscribers is uncongested – in fact has plenty of capacity sitting there waiting to be used. Above, I confirmed exactly the same thing for the Level 3 network. So in fact, we could fix this congestion in about five minutes simply by connecting up more 10Gbps ports on those routers. Simple. Something we’ve been asking Verizon to do for many, many months, and something other providers regularly do in similar circumstances. But Verizon has refused. So Verizon, not Level 3 or Netflix, causes the congestion. Why is that? Maybe they can’t afford a new port card because they’ve run out – even though these cards are very cheap, just a few thousand dollars for each 10 Gbps card which could support 5,000 streams or more. If that’s the case, we’ll buy one for them. Maybe they can’t afford the small piece of cable between our two ports. If that’s the case, we’ll provide it. Heck, we’ll even install it.

Level3 has provided a (not quite as nicely designed) image to zoom in on the border router situation, showing that it has plenty of capacity ready -- all it needs is for Verizon to let it connect more ports:

Again, this is what plenty of people have been saying since the beginning of this interconnection fight. Verizon, Comcast and AT&T have deliberately made the decision not to make rather basic and inexpensive upgrades to their interconnection points that would solve the congestion problems with Netflix. In doing so, they are the ones creating the bottleneck and congestion -- and effectively using it to shake down Netflix, getting them to pay extra for the bandwidth that the broadband providers' customers have already paid for.

Looking at this, it once again becomes clear that it's Verizon, AT&T and Comcast that have deliberately caused this congestion, using their positions as dominant players with monopoly control over the last mile to force Netflix to pay them extra. As Level3 notes, it takes two parties to take the "trivial" steps to remove the congestion, and it's Verizon that's the party who isn't cooperating:

All of the networks have ample capacity and congestion only occurs in a small number of locations, locations where networks interconnect with some last mile ISPs like Verizon. The cost of removing that congestion is absolutely trivial. It takes two parties to remove congestion at an interconnect point. I can confirm that Level 3 is not the party refusing to add that capacity. In fact, Level 3 has asked Verizon for a long time to add interconnection capacity and to deliver the traffic its customers are requesting from our customers, but Verizon refuses.

As we've discussed, Verizon, Comcast and AT&T know exactly what they're doing here. People hadn't been so concerned with interconnection disputes in the past, because they didn't think the big broadband players would be so crass and so anti-consumer to purposely let interconnection points clog up. But, those three companies have such control over the market at this point that they are able to do that and can effectively shake down internet companies to get them to double pay for the bandwidth that subscribers are already paying for. Today it's Netflix, but soon it's likely to be lots of other companies as well. That's why, in our own comments to the FCC, we noted that the interconnection fights need to be a part of the open internet discussion.

Thank you for reading this Techdirt post. With so many things competing for everyone’s attention these days, we really appreciate you giving us your time. We work hard every day to put quality content out there for our community.

Techdirt is one of the few remaining truly independent media outlets. We do not have a giant corporation behind us, and we rely heavily on our community to support us, in an age when advertisers are increasingly uninterested in sponsoring small, independent sites — especially a site like ours that is unwilling to pull punches in its reporting and analysis.

While other websites have resorted to paywalls, registration requirements, and increasingly annoying/intrusive advertising, we have always kept Techdirt open and available to anyone. But in order to continue doing so, we need your support. We offer a variety of ways for our readers to support us, from direct donations to special subscriptions and cool merchandise — and every little bit helps. Thank you.

–The Techdirt Team

Filed Under: border routers, congestion, interconnection, streaming
Companies: level3, netflix, verizon

125 Comments

If you liked this post, you may also be interested in...

Reader Comments

The First Word

“

Re: Re: I'd say they are defaming Level-3

In all honesty, directly peering with the content provider is usually preferred no matter who is providing the transit network.

Not necessarily. There are situations - including this one - where it makes a lot of sense to work with a single transit provider (or for redundancy 2 or 3) instead of needing to both pay the transit providers AND directly peer with a dozen or more ISPs.

Netflix works with Level3 and Cogent. Both are Tier 1 networks - which means that's all Netflix needs to work with to ensure that their content is reachable by anyone on the internet. So long as they can provide sufficient speed, Netflix wouldn't need to worry about making deals with Verizon, and AT&T, and Comcast, and Time Warner, and Cablevision, and Google Fiber, and Qwest, and so on.

And Verizon, AT&T and Comcast know this. Verizon is also a Tier 1 transit provider. So is AT&T. Comcast is trying to be. They all compete with Level3 and Cogent in the transit business - which is cutthroat and there's good money in it. Here's the thing. Level 3 and Cogent do not compete with those companies in the residential broadband business.

So it comes down to this: Verizon and Comcast can abuse their monopoly position over their residential business in order to force Netflix to directly peer with them, costing Level3 that transit business with Netflix, while losing nothing (since their residential customers have no choice and can't move to another provider).

—Josh in CharlotteNC

”

Subscribe: RSS

View by: Time | Thread

Josh in CharlotteNC (profile), 18 Jul 2014 @ 11:01am

Read the Level3 blog
If you want even more proof of cable/telco company fuckery, read the whole article on Level3's blog.

One striking example is that Level3's network connects to Verizon's in 10 locations. 7 in the US and 3 in Europe. Only in places where Verizon has a significant share of residential broadband customers are the ports at high/full capacity. In Europe, where Verizon has to compete against other ISPs, the ports are not congested.
[ link to this | view in thread ]
halley (profile), 18 Jul 2014 @ 11:17am

Sounds like Verizon is guilty of exactly the same thing that Chris Christy's staff got caught in. They wanted to put the squeeze financially, so they put the squeeze on the available lanes through the tunnel at rush hour. For the simple price of removing a barrier or adding a bit of CAT5, there would be no problem. No problem means no chance to beg for more money.
[ link to this | view in thread ]
Anonymous Coward, 18 Jul 2014 @ 11:17am

The insidiousness of these companies should be a blatant warning of their intentions for the internet as a whole, proper oversight and regulation appear to be a requirement in the interests of the consumer.
[ link to this | view in thread ]
Ninja (profile), 18 Jul 2014 @ 11:20am

I have a big smile in my face right now. I anxiously await the shills twist to this story.

Awesome job Level 3, we love you. And a score to Net Neutrality. May Title II deliver it's fiery hell upon the ISPs.
[ link to this | view in thread ]
David, 18 Jul 2014 @ 11:25am

I'd say they are defaming Level-3
From Verizon's Blog Post:
"Therefore, we are working aggressively with Netflix to establish new, direct connections from Netflix to Verizon’s network. This doesn’t “prioritize” Netflix traffic in any way, but it ensures that their traffic gets on our network through direct connections—not middleman networks—that are up to the task."

So, they are claiming that Level-3 is a 'middleman network' and are not "up to the task" of insuring traffic gets delivered. Yup, if I were Level-3, I'd be a little irritated at that remark.
[ link to this | view in thread ]
Anonymous Coward, 18 Jul 2014 @ 11:25am

Don't give Level3 too much credit, they are just making fun of a competitor.

That being said, Verizon sucks. They ran an ad in my area for FIOs that said that Verizon customer rankings are 3 times better than Cablevisions, they actually listed the number. 3 times Cablevision rankings put them at 40%, so even though they were better than Cablevision, 60% of Verizon customers thought they sucked.
[ link to this | view in thread ]
Ninja (profile), 18 Jul 2014 @ 11:31am

Re: Read the Level3 blog
This also debunks the need for data caps completely. I'm loving how the ISPs shot their own feet in this issue!
[ link to this | view in thread ]
Anonymous Coward, 18 Jul 2014 @ 11:50am

Re: Read the Level3 blog
Because once they convince those "significant share" locations to pay more money for faster connections, then they can open the floodgates and make it look like they're getting what they're paying extra for... until then, keep the squeeze on, until everyone is willing to pay more.
[ link to this | view in thread ]
Anonymous Coward, 18 Jul 2014 @ 11:56am

Re: Read the Level3 blog
In Europe, where Verizon has to compete against other ISPs, the ports are not congested.

Of course they're not. A few thousand bucks for a line card is chump change compared to the cost of trenching fiber, so why would anyone pay the latter and not the former?

Unless of course...they had an effective monopoly and found it useful to create artificial scarcity in order to lie and extort money from third parties.

Having seen the fallout from Verizon's (and Comcast's) network engineers for many years, I have no doubt that their fraudulent claims about bandwidth scarcity would quickly be exposed if an independent team of independent senior, experienced network engineers came in to do an audit of their operation. (Which is of course why that will never happen: at best, they'll hire shills and hand them a prepackaged script.)
[ link to this | view in thread ]
Baron von Robber, 18 Jul 2014 @ 11:58am

CABLE COMPANY FUCKERY!!
[ link to this | view in thread ]
Anonymous Coward, 18 Jul 2014 @ 12:00pm

If one were to make a complaint to the FTC over their ISP's practices, would it make a difference. I personally am completely pissed off with this situation, and use ATT(no other choice) and would love to file as many complaints as possible so I can actually start getting what I pay for.
[ link to this | view in thread ]
Anonymous Anonymous Coward, 18 Jul 2014 @ 12:02pm

Contrast
I noticed that the fix is a few thousand dollars one time, along with whatever ongoing maintenance and electricity costs.

The plan by Verizon is to extort ongoing payments forever for failing to spend the one time money.

They go to the banks with their RMR (Recurring Monthly Revenue) records in hand, so better jack up that RMR.
[ link to this | view in thread ]
Anonymous Coward, 18 Jul 2014 @ 12:02pm

Open a BBB Complaint
BBB has no regulatory power over a provider, but if Verizon is claiming 70Mbps or some other number, and you are not getting what you are promised, open a complaint. Verizon will need to respond to your complaint. (Well, they don't have too, but it will leave many unresolved complaints, and that doesn't look good.)
[ link to this | view in thread ]
Anonymous Coward, 18 Jul 2014 @ 12:09pm

"But, here’s the other interesting thing also shown in the Verizon diagram. This congestion only takes place between Verizon and network providers chosen by Netflix. The providers that Netflix does not use do not experience the same problem. Why is that? Could it be that Verizon does not want its customers to actually use the higher-speed services it sells to them? Could it be that Verizon wants to extract a pound of flesh from its competitors, using the monopoly it has over the only connection to its end-users to raise its competitors’ costs?"

-Level3 VP of Content and Media David Young

I think you hit the nail on the head Mr. Young. Verizon doesn't want Netflix competing against their U-Verse TV packages. It's the same thing with Comcast.

Then we have Aereo. Which the cable and TV networks have declared year round duck season on.

Internet TV is getting assaulted from all directions by residential monopoly ISPs. The FCC is run by a lobbyist. We also have members of Congress such as Marsha Blackburn, attempting to outlaw competition against AT&T, Comcast, and Verizon.

Customers are getting hit from all sides and in every direction! Free market my ass. America is rigged towards monopolies and blocking competition through crony capitalist laws.
[ link to this | view in thread ]
WDS (profile), 18 Jul 2014 @ 12:13pm

Breaking News
Verizon serves Level 3 with cease and desist letter demanding that they stop telling the truth. Details at 11:00.
[ link to this | view in thread ]
Anonymous Coward, 18 Jul 2014 @ 12:19pm

I see something similar firsthand
So, I'm a sonic.net DSL subscriber - my DSL line is an AT&T circuit which terminates at sonic.net's data center. Keep in mind, this means that the circuit's full bandwidth is supposedly "guaranteed" from my house to sonic.net's data center, where my first actual internet hop is located.

I track my ping times from my router to my first hop throughout the day, and during peak times, the latency and bandwidth suffer immensely. This is certainly not my ISP's fault - it is in fact AT&T's fault for not having enough bandwidth available between my house and sonic.net's data center to service the virtual circuit that they've provisioned. Every time I call sonic.net to report severe latency issues, they are forced to report the problem to AT&T to fix the circuit and infrastructure behind it.

This gets really irritating during peak hours, because EVERY connection is slow for me - whereas AT&T subscribers may at least get decent performance for some websites with better peering than my circuit does for its first hop.

Either way, I still refuse to use AT&T as my ISP, even if they own the circuit, as I have better privacy and support with sonic.net than I would ever have with AT&T, and no monthly caps.
[ link to this | view in thread ]
Anonymous Anonymous Coward, 18 Jul 2014 @ 12:20pm

Re: Breaking News
Backstopped with a lawsuit threat, since the allegations of truth telling were true, they are defamatory.
[ link to this | view in thread ]
orbitalinsertion (profile), 18 Jul 2014 @ 12:21pm

And the Level 3 article is down.
[ link to this | view in thread ]
Anonymous Coward, 18 Jul 2014 @ 12:26pm

Re: I see something similar firsthand
I see the same thing. I'm currently a Verizon DSL subscriber -- no other choice. However, prior to this I was a DSL subscriber of a regional ISP which resold Verizon DSL.

Therefore I still have the same copper, the same CO, the same everything as I had before.

However now I have about half the performance that I had previously and of course technical support is miserably incompetent, able only to read from a script and give worthless inapplicable suggestions. (Or try to get me to upgrade to FIOS, which is not available in my area and WILL NEVER BE AVAILABLE IN MY AREA because of course Verizon has no intention of spending the money to lay fiber out here when it's much more profitable to sell to infill customers in more densely populated areas.)

I'm just out of range of a wireless ISP and have begun resorting to blatant bribery (e.g., a good bottle of bourbon) in an attempt to persuade the owner to please please pllleeeease add a tower to extend their footprint to one that covers me.
[ link to this | view in thread ]
PRMan, 18 Jul 2014 @ 12:28pm

Re: Open a BBB Complaint
I've heard that they just pay the BBB to make it go away without your input. Is this true?
[ link to this | view in thread ]
lucidrenegade (profile), 18 Jul 2014 @ 12:31pm

Re:
Interesting. The link on their main blog page is still there though.

http://blog.level3.com/
[ link to this | view in thread ]
lucidrenegade (profile), 18 Jul 2014 @ 12:31pm

Re: Re:
If I had to guess I'd say the damn lawyers got involved.
[ link to this | view in thread ]
Anonymous Coward, 18 Jul 2014 @ 12:32pm

Re: Re: Open a BBB Complaint
Certainly, this is America man.
[ link to this | view in thread ]
Baron von Robber, 18 Jul 2014 @ 12:34pm

Re:
Wonder if Level3 got a C&D letter or a copyright take down. :/
[ link to this | view in thread ]
Desert Viking, 18 Jul 2014 @ 12:37pm

Hope?
This might explain some of the inconsistencies in 'bandwidth' that I've seen recently. Google Fiber is available in my city. They sell two levels, a 'free' (30 dollar connection fee) version for 5mbps and and a 1gbps (yes, gigabit) version for 70$ a month. The obvious idea is to bait you into the free at a version just slow enough to make you want something better and go all the way up.

But here's the problem: they actually deliver 5mbps. So the stats that say it shouldn't be enough are wrong. I've had all sorts of internet connections in the past few years, and its never been better.

So yeah, internet providers customarily lie about their bandwidth. And I'm glad its becoming an issue.
[ link to this | view in thread ]
Anonymous Coward, 18 Jul 2014 @ 12:42pm

Does this fall under extortion?
At the very least this is proof of a lot of false advertising by Verizon and others. But is this enough proof that Verizon is engaging in illegal acts of extortion of Netflix?

If we had a functioning justice system, they'd be throwing the books at Verizon and the other ISP's trying to do this crap.
[ link to this | view in thread ]
Anonymous Coward, 18 Jul 2014 @ 12:45pm

Re: Re: I see something similar firsthand
The wireless options around here are pretty dismal - I laughed out loud on the phone when the sales lady told me the $65/mo wireless plan had a 30gb/mo limit...
[ link to this | view in thread ]
ECA (profile), 18 Jul 2014 @ 12:47pm

funny
And this is a problem, HOW??
that THOSE companies dont HAVE facilities to UPGRADE their OWN SYSTEMS..
I will BEt they have cut the number of Inter-connects to the MINIMUM required...and not ONLY do they need an inter-connect, they need a WHOLE SERVER setup...

when the internet took off, in Portland Or, it took the Companies 6 MONTHS to redo ALL the MAIN connections to Fiber. they were satisfied with a 6% Copper wired service..It hardly ever HIT the Over load..
But after Cellphones and Internet took over, the amount of bandwidth needed went upto 80% in 1 year.

What this means is that for ALL the people in the Metro area, only 6% of those people and business NEEDED access tot he net or other services.. So they didnt NEED Extra PEOPLE, SERVICES and the traffic was able to be handled with 6% coverage..
After the NEt and CELLPHONEs took over 80% of the people and business needed Some sort of access to communications..
And that was in the LATE 90's..
[ link to this | view in thread ]
Anonymous Coward, 18 Jul 2014 @ 12:48pm

It's very very easy to know Verizon is full of shit by simply using a VPN to access Netflix.
[ link to this | view in thread ]
vastrightwing, 18 Jul 2014 @ 12:52pm

and there's more
The other reason verizon doesn't want to upgrade the peering points is they can get ignorant consumers to pay more for more useless bandwidth. They tried to play that card with me. This is why I'm with RCN now.
[ link to this | view in thread ]
Anonymous Coward, 18 Jul 2014 @ 12:52pm

"If that’s the case, we’ll buy one for them. Maybe they can’t afford the small piece of cable between our two ports. If that’s the case, we’ll provide it. Heck, we’ll even install it."

---- I mean this is it right? End of discussion.
[ link to this | view in thread ]
Trevor, 18 Jul 2014 @ 1:08pm

Well
The thing is, Verizon never promised to deliver 50 or 70mbps. They promised UP TO 50 or 70mbps.

Where I currently live, we had to get Verizon DSL, and were "promised" up to 3mbps (Didn't care, asked for the minimum to stream Netflix for the kids). Not once in the past year has it peaked above 1.5mbps.

We are moving in a few weeks, and the only available internet is FiOS. We were "promised" up to 50mbps. We shall see. If it breaks 10mbps, I'll be impressed.

I have a question:

If Verizon is saying (as is depicted in the diagram) that they are only utilizing "48% peak utilization" from the terminal to wires to my house, shouldn't netflix stream at the promised rate seeing how they have room for the bandwidth, but aren't using it? From what I've experienced, Netflix still lags.

The diagram looks like the problem is all Netflix to Verizon, but Verizon to my house has no hiccups (ha). As with a bottle neck, once past the blockage shouldn't the speed pick back up? If you're driving down the freeway and come across an accident, everyone slows down. Once past the accident, it's open sailing for a while. Isn't this the same concept? Or is the diagram misleading, and the red line should extend all the way from Netflix to my house?

And if it does, isn't that the point of buffering? To catch it up so it doesn't trickle down to me?
[ link to this | view in thread ]
Anonymous Coward, 18 Jul 2014 @ 1:09pm

Re: I'd say they are defaming Level-3
In all honesty, directly peering with the content provider is usually preferred no matter who is providing the transit network. And I would say that Level3 is proud of the fact that they are a "middleman network", in the ISP would it's called a "Tier 1" provider able to reach globally.
[ link to this | view in thread ]
Anonymous Coward, 18 Jul 2014 @ 1:14pm

Re: Well
It's not the same concept. The effective end-to-end throughput that you'll see is (to a good first approximation) limited by the slowest link.

Buffering (if in play) can help mitigate some of the variations in throughput, but it can't compensate for a fundamental lack of end-to-end speed.
[ link to this | view in thread ]
m3mnoch (profile), 18 Jul 2014 @ 1:14pm

watching proof of the slowdown
this guys has my favorite example:
https://www.youtube.com/watch?v=5vs3QhEx_3w

he vpn's in (which, incidentally encrypts his network packets from the netflix packet sniffer verizon is running) and gets a giant burst of speed.

who knew? vpns are FASTER!
[ link to this | view in thread ]
Anonymous Coward, 18 Jul 2014 @ 1:14pm

Re: Re: Re: I see something similar firsthand
I think he's talking about "Fixed Wireless". Typically, you would get a hard limit though I'm sure some providers might try if they can get away with it.
Check here: WispA map
[ link to this | view in thread ]
Anonymous Coward, 18 Jul 2014 @ 1:30pm

Re: Re: Re: Re: I see something similar firsthand
Yep, that's what I'm talking about - line of site, microwave transmission.
[ link to this | view in thread ]
orbitalinsertion (profile), 18 Jul 2014 @ 1:32pm

Re: Re:
Yes, that link and blurb were still there, in fact, the link was listed as a current article on the 404 page that was delivered.

However, it simply seems to be back up now. That was...brief.
[ link to this | view in thread ]
Anonymous Coward, 18 Jul 2014 @ 1:35pm

Re: Re: Re: Re: Re: I see something similar firsthand
For reference, this is the provider I'm referring to... looks like they've got a much larger plan selection now, but some of their caps and prices are still pretty ridiculous. Last time I spoke with them was about 3 years ago:

http://www.smarterbroadband.com/Pricing.htm
[ link to this | view in thread ]
Khaim (profile), 18 Jul 2014 @ 2:14pm

I wonder what would happen if some Verizon tech "accidentally" installed and connected another 4 ports. If they don't have the hardware already on site, they certainly have it back in a warehouse somewhere.

Sure, he'd probably get fired, but imagine the PR dance Verizon would have to do to explain how all the congestion magically disappeared.
[ link to this | view in thread ]
Anonymous Coward, 18 Jul 2014 @ 2:32pm

Re: watching proof of the slowdown
Technically, he's just choosing a different route to the netflix servers - a route that isn't artificially congested.

Supposedly, routing is supposed to choose the most efficient routes, but if there aren't enough statistics for routers to make routing decisions (such as the congestion level upstream), they may make poor decisions. By connecting to a VPN, you're basically forcing the route to go through another 3rd party server, so this can indeed avoid the largely congested areas, and thus his bandwidth becomes better. In theory, his latency might actually be worse, but for streaming video, that seldom matters much.
[ link to this | view in thread ]
John Fenderson (profile), 18 Jul 2014 @ 2:42pm

Re: Open a BBB Complaint
The BBB is worthless.
[ link to this | view in thread ]
Rabbit80, 18 Jul 2014 @ 2:52pm

Re: funny
Do you really SHOUT that many times in a sentence when you talk? Would irritate the hell out of me! (And does on t'internet as well!)
[ link to this | view in thread ]
Anonymous Coward, 18 Jul 2014 @ 2:57pm

What makes no sense, from a network administrator point of view, is that the same routers are used for everything. Congestion would happen elsewhere, not just on netflix. Unless Verizon have dedicated routers hooked just for netflix... but that's very unlikely.
[ link to this | view in thread ]
Anonymous Coward, 18 Jul 2014 @ 3:03pm

Re:
The entire point of this article is that Verizon has connections to Level 3, and all of those connections are saturated, but Verizon refuses to add more. Other sites that use Level 3 also deal with this congestion - twitch, for example, stutters during peak Netflix/Youtube traffic for the same reasons.
[ link to this | view in thread ]
Rabbit80, 18 Jul 2014 @ 3:06pm

Re:
Not sure if that's insightful or funny, so voted both!

This is clearly Verizons fault, though I have a question for anyone good at maths...

Given that Verizons capacity at its own weakest link is at 65%, and that Netflix accounts for over 1/3rd the US traffic.. What impact would full speed Netflix have on Verizons network?

Its a tricky question given that:
• We don't know the ratio of Verizons traffic once on their network that is Netflix
• Verizon's contribution to the total US traffic
• The number of new subscribers if Netflix had a good connection on Verizon
• The increase in total traffic on Verizons network

Would like to read theories on the above?
[ link to this | view in thread ]
Anonymous Coward, 18 Jul 2014 @ 3:54pm

Re: Re:
Indeed, Level3 is no small network - and I'm sure Verizon LOVES the idea that Level3 customers (or Level3 themselves) might even start paying Verizon for "fast lane" access to Verizon's customers.

What really needs to happen is Verizon's customers call and complain that Verizon is not providing adequate peering to a network the size of Level3. Maybe if enough knowledgeable customers complain they'll be forced to actually address te issue.
[ link to this | view in thread ]
Anonymous Coward, 18 Jul 2014 @ 4:32pm

Re: Re: Re: Re: Re: Re: I see something similar firsthand
Did you check WispA? Looks like there are two other options. https://www.cal.net/ and http://www.calwisp.com/
Hopefully, either of them can hit you. Perhaps if your out in the sticks enough on a hill offer to allow a rohn tower to extend their footprint. I used to do wireless until it really wasn't feasible to be in the ISP business.
[ link to this | view in thread ]
andypandy, 18 Jul 2014 @ 5:03pm

Re: Re: Read the Level3 blog
I seriously hope that when this is all settled and they are fined or Netflix takes hem to court for fraud as they did receive funds to make netflix faster for their customers and have been caught out purposefully blocking traffic. Maybe the FCC will make a rule that no braadband supplier is allowed to make caps as part of a contract and that includes the mobile networks if it is found they are doing anything like this.

I just hope the FCC does something about this and makes it very vocal so that all big broadband customers are aware of the problem with isp's and where their customers can demand the problems be resolved or report them to the FCC or even sue them,A simple question, could verizon customers not sue them now that there is clear evidence they are not interested in supplying what they have sold.
[ link to this | view in thread ]
This comment has been flagged by the community. Click here to show it

Whatever (profile), 18 Jul 2014 @ 5:22pm

weird
What doesn't make sense here is that Netflix seemed bound and determined to use no other networks except level3. I see by Verizon's own diagram plenty of unused connectivity with other providers. Could it be that Netflix being stubborn about their sole carrier (can you say discount for volume) gets them into trouble?

Also, it is really fair to oblige the ISPs to upgrade their connections to Level3 (at a price, natch) to provide more bandwidth for Netflix to make money? The diagrams clearly show that Netflix demands and requires a substantial amount of bandwidth above and beyond reasonable use.

Yes, Verizon could "fix" the problem, by giving in and spending more money to support Netflix business model, and to support their decision to use a single carrier for all of their bandwidth.

The Verizon graphic clearly shows that Netflix is a bandwidth hog and their business model and profit is based on it. Is it really fair to ask the ISPs to spend more to support their business models?
[ link to this | view in thread ]
ltlw0lf (profile), 18 Jul 2014 @ 5:48pm

Re: Hope?
But here's the problem: they actually deliver 5mbps. So the stats that say it shouldn't be enough are wrong. I've had all sorts of internet connections in the past few years, and its never been better.

God I wish I could consistently get 5mbps from my provider. Then again, I'd pay $70/mo for the gbps, since I already pay $70 for up to 70mbps (which is consistently around 3mpbs on a good day.)
[ link to this | view in thread ]
dog, 18 Jul 2014 @ 8:15pm

Re: Re: Hope?
your 70 meg connection would be your in network connection speed, and not the speed that the whole internet runs.

what speed does techdirt come down at? do they need a better connection or better peering to deliver ?
[ link to this | view in thread ]
Anonymous Coward, 18 Jul 2014 @ 8:25pm

Re:
Should be. None of the shills posting on the Level 3 blog suggested that this technical fix wouldn't work. None of them mentioned it at all. If they really thought it wasn't a solution then they would have disputed this (in fact, I'm a little surprised they didn't try to dispute it anyway). Instead they were completely silent on the issue. This indicates they know it would solve the problem, and since Level 3 is offering to pay for parts and labor, they really have no argument at all.
[ link to this | view in thread ]
That One Guy (profile), 18 Jul 2014 @ 8:37pm

Re: weird
Netflix's 'business model'? You mean sending people what they pay for, through the connections that they've paid for? That 'business model'?

Verizon is intentionally not upgrading their service, instead letting it congest, punishing the customers who've already paid for that service, in order to shake down Netflix for 'preferential' treatment on the network, and since upgrading the connections would remove the need for that 'preferential treatment', you can bet that none of the money they are shaking Netflix down for will be used to upgrade the currently congested systems.

Verizon, and likely the others pulling the same stunt, could 'solve' the problem without difficulty, and for cheap at that, and instead they're letting it continue and grow, all so they can shake down Netflix, and get them to pay, again, for content that's already been paid for.
[ link to this | view in thread ]
ltlw0lf (profile), 18 Jul 2014 @ 9:58pm

Re: Re: Re: Hope?
your 70 meg connection would be your in network connection speed, and not the speed that the whole internet runs.

I'm not, nor is anyone else here that I'm aware of, saying that the whole internet runs at 70Mbps. I'd love to get a 70 meg connection from my house to the edge of my cable internet provider's network. And maybe something close to that going from their network to their peers and transits.

After all, the internet is wireless and runs in a little black box with a red light on it...I saw it on TV once when Roy and Moss showed it to Jen and she proceeded to break it. Do me a favor...don't ever break the internet.

what speed does techdirt come down at? do they need a better connection or better peering to deliver ?

My co-loc provider provides a minimum of 100 Mbps, I get at least 1 Gbps on my servers, and I know he is connected to Level 3 at 10 Gbps or more. My work has several connections that run in the 600 Mbps to 10 Gbps range, and I know that they have bound 10 Gbps connections. My mirroring sessions at work routinely sustain 20 - 60 Mbps thoroughput across the country. And I suspect Floor64, being so close to Silicon Valley probably has the same connectivity at their co-loc as well.

On the real internet, not the AOL bubble version Comcast/Verizon/AT&T tend to believe they live on, bandwidth is cheap and getting far cheaper. My cloud networks measure bandwidth at MB at about 2.4 hundreths of a penny each, and my co-loc doesn't even charge me for bandwidth because the cost is so lo that he can recover it through my monthly connection fees.

This is little more than tin-pot sheriffs setting up their tollbooths, attempting to rival the government on collecting taxes for nothing, just like they did with the billions we spent on upgrading the broadband infrastructure for "the information superhighway" and E911 and K-12 and FIOS and every other boondoggle they've asked for money, changed the goal posts midstream, and then asked the government to forget the requirements and then pocket the money for the services they didn't provide.
[ link to this | view in thread ]
This comment has been flagged by the community. Click here to show it

Anonymous Coward, 18 Jul 2014 @ 10:02pm

Re: weird
Holy fuck you are a dumbass. Netflix already paid Verizon to fix their goddamn shit:
http://arstechnica.com/tech-policy/2014/04/netflix-and-verizon-reach-interconnection-deal-to-sp eed-up-video/
[ link to this | view in thread ]
Anonymous Coward, 18 Jul 2014 @ 10:20pm

Re: weird
And Jesus Tapdancing Christ it's like you don't even read the snippets from the source Mike gives you:
http://blog.level3.com/global-connectivity/verizons-accidental-mea-culpa/

Verizon has confirmed that everything between that router in their network and their subscribers is uncongested – in fact has plenty of capacity sitting there waiting to be used. Above, I confirmed exactly the same thing for the Level 3 network. So in fact, we could fix this congestion in about five minutes simply by connecting up more 10Gbps ports on those routers. Simple. Something we’ve been asking Verizon to do for many, many months, and something other providers regularly do in similar circumstances. But Verizon has refused. So Verizon, not Level 3 or Netflix, causes the congestion. Why is that? Maybe they can’t afford a new port card because they’ve run out – even though these cards are very cheap, just a few thousand dollars for each 10 Gbps card which could support 5,000 streams or more. If that’s the case, we’ll buy one for them. Maybe they can’t afford the small piece of cable between our two ports. If that’s the case, we’ll provide it. Heck, we’ll even install it.

If you can't fucking tell, you've got shit on your face.
[ link to this | view in thread ]
techflaws (profile), 18 Jul 2014 @ 10:22pm

Re: weird
Someone above waited for shills to try to spin this but it's only you failing again at making a point. Pathetic.
[ link to this | view in thread ]
Atkray (profile), 18 Jul 2014 @ 11:23pm

Re:
I thought 3 X 0 = 0
[ link to this | view in thread ]
Sheogorath (profile), 19 Jul 2014 @ 12:03am

To use a shipping company analogy
As a customer, I am paying the shipping company to deliver my package. If a few million people order stuff at the same time, it's understandable that there may be some delays. However, it would be completely ludicrous for the shipping company to blame Amazon or whoever for sending out too many packages.
[ link to this | view in thread ]
Anonymous Coward, 19 Jul 2014 @ 12:05am

Re: Re:
You're not using Verizon math - where 3*0 is 40GB/s.
[ link to this | view in thread ]
Sheogorath (profile), 19 Jul 2014 @ 12:11am

Re: weird
The diagrams clearly show that Netflix demands and requires a substantial amount of bandwidth above and beyond what was reasonable use twenty years ago.
So Verizon need to update their network. Your point?
[ link to this | view in thread ]
This comment has been flagged by the community. Click here to show it

Whatever (profile), 19 Jul 2014 @ 3:52am

Re: Re: weird
If you would quit cussing and jumping around, you might actually notice something:

I AGREE WITH YOU!

However, Netflix is paying for a reason, because their network demands are over and above what is reasonable for any peering arrangement. Yes, verizon could have just kept adding bandwidth to satisfy the Netflix business model, but where do you think the money comes from to do it? Those interconnects are not free to set up, sad to say.

Level3 generally charges for connections to their network - perhaps they want Verizon to spend a whole lot more money with them?
[ link to this | view in thread ]
Anonymous Coward, 19 Jul 2014 @ 4:21am

In other cases there may be traffic imbalances, but the networks or content providers have entered into paid arrangements with us to ensure connections and capacity to meet their needs for their out-of-balance traffic.

Because they connect to domestic users, and provide asymmetrical connection speeds, and ban servers, Verizon will always have a traffic imbalance with backbone providers like level 3, and content providers like Netflix, its the nature of the business they are in. Therefore they will always have an excuse to charge for peering arrangements, and double dip for the delivery bandwidth they promised their customers. That they know this is obvious because their touted speeds download to the customer, and you have to look hard to find what uploads speeds they are offering.
[ link to this | view in thread ]
Anonymous Coward, 19 Jul 2014 @ 4:59am

Re: Re: I see something similar firsthand
I'm just wondering how many AT&T customers, in this fellow's situation, but without the knowledge, have "upgraded" to and are paying for FIOS, when it physically isn't available to their home? Could be a great cash cow for AT&T.
[ link to this | view in thread ]
is4u2p, 19 Jul 2014 @ 7:32am

Bull!
I could argue it is showing the Verizon's internal bandwidth is 3x the size of the providers netflix is using.
[ link to this | view in thread ]
is4u2p, 19 Jul 2014 @ 7:35am

Re: Bull!
Here's another thing... You don't want to run your line at 100% because then you get collisions and that's a bad thing.
[ link to this | view in thread ]
Justin Gruenberg (profile), 19 Jul 2014 @ 7:46am

Re: To use a shipping company analogy
If I pay FedEx for overnight service, I expect my packages to be delivered... overnight. But I'm a reasonable person. If FedEx gets an unexpected flood of packages, maybe I don't get the quality of service I paid for on a single days shipments. If it becomes a pattern, I'd expect FedEx to hire more people, re-evaluate their logistics, and, you know, do something about it.

I suppose I'm being a bit lazy in that analogy on who pays and who the customer is. In this analogy, the shipper is the customer and the shipper pays. Asking the receiver to pay, in addition to the shipper, would be pure rent seeking.

The fact is, I pay my ISP to connect me to the internet. I expect that (1) I get connected to the entire internet, and (2) the ISP reasonably peers with other networks in order to get good speeds to the other networks that I request data from. If ISPs were truly customer-centered, this is a no brainer. The customers want Netflix, so any ISP worth anything is going to make sure there is adequate bandwidth to Netflix.
[ link to this | view in thread ]
Josh in CharlotteNC (profile), 19 Jul 2014 @ 7:54am

Re: Re: I'd say they are defaming Level-3
In all honesty, directly peering with the content provider is usually preferred no matter who is providing the transit network.

Not necessarily. There are situations - including this one - where it makes a lot of sense to work with a single transit provider (or for redundancy 2 or 3) instead of needing to both pay the transit providers AND directly peer with a dozen or more ISPs.

Netflix works with Level3 and Cogent. Both are Tier 1 networks - which means that's all Netflix needs to work with to ensure that their content is reachable by anyone on the internet. So long as they can provide sufficient speed, Netflix wouldn't need to worry about making deals with Verizon, and AT&T, and Comcast, and Time Warner, and Cablevision, and Google Fiber, and Qwest, and so on.

And Verizon, AT&T and Comcast know this. Verizon is also a Tier 1 transit provider. So is AT&T. Comcast is trying to be. They all compete with Level3 and Cogent in the transit business - which is cutthroat and there's good money in it. Here's the thing. Level 3 and Cogent do not compete with those companies in the residential broadband business.

So it comes down to this: Verizon and Comcast can abuse their monopoly position over their residential business in order to force Netflix to directly peer with them, costing Level3 that transit business with Netflix, while losing nothing (since their residential customers have no choice and can't move to another provider).
[ link to this | view in thread ]
techflaws (profile), 19 Jul 2014 @ 8:36am

Re: Re: Re: weird
However, Netflix is paying for a reason, because their network demands are over and above what is reasonable for any peering arrangement.

It's not. Netflix already paid their ISP and the customers paid theirs. And as you can see in the diagram, there's plenty of bandwith to spare - for the costs already paid.
[ link to this | view in thread ]
This comment has been flagged by the community. Click here to show it

Anonymous Coward, 19 Jul 2014 @ 9:13am

Re: Re: Re: weird
Read.
The.
Fucking.
Article.
You.
Twat.
[ link to this | view in thread ]
Anonymous Coward, 19 Jul 2014 @ 9:24am

Re: Re: Bull!
It's only running at 100% where Level3 and Verizon meet. Both have internal bandwidth to spare.
Which is why Whatever needs to pull his face out of Verizon's ass long enough to RTFA.
[ link to this | view in thread ]
alternatives(), 19 Jul 2014 @ 11:58am

Re: weird
What doesn't make sense here is that Netflix seemed bound and determined to use no other networks except level3

Except that isn't true.

But you keep shining on you crazy diamond.
[ link to this | view in thread ]
Anonymous Coward, 19 Jul 2014 @ 2:27pm

Verizon doesn't want to upgrade their networks. That's why.
[ link to this | view in thread ]
nasch (profile), 19 Jul 2014 @ 8:57pm

Re: weird
Also, it is really fair to oblige the ISPs to upgrade their connections to Level3 (at a price, natch) to provide more bandwidth for Netflix to make money?

Did you read the part that states the upgrades are very cheap, and Level3 offered to pay for them?

The Verizon graphic clearly shows that Netflix is a bandwidth hog and their business model and profit is based on it.

What it shows is that Verizon isn't interested in giving their customers what they've promised them.
[ link to this | view in thread ]
CJ (profile), 19 Jul 2014 @ 10:20pm

I already said it was Verizons fault!
Why? because of a remark by Verizon months ago and that was they said "You pay us we can fix the problem" I also said that if they can fix it if you pay then they can fix it for all customers. Read between the lines this is all about money and control and nothing else Verizon doesn't care about Netflix they don't care about the consumer either they just want to increase their bottom line, and jack up their paychecks. The thing is in the end the Internet will look and act like cable and the public will be asking the FCC why they did not see this coming. The FCC wont see it because of who is running the FCC.
[ link to this | view in thread ]
Sheogorath (profile), 19 Jul 2014 @ 10:30pm

Re: Re: To use a shipping company analogy
Which was kinda my point. Verizon should be giving their customers what they paid for instead of disregarding them and blaming Netflix for its woefully outdated network.
[ link to this | view in thread ]
This comment has been flagged by the community. Click here to show it

Whatever (profile), 19 Jul 2014 @ 10:54pm

Re: Re: Re: Re: weird
It's not. Netflix already paid their ISP and the customers paid theirs. And as you can see in the diagram, there's plenty of bandwith to spare - for the costs already paid.

Not true. Netflix paid for theirs, but the ISP customers did not pay for an endless fountain of bandwidth at a given speed end to end.

See, to me, this is where the problem lies. Your 5meg connection doesn't mean your connection to everything on the net will flow at 5 meg per second. It's a peak speed, it's the speed that you are connected to the head end of you ISP - but is no assurance of end to end performance to the entire internet.

Most people miss that point, and it's pretty key. Your connection speed does not mean that you get that bandwidth x 60 seconds x 60 minutes x 24 hours per day x 30.4 days a month (average month length). It's fastest that things will be delivered - the rest may be slower, especially in times of peak demand.

The graphic says it all: Netflix takes a link that would be about 40-50% (on par with the other peering) and wipes it out with heavy, constant, long term usage - 60% or so of the total connection. That's pretty out there.

If Netflix learned how to use more that one provider for their bandwidth, they would likely be able to do a lot more... why don't they have more than one arrangement?
[ link to this | view in thread ]
This comment has been flagged by the community. Click here to show it

Whatever (profile), 19 Jul 2014 @ 10:54pm

Re: Re: Re: Re: weird
ad.
homs.
suck.
[ link to this | view in thread ]
This comment has been flagged by the community. Click here to show it

Whatever (profile), 19 Jul 2014 @ 10:58pm

Re: Re: Re: Bull!
Hi.

I read the article carefully. It's carefully crafted by Netflix and Level3 to try to make Verizon look like the guilty party, when in fact it's equally clear that they aren't playing well with others and are not allowing their stuff to peer through any other provider - likely to keep their own costs down.

Verizon has more peering running 40% or so. If Level3 peered with those companies and announced their routes through them, then at least some of the traffic could be routed that way. In fact, it would be fairly simply for Netflix in their software to poll for multiple routes and select the best one - instead of resolutely trying to jam their stuff down a full pipe.

Why don't they do it? I suspect that Level3 isn't playing well with others, and is trying to avoid sending traffic through other providers, a true bottom line choice. Basically, they only route internal, unless it's absolutely unavoidable.

Opening up their routing, and the Verizon problem would have disappeared.

Try not to just FTFA, try actually thinking a moment.
[ link to this | view in thread ]
This comment has been flagged by the community. Click here to show it

Whatever (profile), 20 Jul 2014 @ 12:31am

Re: Re: Re: I'd say they are defaming Level-3
The thing is, if Netflix (and Level 3 for that matter) improved their peering with other tier 1 providers, there would be no issue, as they could route a portion of their traffic in other ways and not overload Verizon.

It appears that the single mindedness of Level3 (and certainly some seeming willful blindness from Verizon) means that a single source is overloaded, and other work just fine.

It probably explains the Level 3 attack here: They realize they are part of the problem and not the solution.

As for trying to cut Level 3 out, just remember that Netflix traffic still has to get to the new paid peering point in some manner. It may end up being a "netflix" peering point instead of a Level 3 peering point on a purely technical level, but you can bet because of their agreements that Netflix will use Level3 right to that point anyway, cutting out nobody.
[ link to this | view in thread ]
nasch (profile), 20 Jul 2014 @ 7:41am

Re: Re: Re: Re: Re: weird
The graphic says it all: Netflix takes a link that would be about 40-50% (on par with the other peering) and wipes it out with heavy, constant, long term usage - 60% or so of the total connection.

The graphic says it all: Verizon takes a link that would be about 40-50% (on par with the other peering) and wipes it out by refusing to do ordinary industry standard upgrades on it.

FTFY.

The point is not that Verizon customers are expecting their theoretical top speed all the time, the point is that they're expecting (or perhaps "dreaming of" is a better term) Verizon to make a best effort to deliver whatever traffic the customer demands. Instead, they are intentionally screwing up the connection. I don't understand why you're defending this, is it just to disagree with TechDirt?

If Netflix learned how to use more that one provider for their bandwidth, they would likely be able to do a lot more... why don't they have more than one arrangement?

So rather than call on Verizon to just do what they should have been doing all along, you want Netflix to alter their strategy to accommodate the extortionist.
[ link to this | view in thread ]
nasch (profile), 20 Jul 2014 @ 7:44am

Re: Re: Re: Re: Bull!
it's equally clear that they aren't playing well with others and are not allowing their stuff to peer through any other provider - likely to keep their own costs down.

So what? Why should they have to peer through other providers? You keep saying this as though it's Netflix's duty, but if they get the best deal from Level3, what's the problem with that? If Verizon didn't intentionally screw them, nobody would even notice.

Why don't they do it? I suspect that Level3 isn't playing well with others, and is trying to avoid sending traffic through other providers, a true bottom line choice. Basically, they only route internal, unless it's absolutely unavoidable.

Opening up their routing, and the Verizon problem would have disappeared.

Got a citation for that, or are you making stuff up?
[ link to this | view in thread ]
nasch (profile), 20 Jul 2014 @ 7:45am

Re: Re: Re: Re: I'd say they are defaming Level-3
and certainly some seeming willful blindness from Verizon

Some seeing? So you're saying Verizon might not be intentionally neglecting their routing hardware? Why are you such a Verizon cheerleader?
[ link to this | view in thread ]
Anonymous Coward, 20 Jul 2014 @ 8:49am

Re: Re: Re: Re: I'd say they are defaming Level-3
Willful ignorance - you wear it well.
[ link to this | view in thread ]
orbitalinsertion (profile), 20 Jul 2014 @ 9:27am

Re: Re: Re: Re: Re: weird
Not true. Netflix paid for theirs, but the ISP customers did not pay for an endless fountain of bandwidth at a given speed end to end.

No. You are just taking the piss-poor business model of consumer ISPs as default and proper. They never provides the speeds they advertise, with their "up to" caveat (already ridiculous), and the same sort of stupid "overbooking" ploy as airlines, where that practice, while poor, actually has some merit.

This is easily seen by the fact that they have existing rackspace with empty router card slots, and existing dark fiber. They paid to lay fiber which they basically are not using because they want to extort more money from everyone. Everyone who already overpays for substandard service.

Unlike 10 years ago, when you could possibly make an argument that people paying to look at a website once a day or send a couple emails were paying the same amount as heavy downloaders, P@P users, and streaming fans, the whole internet has a lot of high throughput applications these days, and is built for it, including Verizon's co-los and last mile networks. They are just playing games with it.
[ link to this | view in thread ]
orbitalinsertion (profile), 20 Jul 2014 @ 9:28am

Re: Re: Re: Re: Re: weird
People who don't know what an ad hominem argument is, but like to use the term incorrectly, suck more.
[ link to this | view in thread ]
Josh in CharlotteNC (profile), 20 Jul 2014 @ 11:37am

Re: Re: Re: Re: I'd say they are defaming Level-3
You have no idea what is going on. Willfull blindness and outright lies are the only thing you're good at.

The thing is, if Netflix (and Level 3 for that matter) improved their peering with other tier 1 providers,

Read the article. Read Level3's blog. Level3 is willing to buy Verizon the cards, cabling, and install them. I bet they'd even pay a network engineer to update a few routing tables for Verizon. But Verizon won't let them - because if Level3 could fix the problem in a few hours, Verizon would no longer have the threat hanging over Netflix and Level3's heads.

If I was Level3, I would be doing more than just blog posts. I would be throwing fire, making very loud public complaints. I'd be pounding on the FCC's and the FTC's doors. I'd be digging up every penny Verizon has spent on lobbying and every campaign donation.

Single-mindedness of Level3? Completely unarguable fact time: it is 100% Verizon's fault for not upgrading their own infrastructure. Verizon won't spend a few thousand bucks to help hundreds of thousands of their own customers.

Give me one single fact that disputes all of this. Just one tiny piece of information that is based in reality and not whatever delusional universe you live in.
[ link to this | view in thread ]
This comment has been flagged by the community. Click here to show it

Whatever (profile), 20 Jul 2014 @ 5:06pm

Re: Re: weird
Aside from the insults and ad homs, you understand that the whole discussion is about the situation BEFORE that agreement, right?

I'll let you call yourself names now.
[ link to this | view in thread ]
Whatever (profile), 20 Jul 2014 @ 5:13pm

Re: Re: Re: Re: Re: I'd say they are defaming Level-3
Read the article. Read Level3's blog. Level3 is willing to buy Verizon the cards, cabling, and install them. I bet they'd even pay a network engineer to update a few routing tables for Verizon. But Verizon won't let them - because if Level3 could fix the problem in a few hours, Verizon would no longer have the threat hanging over Netflix and Level3's head

So let's say they did that. How much more traffic would be acceptable? How much can Verizon accept before their own internal network becomes overloaded?

The Level3 "free" solution that you point to isn't free at all. It's just bait for a long series of expensive network upgrades to try to keep up with Netflix business model. It's never ending, capitulating with them today pretty much creates a situation where Verizon would be required to upgrade their network to support anyone who showed up with a bandwidth intensive business model.

Verizon won't spend a few thousand bucks to help hundreds of thousands of their own customers.

If it was only a few thousand bucks. You only have to look at the chart to understand. Level3 appeared to want to double the bandwidth coming in. That would have been enough to overload other points in Verizon's network. More upgrades. Level3 comes back to add MORE connections (because Netflix is so good and cheap!) and Verizon has to upgrade more stuff. Verizon can't win.

The money they are getting in this deal is what will end up paying for those network upgrades in the long run, and at the same time sets a hard cap on what comes in - if they want more, they can pay for it.
[ link to this | view in thread ]
BernardoVerda (profile), 20 Jul 2014 @ 5:41pm

Re: Re: Re: Re: Re: Re: weird
Yes, the trolls love to pretend that calling said trolls stupid on the bases of their already proven-to-be-wrong arguments, is somehow a logical fallacy, equivalent to calling the argument wrong because of who made it.

Is ithis deliberate sophistry? Or does that kind of confused thinking explain their stubborn adherence to those unsupportable opinions in the first place?
[ link to this | view in thread ]
BernardoVerda (profile), 20 Jul 2014 @ 5:55pm

Re: Re: Re: Re: Re: Bull!
He's making it up. Level 3 has already pointed out -- with documentation -- that this alleged "problem" only crops up when they are dealing with an ISP which has an effective monopoly lock on an end-user customer base (and 6 times out of 7, that turns out to be one of the big, bad, American consumer ISP services).
[ link to this | view in thread ]
nasch (profile), 20 Jul 2014 @ 6:24pm

Re: Re: Re: Re: Re: Re: I'd say they are defaming Level-3
It's just bait for a long series of expensive network upgrades to try to keep up with Netflix business model.

It's upgrades to keep up with VERIZON'S business model, which is delivering internet service to their customers. In places with competition, ISPs have no trouble doing these upgrades. It's only here in the US and other places with no competition that they can get away with this crap. So why do you continue to defend it?
[ link to this | view in thread ]
Whatever (profile), 20 Jul 2014 @ 7:05pm

Re: Re: Re: Re: Re: Re: Re: weird
What part of "you twat" do you think adds to the discussion?

Seriously, you call people trolls yet you support name calling and insults. Why?
[ link to this | view in thread ]
This comment has been flagged by the community. Click here to show it

Whatever (profile), 20 Jul 2014 @ 7:08pm

Re: Re: Re: Re: Re: Re: Bull!
Let me rephrase that for you:

"the defendant said "I never saw that woman before I when I hit her it wasn't that hard!"".

Level3 has a vested interest here in showing others at fault, yet their desire to keep everything on their own network and never advertise routes outside their network ends up causing problems as well. There was clearly bandwidth available into Verizon, but Level3 seemed unable or didn't have the desire to route to it. Instead, they steadfastly dumped as much traffic at the already overloaded link as they could.

Taking Level3's word for it isn't exactly compelling evidence, considering they are trying to make others look bad to fix their reputation in this one.
[ link to this | view in thread ]
Ferel (profile), 20 Jul 2014 @ 8:08pm

Re: Re: Re: weird
http://arstechnica.com/business/2014/06/netflix-got-worse-on-verizon-even-after-netflix-agreed-to-pa y-verizon/
[ link to this | view in thread ]
Ferel (profile), 20 Jul 2014 @ 8:12pm

Re: Re: Re: Re: Re: Re: Re: Bull!
So we take Verizon's word as "compelling evidence" instead? /logic
[ link to this | view in thread ]
This comment has been flagged by the community. Click here to show it

Whatever (profile), 20 Jul 2014 @ 9:18pm

Re: Re: Re: Re: Re: Re: Re: Re: Bull!
Not a logical argument. Assuming that Level3 is covering it's ass and trying to paint things it's way doesn't make Verizon any less crooked or cheap. It's not a question of picking sides, it's a question of finding the ultimate solution to a bigger problem.

For what it's worth, the only information on Verizon I am using it that which Level3 has said is correct.

Can you accept that the world isn't black and white and that not believing Level3 doesn't mean that Verizon is suddenly perfect?
[ link to this | view in thread ]
Anonymous Coward, 20 Jul 2014 @ 10:33pm

Re: Re: Re: Re: Re: weird
so do you
[ link to this | view in thread ]
Anonymous Coward, 20 Jul 2014 @ 10:40pm

Re: Re: Re: Re: Re: Re: Re: Re: Re: Bull!
I can accept that you are a contrite moron
[ link to this | view in thread ]
Anonymous Coward, 21 Jul 2014 @ 12:21am

Re: Re: Re: Re: Bull!
Level3 has a vested interest here in showing others at fault, yet their desire to keep everything on their own network and never advertise routes outside their network ends up causing problems as well. There was clearly bandwidth available into Verizon,

Lets Summarize:
Verizon has a vested interest in growing its tier 1 business, and by refusing to increase the ports connected between it's and level 3 routers, in the same building, it is trying to force Netflix into buying tier 1 service from it.
There is nothing wrong with Level 3 keeping traffic from its customers on its own network all the way to the final mile service providers. There is a lot wrong with Verizon trying to force companies to use its tier 1 services rather than that of a competitor by controlling access to the final mile delivery. Level 3 has offered to provide the hardware and engineers to fix the problem, Verizon is refusing because it wants to take over Level 3 customers, and/or increase the costs to competitor to ins content business.
[ link to this | view in thread ]
This comment has been flagged by the community. Click here to show it

Whatever (profile), 21 Jul 2014 @ 2:22am

Re: Re: Re: Re: Re: Bull!
it is trying to force Netflix into buying tier 1 service from it.

I don't see them trying to sell tier 1 transit services at all. They are selling "over the top" network access for a service that uses an incredibly high amount of bandwidth, and mostly at peak times.

I don't see them trying to take over level3 customers, I don't think that is the plan at all.
[ link to this | view in thread ]
Anonymous Coward, 21 Jul 2014 @ 2:46am

Re: Re: Re: Re: Re: Re: Re: Re: Re: Re: Bull!
Not forgetting a TOR-using terrorist, contrarian jackass.
[ link to this | view in thread ]
techflaws (profile), 21 Jul 2014 @ 2:48am

Re: Re: Re: Re: Re: weird
Most people miss that point, and it's pretty key.

No, they don't cause that's not what their plan says. It say flat and Internet. Not at certain times to certain services. And no, the users are not responsible for ISPs constantly overselling what they can't deliver.
[ link to this | view in thread ]
techflaws (profile), 21 Jul 2014 @ 2:52am

Re: Re: Re: Re: Re: Re: Bull!
So, it's you who suddenly defines "over the top". Or Verizon? Yeah, right.
[ link to this | view in thread ]
Anonymous Coward, 21 Jul 2014 @ 3:03am

Re: Re: Re: Re: Re: Re: Bull!
From the embedded letter from Verizon
Netflix has the ability to directly connect to every broadband network in America should it choose to do so.

That is asking Netflix to use their tier 1 services, rather than Level 3's.

P.S. How much is Verizon paying you to lie on their behalf?
[ link to this | view in thread ]
Ninja (profile), 21 Jul 2014 @ 4:48am

Re: Re: Re: Re: Re: Re: I'd say they are defaming Level-3
SO VERIZON MUST UPGRADE THEIR FUCKING INTERNAL NETWORK OR SELL LOWER SPEEDS.

I'd ask if it is that hard to understand but moronic shills will always be moronic shills.
[ link to this | view in thread ]
That One Guy (profile), 21 Jul 2014 @ 5:14am

Minor fix
That is 'asking' Netflix to use their tier 1 services, rather than Level 3's.

'Hey, that's a nice business you got there. I see it requires a nice, steady signal, would be a shame if something were to happen to it at the last mile. Now, if such a hypothetical problem were to occur while you were signed with another company, why, there's nothing we could do about it. On the other hand, if you were doing business with us, instead of that other company, I'm sure a problem like that would be much less likely to crop up...'
[ link to this | view in thread ]
That One Guy (profile), 21 Jul 2014 @ 5:24am

Forgot a few words
It's upgrades to keep up with VERIZON'S business model, which is delivering the lousiest internet service they can get away with to their customers, while charging as much as possible.

Upgrading the network to keep up with current and future demands costs money, and not just once, but multiple times. Companies like money, but don't like spending money, especially if they think they can get away with not doing so.

Now, if there was any real competition, then refusal to upgrade would drive customers to other companies who did upgrade, but since, due to the purchase of laws and politicians, no such competition is available, companies like Verizon feel pretty safe putting off any upgrades until they absolutely have to for whatever reason, safe in the knowledge that it doesn't matter how crappy their service is, their customers have no other choice but to pay them.
[ link to this | view in thread ]
Josh in CharlotteNC (profile), 21 Jul 2014 @ 7:01am

Re: Re: Re: Re: Re: Re: I'd say they are defaming Level-3
Thank you for confirming Verizon is 100% the problem here. A complete opposite of what you've been saying for weeks - kinda hard to keep all those lies straight, isn't it?

Yes, Verizon needs to continue to make upgrades to their own network, as Verizon customers keep using more of the bandwidth that Verizon sells them. Either Verizon makes the upgrades, or it tells the truth to its customers and admits that it's their fault and institutes caps.

I've never once claimed that infrastructure improvements were free - nice try at a strawman. Of course they cost money. But its the normal cost of doing business.

If you're curious about who should pay for it - then go and take a look at rate of return the telcos and cable companies have in the broadband divisions. Not even jewelry and furniture stores can boast the kind of insane markups that they have. Guess that's what abuse of monopoly positions can get.

The sad thing is, Verizon for a few years was actually upgrading their entire infrastructure. While Ivan Seidenberg was CEO, he was pushing FIOS everywhere and making those upgrades. But those poor Wall Street investors didn't like infrastructure improvements and got him forced out.
[ link to this | view in thread ]
nasch (profile), 21 Jul 2014 @ 7:02am

Re: Re: Re: Re: Re: Re: Re: Re: Re: Re: Bull!
I can accept that you are a contrite moron

I do not think that word contrite means what you think it means. I would never apply that to Whatever.
[ link to this | view in thread ]
nasch (profile), 21 Jul 2014 @ 7:04am

Re: Re: Re: Re: Re: Re: Bull!
I don't see them trying to take over level3 customers, I don't think that is the plan at all.

Because you refuse to acknowledge that Verizon is doing anything wrong here.
[ link to this | view in thread ]
nasch (profile), 21 Jul 2014 @ 7:06am

Re: Forgot a few words
That One Guy, I could not agree more.
[ link to this | view in thread ]
BernardoVerda (profile), 21 Jul 2014 @ 6:01pm

Re: Minor fix
<>Precisely why this kind of situation only crops up in those markets where Verizon or some other similar "consumer-oriented" ISP has effective monopoly control over those customers' internet connectivity.
[ link to this | view in thread ]
ancient geek, 22 Jul 2014 @ 8:51am

objective view
A whole lot of biased opinions that are pretty clueless about how things actually work.
[ link to this | view in thread ]
ancient geek, 22 Jul 2014 @ 9:05am

Let's be fair here
You could view "consumer traffic" as a monopoly but it's also a responsibility to deliver service. Bandwidth is not infinite and not free (contrary to several comments about the "bad guy" having plenty of BW to spare --like you have a clue what's involved)

Peering arrangements are critical but very fragile. Having a peering partner suddenly start dumping huge amounts of traffic onto you might be an easy fix at that connection, but it's a huge imposition to distribute it down the line. If traffic is not equal, then somebody is being taken advantage of. Period.

And to the consumers who think they pay for an "X" MB pipe and expect that BW to be delivered to any destination, --that's exactly like claiming your two-lane driveway should guarantee you two unobstructed lanes anywhere you choose to drive.
[ link to this | view in thread ]
nasch (profile), 22 Jul 2014 @ 9:10am

Re: Let's be fair here
Bandwidth is not infinite and not free (contrary to several comments about the "bad guy" having plenty of BW to spare --like you have a clue what's involved)

It's Verizon claiming they have plenty of bandwidth (except for that one saturated link).

If traffic is not equal, then somebody is being taken advantage of.

No, if traffic is not equal, then one party has to pay.
[ link to this | view in thread ]
Anonymous Coward, 22 Jul 2014 @ 1:28pm

Re: Re: Re: Re: Re: weird
"the ISP customers did not pay for an endless fountain of bandwidth at a given speed end to end."

You are being dishonest here. You know the lack of competition in the U.S. results in less service (to customers) at a higher price. You know the customers are being very much overcharged for the service they are receiving. You know that regional monopolists are not entitled to have the government set those monopolies and that in a free market the customers would get much better service for much cheaper. Everything regional monopolists receive above what they would receive in a free market is essentially economic rent and that's equivalent to stealing because they are not entitled to a government established monopoly and so they are not entitled to the monopoly rents they receive when they abuse such monopolies. The government should either require them to charge customers and provide them with what they would be forced to charge and provide in a free market (which what they currently offer is way overpriced) or they should be forced to compete in a free market. Anything other than one of those two options is theft.

Due to a lack of competition customers are overpaying for the service they are receiving. That needs to change.
[ link to this | view in thread ]
Anonymous Coward, 22 Jul 2014 @ 1:48pm

Re: Re: Re: Re: Re: Re: Re: Bull!
Verizon would have to be doing nothing wrong for him to be claiming that they are doing something wrong. After all even those that are wrong deserve representation right?
[ link to this | view in thread ]
Anonymous Coward, 23 Jul 2014 @ 2:00am

Re:
My satisfaction with Cablevision over the last 15 years has been a solid 100%, but then I think our local franchise takes their customers seriously. We get the speeds we pay for pretty much 100% of the time, and outages are rare. Even when they occur, it's usually minutes, not hours or days, for the reroutes to be made while they make repairs.
[ link to this | view in thread ]
Anonymous Coward, 23 Jul 2014 @ 2:06am

Re: Re: Re: Re: Re: weird
Not true. Netflix paid for theirs, but the ISP customers did not pay for an endless fountain of bandwidth at a given speed end to end.

I pay for a set speed that I had better get day in and day out. If you can't give it to me, don't sell it to me.
[ link to this | view in thread ]
Anonymous Coward, 23 Jul 2014 @ 2:12am

Re: Re: Re: Re: Bull!
False. Netflix also uses Cogent for a backup. Please at least know what you're talking about when the lies drip from your lips.
[ link to this | view in thread ]
Anonymous Coward, 23 Jul 2014 @ 7:42am

Re: Re: Re: Re: Re: Re: Re: Re: Re: Bull!
"it's a question of finding the ultimate solution to a bigger problem."

The problem is that Verizon wants to charge too much for access to their regional monopolies. One possible solution is that we need more competition in the marketplace and that governments shouldn't grant regional monopolies. This may not be the solution you or Verizon wants but that doesn't diminish the fact that this is a good solution.

No solution will make everyone happy. You have industry interests that want government to give them an unfair and unearned monopoly on one hand and you have the public interest that benefits from competition on the other. The government should serve only the public interest. Such solutions may not make industry interests that want monopolies and invest in buying and paying for politicians happy but that doesn't change the fact that the government should only serve the public interest. A government that serves the public interest maybe a problem for those wanting laws passed in their favor but it's a solution to the public and the government should only serve the public interest. So when you speak in terms of 'problems' and 'solutions' it is important to understand the interests of the parties involved and the interest that the government is supposed to serve.
[ link to this | view in thread ]
Anonymous Coward, 23 Jul 2014 @ 7:53am

Re: Re: Re: Re: Re: Re: Re: Bull!
"yet their desire to keep everything on their own network and never advertise routes outside their network ends up causing problems as well."

First of all why should they advertise for others. Verizon has its own marketing department and they can do their own advertising.

and, no, the fact that one company doesn't want to advertise for another doesn't cause 'problems'. That's like saying that Toyota refusing to advertise for Honda causes 'problems' for Toyota. What kinda silly argument is this. There is no problem, the only problem here is a fabrication of your imagination.

But, more importantly, the point of net neutrality is that service providers don't have to pay every regional government established monopoly what they demand just to get access to its customer base. In a competitive landscape, where governments don't set monopolies, this wouldn't be as much of an issue, but since local governments do set local monopolies if each service provider had to go through a regional monopoly to deliver its service to those customers it would be prohibitively expensive to start a service since government established monopolists would naturally abuse their monopoly position to maximize profits. This ultimately hurts consumers and so it creates a 'problem' for consumers.

"There was clearly bandwidth available into Verizon, but Level3 seemed unable or didn't have the desire to route to it. Instead, they steadfastly dumped as much traffic at the already overloaded link as they could."

So then you admit that Verizon clearly had available bandwidth to spare but they refused to route Level three traffic through those routes?

Level three's responsibility is only to route traffic to last mile providers and then it is up to those last mile providers to route the bandwidth from there.

If what you mean is that Verizon may offer more bandwidth to Netflix directly through Verizon provided that Netflix paid Verizon for such access then see my points on net neutrality and government established regional monopolies and how that hurts consumers above.

"Instead, they steadfastly dumped as much traffic at the already overloaded link as they could."

So Verizon should simply upgrade the link. It's not expensive.
[ link to this | view in thread ]
Yazz (profile), 21 May 2015 @ 11:50am

Comcast?
"As we've discussed, Verizon, Comcast and AT&T know exactly what they're doing here."
I use comcast, for the past 2 years the cheapest cable modem plan they offer - and have never had any problem streaming to multiple devices at once. Highest quality option available for the video settings.
[ link to this | view in thread ]