Level3 Proves That Verizon Is Absolutely To Blame For Netflix Congestion... Using Verizon's Own Blog Post
from the ouch dept
As you know, despite Netflix having already agreed to pay Verizon's shake down fee to avoid interconnection congestion for its streaming, Netflix and Verizon are fighting a very public fight about who's to "blame" for the network congestion. Netflix took a public shot at Verizon by publicly highlighting Verizon's congested network to users, leading Verizon to send a cease-and-desist letter, claiming it was misleading. As we noted, there's no doubt that it's actually Verizon at fault, because it's not delivering exactly what it sold customers (and we wonder why the FTC is still not getting involved yet). If Verizon's network is getting congested, that's on Verizon to fix, since it sold its users a promise that they could reach anywhere on the internet, including Netflix.Verizon tried to spin the story back in its favor last week, with a blog post about "the congestion myth," in which it claimed that the real problem was how Netflix chose to route its traffic to Verizon. It presented the following nifty chart, claiming that there was no congestion at all on Verizon's network, and saying that it was all about how Netflix was choosing to deliver its traffic to Verizon's network:
One might wonder why Netflix and its transit providers were the only ones that ran into congestion issues. What it boils down to is this: these other transit and content providers took steps to ensure that there was adequate capacity for their traffic to enter our network. In some cases, these are settlement-free peering arrangements, where the relative traffic flows between an IP network provider and us remain roughly equal, and both parties invest in sufficient facilities to match these roughly equal flows. That is the traditional basis for such deals. In other cases there may be traffic imbalances, but the networks or content providers have entered into paid arrangements with us to ensure connections and capacity to meet their needs for their out-of-balance traffic.Some reporters took this at face value, but it never made any sense at all. The chart above pretty clearly shows that the congestion point is actually Verizon's border router. And if it just made a basic upgrade to accept the traffic that it has promised to consumers, there would be no problem at all.
Level3 has now jumped into this debate as well, with even more data showing that Verizon is the real culprit here. Level3 is carrying a bunch of that Netflix traffic, and notes that it has more than enough bandwidth to carry it. It says the only problem is Verizon refusing to take 5 minutes to upgrade its system:
Verizon has confirmed that everything between that router in their network and their subscribers is uncongested – in fact has plenty of capacity sitting there waiting to be used. Above, I confirmed exactly the same thing for the Level 3 network. So in fact, we could fix this congestion in about five minutes simply by connecting up more 10Gbps ports on those routers. Simple. Something we’ve been asking Verizon to do for many, many months, and something other providers regularly do in similar circumstances. But Verizon has refused. So Verizon, not Level 3 or Netflix, causes the congestion. Why is that? Maybe they can’t afford a new port card because they’ve run out – even though these cards are very cheap, just a few thousand dollars for each 10 Gbps card which could support 5,000 streams or more. If that’s the case, we’ll buy one for them. Maybe they can’t afford the small piece of cable between our two ports. If that’s the case, we’ll provide it. Heck, we’ll even install it.Level3 has provided a (not quite as nicely designed) image to zoom in on the border router situation, showing that it has plenty of capacity ready -- all it needs is for Verizon to let it connect more ports:
Looking at this, it once again becomes clear that it's Verizon, AT&T and Comcast that have deliberately caused this congestion, using their positions as dominant players with monopoly control over the last mile to force Netflix to pay them extra. As Level3 notes, it takes two parties to take the "trivial" steps to remove the congestion, and it's Verizon that's the party who isn't cooperating:
All of the networks have ample capacity and congestion only occurs in a small number of locations, locations where networks interconnect with some last mile ISPs like Verizon. The cost of removing that congestion is absolutely trivial. It takes two parties to remove congestion at an interconnect point. I can confirm that Level 3 is not the party refusing to add that capacity. In fact, Level 3 has asked Verizon for a long time to add interconnection capacity and to deliver the traffic its customers are requesting from our customers, but Verizon refuses.As we've discussed, Verizon, Comcast and AT&T know exactly what they're doing here. People hadn't been so concerned with interconnection disputes in the past, because they didn't think the big broadband players would be so crass and so anti-consumer to purposely let interconnection points clog up. But, those three companies have such control over the market at this point that they are able to do that and can effectively shake down internet companies to get them to double pay for the bandwidth that subscribers are already paying for. Today it's Netflix, but soon it's likely to be lots of other companies as well. That's why, in our own comments to the FCC, we noted that the interconnection fights need to be a part of the open internet discussion.
Thank you for reading this Techdirt post. With so many things competing for everyone’s attention these days, we really appreciate you giving us your time. We work hard every day to put quality content out there for our community.
Techdirt is one of the few remaining truly independent media outlets. We do not have a giant corporation behind us, and we rely heavily on our community to support us, in an age when advertisers are increasingly uninterested in sponsoring small, independent sites — especially a site like ours that is unwilling to pull punches in its reporting and analysis.
While other websites have resorted to paywalls, registration requirements, and increasingly annoying/intrusive advertising, we have always kept Techdirt open and available to anyone. But in order to continue doing so, we need your support. We offer a variety of ways for our readers to support us, from direct donations to special subscriptions and cool merchandise — and every little bit helps. Thank you.
–The Techdirt Team
Filed Under: border routers, congestion, interconnection, streaming
Companies: level3, netflix, verizon
Reader Comments
The First Word
“Re: Re: I'd say they are defaming Level-3
In all honesty, directly peering with the content provider is usually preferred no matter who is providing the transit network.Not necessarily. There are situations - including this one - where it makes a lot of sense to work with a single transit provider (or for redundancy 2 or 3) instead of needing to both pay the transit providers AND directly peer with a dozen or more ISPs.
Netflix works with Level3 and Cogent. Both are Tier 1 networks - which means that's all Netflix needs to work with to ensure that their content is reachable by anyone on the internet. So long as they can provide sufficient speed, Netflix wouldn't need to worry about making deals with Verizon, and AT&T, and Comcast, and Time Warner, and Cablevision, and Google Fiber, and Qwest, and so on.
And Verizon, AT&T and Comcast know this. Verizon is also a Tier 1 transit provider. So is AT&T. Comcast is trying to be. They all compete with Level3 and Cogent in the transit business - which is cutthroat and there's good money in it. Here's the thing. Level 3 and Cogent do not compete with those companies in the residential broadband business.
So it comes down to this: Verizon and Comcast can abuse their monopoly position over their residential business in order to force Netflix to directly peer with them, costing Level3 that transit business with Netflix, while losing nothing (since their residential customers have no choice and can't move to another provider).
Subscribe: RSS
View by: Time | Thread
Read the Level3 blog
One striking example is that Level3's network connects to Verizon's in 10 locations. 7 in the US and 3 in Europe. Only in places where Verizon has a significant share of residential broadband customers are the ports at high/full capacity. In Europe, where Verizon has to compete against other ISPs, the ports are not congested.
[ link to this | view in chronology ]
Re: Read the Level3 blog
[ link to this | view in chronology ]
Re: Re: Read the Level3 blog
I just hope the FCC does something about this and makes it very vocal so that all big broadband customers are aware of the problem with isp's and where their customers can demand the problems be resolved or report them to the FCC or even sue them,A simple question, could verizon customers not sue them now that there is clear evidence they are not interested in supplying what they have sold.
[ link to this | view in chronology ]
Re: Read the Level3 blog
[ link to this | view in chronology ]
Re: Read the Level3 blog
Of course they're not. A few thousand bucks for a line card is chump change compared to the cost of trenching fiber, so why would anyone pay the latter and not the former?
Unless of course...they had an effective monopoly and found it useful to create artificial scarcity in order to lie and extort money from third parties.
Having seen the fallout from Verizon's (and Comcast's) network engineers for many years, I have no doubt that their fraudulent claims about bandwidth scarcity would quickly be exposed if an independent team of independent senior, experienced network engineers came in to do an audit of their operation. (Which is of course why that will never happen: at best, they'll hire shills and hand them a prepackaged script.)
[ link to this | view in chronology ]
[ link to this | view in chronology ]
[ link to this | view in chronology ]
Awesome job Level 3, we love you. And a score to Net Neutrality. May Title II deliver it's fiery hell upon the ISPs.
[ link to this | view in chronology ]
I'd say they are defaming Level-3
"Therefore, we are working aggressively with Netflix to establish new, direct connections from Netflix to Verizon’s network. This doesn’t “prioritize” Netflix traffic in any way, but it ensures that their traffic gets on our network through direct connections—not middleman networks—that are up to the task."
So, they are claiming that Level-3 is a 'middleman network' and are not "up to the task" of insuring traffic gets delivered. Yup, if I were Level-3, I'd be a little irritated at that remark.
[ link to this | view in chronology ]
Re: I'd say they are defaming Level-3
[ link to this | view in chronology ]
Re: Re: I'd say they are defaming Level-3
Not necessarily. There are situations - including this one - where it makes a lot of sense to work with a single transit provider (or for redundancy 2 or 3) instead of needing to both pay the transit providers AND directly peer with a dozen or more ISPs.
Netflix works with Level3 and Cogent. Both are Tier 1 networks - which means that's all Netflix needs to work with to ensure that their content is reachable by anyone on the internet. So long as they can provide sufficient speed, Netflix wouldn't need to worry about making deals with Verizon, and AT&T, and Comcast, and Time Warner, and Cablevision, and Google Fiber, and Qwest, and so on.
And Verizon, AT&T and Comcast know this. Verizon is also a Tier 1 transit provider. So is AT&T. Comcast is trying to be. They all compete with Level3 and Cogent in the transit business - which is cutthroat and there's good money in it. Here's the thing. Level 3 and Cogent do not compete with those companies in the residential broadband business.
So it comes down to this: Verizon and Comcast can abuse their monopoly position over their residential business in order to force Netflix to directly peer with them, costing Level3 that transit business with Netflix, while losing nothing (since their residential customers have no choice and can't move to another provider).
[ link to this | view in chronology ]
Re: Re: Re: I'd say they are defaming Level-3
It appears that the single mindedness of Level3 (and certainly some seeming willful blindness from Verizon) means that a single source is overloaded, and other work just fine.
It probably explains the Level 3 attack here: They realize they are part of the problem and not the solution.
As for trying to cut Level 3 out, just remember that Netflix traffic still has to get to the new paid peering point in some manner. It may end up being a "netflix" peering point instead of a Level 3 peering point on a purely technical level, but you can bet because of their agreements that Netflix will use Level3 right to that point anyway, cutting out nobody.
[ link to this | view in chronology ]
Re: Re: Re: Re: I'd say they are defaming Level-3
Some seeing? So you're saying Verizon might not be intentionally neglecting their routing hardware? Why are you such a Verizon cheerleader?
[ link to this | view in chronology ]
Re: Re: Re: Re: I'd say they are defaming Level-3
[ link to this | view in chronology ]
Re: Re: Re: Re: I'd say they are defaming Level-3
The thing is, if Netflix (and Level 3 for that matter) improved their peering with other tier 1 providers,
Read the article. Read Level3's blog. Level3 is willing to buy Verizon the cards, cabling, and install them. I bet they'd even pay a network engineer to update a few routing tables for Verizon. But Verizon won't let them - because if Level3 could fix the problem in a few hours, Verizon would no longer have the threat hanging over Netflix and Level3's heads.
If I was Level3, I would be doing more than just blog posts. I would be throwing fire, making very loud public complaints. I'd be pounding on the FCC's and the FTC's doors. I'd be digging up every penny Verizon has spent on lobbying and every campaign donation.
Single-mindedness of Level3? Completely unarguable fact time: it is 100% Verizon's fault for not upgrading their own infrastructure. Verizon won't spend a few thousand bucks to help hundreds of thousands of their own customers.
Give me one single fact that disputes all of this. Just one tiny piece of information that is based in reality and not whatever delusional universe you live in.
[ link to this | view in chronology ]
Re: Re: Re: Re: Re: I'd say they are defaming Level-3
So let's say they did that. How much more traffic would be acceptable? How much can Verizon accept before their own internal network becomes overloaded?
The Level3 "free" solution that you point to isn't free at all. It's just bait for a long series of expensive network upgrades to try to keep up with Netflix business model. It's never ending, capitulating with them today pretty much creates a situation where Verizon would be required to upgrade their network to support anyone who showed up with a bandwidth intensive business model.
Verizon won't spend a few thousand bucks to help hundreds of thousands of their own customers.
If it was only a few thousand bucks. You only have to look at the chart to understand. Level3 appeared to want to double the bandwidth coming in. That would have been enough to overload other points in Verizon's network. More upgrades. Level3 comes back to add MORE connections (because Netflix is so good and cheap!) and Verizon has to upgrade more stuff. Verizon can't win.
The money they are getting in this deal is what will end up paying for those network upgrades in the long run, and at the same time sets a hard cap on what comes in - if they want more, they can pay for it.
[ link to this | view in chronology ]
Re: Re: Re: Re: Re: Re: I'd say they are defaming Level-3
It's upgrades to keep up with VERIZON'S business model, which is delivering internet service to their customers. In places with competition, ISPs have no trouble doing these upgrades. It's only here in the US and other places with no competition that they can get away with this crap. So why do you continue to defend it?
[ link to this | view in chronology ]
Forgot a few words
Upgrading the network to keep up with current and future demands costs money, and not just once, but multiple times. Companies like money, but don't like spending money, especially if they think they can get away with not doing so.
Now, if there was any real competition, then refusal to upgrade would drive customers to other companies who did upgrade, but since, due to the purchase of laws and politicians, no such competition is available, companies like Verizon feel pretty safe putting off any upgrades until they absolutely have to for whatever reason, safe in the knowledge that it doesn't matter how crappy their service is, their customers have no other choice but to pay them.
[ link to this | view in chronology ]
Re: Forgot a few words
[ link to this | view in chronology ]
Re: Re: Re: Re: Re: Re: I'd say they are defaming Level-3
I'd ask if it is that hard to understand but moronic shills will always be moronic shills.
[ link to this | view in chronology ]
Re: Re: Re: Re: Re: Re: I'd say they are defaming Level-3
Yes, Verizon needs to continue to make upgrades to their own network, as Verizon customers keep using more of the bandwidth that Verizon sells them. Either Verizon makes the upgrades, or it tells the truth to its customers and admits that it's their fault and institutes caps.
I've never once claimed that infrastructure improvements were free - nice try at a strawman. Of course they cost money. But its the normal cost of doing business.
If you're curious about who should pay for it - then go and take a look at rate of return the telcos and cable companies have in the broadband divisions. Not even jewelry and furniture stores can boast the kind of insane markups that they have. Guess that's what abuse of monopoly positions can get.
The sad thing is, Verizon for a few years was actually upgrading their entire infrastructure. While Ivan Seidenberg was CEO, he was pushing FIOS everywhere and making those upgrades. But those poor Wall Street investors didn't like infrastructure improvements and got him forced out.
[ link to this | view in chronology ]
That being said, Verizon sucks. They ran an ad in my area for FIOs that said that Verizon customer rankings are 3 times better than Cablevisions, they actually listed the number. 3 times Cablevision rankings put them at 40%, so even though they were better than Cablevision, 60% of Verizon customers thought they sucked.
[ link to this | view in chronology ]
Re:
[ link to this | view in chronology ]
Re: Re:
[ link to this | view in chronology ]
Re:
[ link to this | view in chronology ]
[ link to this | view in chronology ]
[ link to this | view in chronology ]
Contrast
The plan by Verizon is to extort ongoing payments forever for failing to spend the one time money.
They go to the banks with their RMR (Recurring Monthly Revenue) records in hand, so better jack up that RMR.
[ link to this | view in chronology ]
Open a BBB Complaint
[ link to this | view in chronology ]
Re: Open a BBB Complaint
[ link to this | view in chronology ]
Re: Re: Open a BBB Complaint
[ link to this | view in chronology ]
Re: Open a BBB Complaint
[ link to this | view in chronology ]
-Level3 VP of Content and Media David Young
I think you hit the nail on the head Mr. Young. Verizon doesn't want Netflix competing against their U-Verse TV packages. It's the same thing with Comcast.
Then we have Aereo. Which the cable and TV networks have declared year round duck season on.
Internet TV is getting assaulted from all directions by residential monopoly ISPs. The FCC is run by a lobbyist. We also have members of Congress such as Marsha Blackburn, attempting to outlaw competition against AT&T, Comcast, and Verizon.
Customers are getting hit from all sides and in every direction! Free market my ass. America is rigged towards monopolies and blocking competition through crony capitalist laws.
[ link to this | view in chronology ]
Breaking News
[ link to this | view in chronology ]
Re: Breaking News
[ link to this | view in chronology ]
I see something similar firsthand
I track my ping times from my router to my first hop throughout the day, and during peak times, the latency and bandwidth suffer immensely. This is certainly not my ISP's fault - it is in fact AT&T's fault for not having enough bandwidth available between my house and sonic.net's data center to service the virtual circuit that they've provisioned. Every time I call sonic.net to report severe latency issues, they are forced to report the problem to AT&T to fix the circuit and infrastructure behind it.
This gets really irritating during peak hours, because EVERY connection is slow for me - whereas AT&T subscribers may at least get decent performance for some websites with better peering than my circuit does for its first hop.
Either way, I still refuse to use AT&T as my ISP, even if they own the circuit, as I have better privacy and support with sonic.net than I would ever have with AT&T, and no monthly caps.
[ link to this | view in chronology ]
Re: I see something similar firsthand
Therefore I still have the same copper, the same CO, the same everything as I had before.
However now I have about half the performance that I had previously and of course technical support is miserably incompetent, able only to read from a script and give worthless inapplicable suggestions. (Or try to get me to upgrade to FIOS, which is not available in my area and WILL NEVER BE AVAILABLE IN MY AREA because of course Verizon has no intention of spending the money to lay fiber out here when it's much more profitable to sell to infill customers in more densely populated areas.)
I'm just out of range of a wireless ISP and have begun resorting to blatant bribery (e.g., a good bottle of bourbon) in an attempt to persuade the owner to please please pllleeeease add a tower to extend their footprint to one that covers me.
[ link to this | view in chronology ]
Re: Re: I see something similar firsthand
[ link to this | view in chronology ]
Re: Re: Re: I see something similar firsthand
Check here: WispA map
[ link to this | view in chronology ]
Re: Re: Re: Re: I see something similar firsthand
[ link to this | view in chronology ]
Re: Re: Re: Re: Re: I see something similar firsthand
http://www.smarterbroadband.com/Pricing.htm
[ link to this | view in chronology ]
Re: Re: Re: Re: Re: Re: I see something similar firsthand
Hopefully, either of them can hit you. Perhaps if your out in the sticks enough on a hill offer to allow a rohn tower to extend their footprint. I used to do wireless until it really wasn't feasible to be in the ISP business.
[ link to this | view in chronology ]
Re: Re: I see something similar firsthand
[ link to this | view in chronology ]
[ link to this | view in chronology ]
Re:
http://blog.level3.com/
[ link to this | view in chronology ]
Re: Re:
[ link to this | view in chronology ]
Re: Re:
However, it simply seems to be back up now. That was...brief.
[ link to this | view in chronology ]
Re:
[ link to this | view in chronology ]
Hope?
But here's the problem: they actually deliver 5mbps. So the stats that say it shouldn't be enough are wrong. I've had all sorts of internet connections in the past few years, and its never been better.
So yeah, internet providers customarily lie about their bandwidth. And I'm glad its becoming an issue.
[ link to this | view in chronology ]
Re: Hope?
God I wish I could consistently get 5mbps from my provider. Then again, I'd pay $70/mo for the gbps, since I already pay $70 for up to 70mbps (which is consistently around 3mpbs on a good day.)
[ link to this | view in chronology ]
Re: Re: Hope?
what speed does techdirt come down at? do they need a better connection or better peering to deliver ?
[ link to this | view in chronology ]
Re: Re: Re: Hope?
I'm not, nor is anyone else here that I'm aware of, saying that the whole internet runs at 70Mbps. I'd love to get a 70 meg connection from my house to the edge of my cable internet provider's network. And maybe something close to that going from their network to their peers and transits.
After all, the internet is wireless and runs in a little black box with a red light on it...I saw it on TV once when Roy and Moss showed it to Jen and she proceeded to break it. Do me a favor...don't ever break the internet.
what speed does techdirt come down at? do they need a better connection or better peering to deliver ?
My co-loc provider provides a minimum of 100 Mbps, I get at least 1 Gbps on my servers, and I know he is connected to Level 3 at 10 Gbps or more. My work has several connections that run in the 600 Mbps to 10 Gbps range, and I know that they have bound 10 Gbps connections. My mirroring sessions at work routinely sustain 20 - 60 Mbps thoroughput across the country. And I suspect Floor64, being so close to Silicon Valley probably has the same connectivity at their co-loc as well.
On the real internet, not the AOL bubble version Comcast/Verizon/AT&T tend to believe they live on, bandwidth is cheap and getting far cheaper. My cloud networks measure bandwidth at MB at about 2.4 hundreths of a penny each, and my co-loc doesn't even charge me for bandwidth because the cost is so lo that he can recover it through my monthly connection fees.
This is little more than tin-pot sheriffs setting up their tollbooths, attempting to rival the government on collecting taxes for nothing, just like they did with the billions we spent on upgrading the broadband infrastructure for "the information superhighway" and E911 and K-12 and FIOS and every other boondoggle they've asked for money, changed the goal posts midstream, and then asked the government to forget the requirements and then pocket the money for the services they didn't provide.
[ link to this | view in chronology ]
Does this fall under extortion?
If we had a functioning justice system, they'd be throwing the books at Verizon and the other ISP's trying to do this crap.
[ link to this | view in chronology ]
funny
that THOSE companies dont HAVE facilities to UPGRADE their OWN SYSTEMS..
I will BEt they have cut the number of Inter-connects to the MINIMUM required...and not ONLY do they need an inter-connect, they need a WHOLE SERVER setup...
when the internet took off, in Portland Or, it took the Companies 6 MONTHS to redo ALL the MAIN connections to Fiber. they were satisfied with a 6% Copper wired service..It hardly ever HIT the Over load..
But after Cellphones and Internet took over, the amount of bandwidth needed went upto 80% in 1 year.
What this means is that for ALL the people in the Metro area, only 6% of those people and business NEEDED access tot he net or other services.. So they didnt NEED Extra PEOPLE, SERVICES and the traffic was able to be handled with 6% coverage..
After the NEt and CELLPHONEs took over 80% of the people and business needed Some sort of access to communications..
And that was in the LATE 90's..
[ link to this | view in chronology ]
Re: funny
[ link to this | view in chronology ]
[ link to this | view in chronology ]
and there's more
[ link to this | view in chronology ]
---- I mean this is it right? End of discussion.
[ link to this | view in chronology ]
Re:
[ link to this | view in chronology ]
Well
Where I currently live, we had to get Verizon DSL, and were "promised" up to 3mbps (Didn't care, asked for the minimum to stream Netflix for the kids). Not once in the past year has it peaked above 1.5mbps.
We are moving in a few weeks, and the only available internet is FiOS. We were "promised" up to 50mbps. We shall see. If it breaks 10mbps, I'll be impressed.
I have a question:
If Verizon is saying (as is depicted in the diagram) that they are only utilizing "48% peak utilization" from the terminal to wires to my house, shouldn't netflix stream at the promised rate seeing how they have room for the bandwidth, but aren't using it? From what I've experienced, Netflix still lags.
The diagram looks like the problem is all Netflix to Verizon, but Verizon to my house has no hiccups (ha). As with a bottle neck, once past the blockage shouldn't the speed pick back up? If you're driving down the freeway and come across an accident, everyone slows down. Once past the accident, it's open sailing for a while. Isn't this the same concept? Or is the diagram misleading, and the red line should extend all the way from Netflix to my house?
And if it does, isn't that the point of buffering? To catch it up so it doesn't trickle down to me?
[ link to this | view in chronology ]
Re: Well
Buffering (if in play) can help mitigate some of the variations in throughput, but it can't compensate for a fundamental lack of end-to-end speed.
[ link to this | view in chronology ]
watching proof of the slowdown
https://www.youtube.com/watch?v=5vs3QhEx_3w
he vpn's in (which, incidentally encrypts his network packets from the netflix packet sniffer verizon is running) and gets a giant burst of speed.
who knew? vpns are FASTER!
[ link to this | view in chronology ]
Re: watching proof of the slowdown
Supposedly, routing is supposed to choose the most efficient routes, but if there aren't enough statistics for routers to make routing decisions (such as the congestion level upstream), they may make poor decisions. By connecting to a VPN, you're basically forcing the route to go through another 3rd party server, so this can indeed avoid the largely congested areas, and thus his bandwidth becomes better. In theory, his latency might actually be worse, but for streaming video, that seldom matters much.
[ link to this | view in chronology ]
Sure, he'd probably get fired, but imagine the PR dance Verizon would have to do to explain how all the congestion magically disappeared.
[ link to this | view in chronology ]
Re:
This is clearly Verizons fault, though I have a question for anyone good at maths...
Given that Verizons capacity at its own weakest link is at 65%, and that Netflix accounts for over 1/3rd the US traffic.. What impact would full speed Netflix have on Verizons network?
Its a tricky question given that:
• We don't know the ratio of Verizons traffic once on their network that is Netflix
• Verizon's contribution to the total US traffic
• The number of new subscribers if Netflix had a good connection on Verizon
• The increase in total traffic on Verizons network
Would like to read theories on the above?
[ link to this | view in chronology ]
[ link to this | view in chronology ]
Re:
[ link to this | view in chronology ]
Re: Re:
What really needs to happen is Verizon's customers call and complain that Verizon is not providing adequate peering to a network the size of Level3. Maybe if enough knowledgeable customers complain they'll be forced to actually address te issue.
[ link to this | view in chronology ]
weird
Also, it is really fair to oblige the ISPs to upgrade their connections to Level3 (at a price, natch) to provide more bandwidth for Netflix to make money? The diagrams clearly show that Netflix demands and requires a substantial amount of bandwidth above and beyond reasonable use.
Yes, Verizon could "fix" the problem, by giving in and spending more money to support Netflix business model, and to support their decision to use a single carrier for all of their bandwidth.
The Verizon graphic clearly shows that Netflix is a bandwidth hog and their business model and profit is based on it. Is it really fair to ask the ISPs to spend more to support their business models?
[ link to this | view in chronology ]
Re: weird
Verizon is intentionally not upgrading their service, instead letting it congest, punishing the customers who've already paid for that service, in order to shake down Netflix for 'preferential' treatment on the network, and since upgrading the connections would remove the need for that 'preferential treatment', you can bet that none of the money they are shaking Netflix down for will be used to upgrade the currently congested systems.
Verizon, and likely the others pulling the same stunt, could 'solve' the problem without difficulty, and for cheap at that, and instead they're letting it continue and grow, all so they can shake down Netflix, and get them to pay, again, for content that's already been paid for.
[ link to this | view in chronology ]
Re: weird
http://arstechnica.com/tech-policy/2014/04/netflix-and-verizon-reach-interconnection-deal-to-sp eed-up-video/
[ link to this | view in chronology ]
Re: Re: weird
I'll let you call yourself names now.
[ link to this | view in chronology ]
Re: Re: Re: weird
[ link to this | view in chronology ]
Re: weird
http://blog.level3.com/global-connectivity/verizons-accidental-mea-culpa/
If you can't fucking tell, you've got shit on your face.
[ link to this | view in chronology ]
Re: Re: weird
I AGREE WITH YOU!
However, Netflix is paying for a reason, because their network demands are over and above what is reasonable for any peering arrangement. Yes, verizon could have just kept adding bandwidth to satisfy the Netflix business model, but where do you think the money comes from to do it? Those interconnects are not free to set up, sad to say.
Level3 generally charges for connections to their network - perhaps they want Verizon to spend a whole lot more money with them?
[ link to this | view in chronology ]
Re: Re: Re: weird
It's not. Netflix already paid their ISP and the customers paid theirs. And as you can see in the diagram, there's plenty of bandwith to spare - for the costs already paid.
[ link to this | view in chronology ]
Re: Re: Re: Re: weird
Not true. Netflix paid for theirs, but the ISP customers did not pay for an endless fountain of bandwidth at a given speed end to end.
See, to me, this is where the problem lies. Your 5meg connection doesn't mean your connection to everything on the net will flow at 5 meg per second. It's a peak speed, it's the speed that you are connected to the head end of you ISP - but is no assurance of end to end performance to the entire internet.
Most people miss that point, and it's pretty key. Your connection speed does not mean that you get that bandwidth x 60 seconds x 60 minutes x 24 hours per day x 30.4 days a month (average month length). It's fastest that things will be delivered - the rest may be slower, especially in times of peak demand.
The graphic says it all: Netflix takes a link that would be about 40-50% (on par with the other peering) and wipes it out with heavy, constant, long term usage - 60% or so of the total connection. That's pretty out there.
If Netflix learned how to use more that one provider for their bandwidth, they would likely be able to do a lot more... why don't they have more than one arrangement?
[ link to this | view in chronology ]
Re: Re: Re: Re: Re: weird
The graphic says it all: Verizon takes a link that would be about 40-50% (on par with the other peering) and wipes it out by refusing to do ordinary industry standard upgrades on it.
FTFY.
The point is not that Verizon customers are expecting their theoretical top speed all the time, the point is that they're expecting (or perhaps "dreaming of" is a better term) Verizon to make a best effort to deliver whatever traffic the customer demands. Instead, they are intentionally screwing up the connection. I don't understand why you're defending this, is it just to disagree with TechDirt?
If Netflix learned how to use more that one provider for their bandwidth, they would likely be able to do a lot more... why don't they have more than one arrangement?
So rather than call on Verizon to just do what they should have been doing all along, you want Netflix to alter their strategy to accommodate the extortionist.
[ link to this | view in chronology ]
Re: Re: Re: Re: Re: weird
No. You are just taking the piss-poor business model of consumer ISPs as default and proper. They never provides the speeds they advertise, with their "up to" caveat (already ridiculous), and the same sort of stupid "overbooking" ploy as airlines, where that practice, while poor, actually has some merit.
This is easily seen by the fact that they have existing rackspace with empty router card slots, and existing dark fiber. They paid to lay fiber which they basically are not using because they want to extort more money from everyone. Everyone who already overpays for substandard service.
Unlike 10 years ago, when you could possibly make an argument that people paying to look at a website once a day or send a couple emails were paying the same amount as heavy downloaders, P@P users, and streaming fans, the whole internet has a lot of high throughput applications these days, and is built for it, including Verizon's co-los and last mile networks. They are just playing games with it.
[ link to this | view in chronology ]
Re: Re: Re: Re: Re: weird
No, they don't cause that's not what their plan says. It say flat and Internet. Not at certain times to certain services. And no, the users are not responsible for ISPs constantly overselling what they can't deliver.
[ link to this | view in chronology ]
Re: Re: Re: Re: Re: weird
You are being dishonest here. You know the lack of competition in the U.S. results in less service (to customers) at a higher price. You know the customers are being very much overcharged for the service they are receiving. You know that regional monopolists are not entitled to have the government set those monopolies and that in a free market the customers would get much better service for much cheaper. Everything regional monopolists receive above what they would receive in a free market is essentially economic rent and that's equivalent to stealing because they are not entitled to a government established monopoly and so they are not entitled to the monopoly rents they receive when they abuse such monopolies. The government should either require them to charge customers and provide them with what they would be forced to charge and provide in a free market (which what they currently offer is way overpriced) or they should be forced to compete in a free market. Anything other than one of those two options is theft.
Due to a lack of competition customers are overpaying for the service they are receiving. That needs to change.
[ link to this | view in chronology ]
Re: Re: Re: Re: Re: weird
I pay for a set speed that I had better get day in and day out. If you can't give it to me, don't sell it to me.
[ link to this | view in chronology ]
Re: Re: Re: weird
The.
Fucking.
Article.
You.
Twat.
[ link to this | view in chronology ]
Re: Re: Re: Re: weird
homs.
suck.
[ link to this | view in chronology ]
Re: Re: Re: Re: Re: weird
[ link to this | view in chronology ]
Re: Re: Re: Re: Re: Re: weird
Is ithis deliberate sophistry? Or does that kind of confused thinking explain their stubborn adherence to those unsupportable opinions in the first place?
[ link to this | view in chronology ]
Re: Re: Re: Re: Re: Re: Re: weird
Seriously, you call people trolls yet you support name calling and insults. Why?
[ link to this | view in chronology ]
Re: Re: Re: Re: Re: weird
[ link to this | view in chronology ]
Re: weird
[ link to this | view in chronology ]
Re: weird
So Verizon need to update their network. Your point?
[ link to this | view in chronology ]
Re: weird
Except that isn't true.
But you keep shining on you crazy diamond.
[ link to this | view in chronology ]
Re: weird
Did you read the part that states the upgrades are very cheap, and Level3 offered to pay for them?
The Verizon graphic clearly shows that Netflix is a bandwidth hog and their business model and profit is based on it.
What it shows is that Verizon isn't interested in giving their customers what they've promised them.
[ link to this | view in chronology ]
To use a shipping company analogy
[ link to this | view in chronology ]
Re: To use a shipping company analogy
I suppose I'm being a bit lazy in that analogy on who pays and who the customer is. In this analogy, the shipper is the customer and the shipper pays. Asking the receiver to pay, in addition to the shipper, would be pure rent seeking.
The fact is, I pay my ISP to connect me to the internet. I expect that (1) I get connected to the entire internet, and (2) the ISP reasonably peers with other networks in order to get good speeds to the other networks that I request data from. If ISPs were truly customer-centered, this is a no brainer. The customers want Netflix, so any ISP worth anything is going to make sure there is adequate bandwidth to Netflix.
[ link to this | view in chronology ]
Re: Re: To use a shipping company analogy
[ link to this | view in chronology ]
Because they connect to domestic users, and provide asymmetrical connection speeds, and ban servers, Verizon will always have a traffic imbalance with backbone providers like level 3, and content providers like Netflix, its the nature of the business they are in. Therefore they will always have an excuse to charge for peering arrangements, and double dip for the delivery bandwidth they promised their customers. That they know this is obvious because their touted speeds download to the customer, and you have to look hard to find what uploads speeds they are offering.
[ link to this | view in chronology ]
Bull!
[ link to this | view in chronology ]
Re: Bull!
[ link to this | view in chronology ]
Re: Re: Bull!
Which is why Whatever needs to pull his face out of Verizon's ass long enough to RTFA.
[ link to this | view in chronology ]
Re: Re: Re: Bull!
I read the article carefully. It's carefully crafted by Netflix and Level3 to try to make Verizon look like the guilty party, when in fact it's equally clear that they aren't playing well with others and are not allowing their stuff to peer through any other provider - likely to keep their own costs down.
Verizon has more peering running 40% or so. If Level3 peered with those companies and announced their routes through them, then at least some of the traffic could be routed that way. In fact, it would be fairly simply for Netflix in their software to poll for multiple routes and select the best one - instead of resolutely trying to jam their stuff down a full pipe.
Why don't they do it? I suspect that Level3 isn't playing well with others, and is trying to avoid sending traffic through other providers, a true bottom line choice. Basically, they only route internal, unless it's absolutely unavoidable.
Opening up their routing, and the Verizon problem would have disappeared.
Try not to just FTFA, try actually thinking a moment.
[ link to this | view in chronology ]
Re: Re: Re: Re: Bull!
So what? Why should they have to peer through other providers? You keep saying this as though it's Netflix's duty, but if they get the best deal from Level3, what's the problem with that? If Verizon didn't intentionally screw them, nobody would even notice.
Why don't they do it? I suspect that Level3 isn't playing well with others, and is trying to avoid sending traffic through other providers, a true bottom line choice. Basically, they only route internal, unless it's absolutely unavoidable.
Opening up their routing, and the Verizon problem would have disappeared.
Got a citation for that, or are you making stuff up?
[ link to this | view in chronology ]
Re: Re: Re: Re: Re: Bull!
[ link to this | view in chronology ]
Re: Re: Re: Re: Re: Re: Bull!
"the defendant said "I never saw that woman before I when I hit her it wasn't that hard!"".
Level3 has a vested interest here in showing others at fault, yet their desire to keep everything on their own network and never advertise routes outside their network ends up causing problems as well. There was clearly bandwidth available into Verizon, but Level3 seemed unable or didn't have the desire to route to it. Instead, they steadfastly dumped as much traffic at the already overloaded link as they could.
Taking Level3's word for it isn't exactly compelling evidence, considering they are trying to make others look bad to fix their reputation in this one.
[ link to this | view in chronology ]
Re: Re: Re: Re: Re: Re: Re: Bull!
[ link to this | view in chronology ]
Re: Re: Re: Re: Re: Re: Re: Re: Bull!
For what it's worth, the only information on Verizon I am using it that which Level3 has said is correct.
Can you accept that the world isn't black and white and that not believing Level3 doesn't mean that Verizon is suddenly perfect?
[ link to this | view in chronology ]
Re: Re: Re: Re: Re: Re: Re: Re: Re: Bull!
[ link to this | view in chronology ]
Re: Re: Re: Re: Re: Re: Re: Re: Re: Re: Bull!
[ link to this | view in chronology ]
Re: Re: Re: Re: Re: Re: Re: Re: Re: Re: Bull!
I do not think that word contrite means what you think it means. I would never apply that to Whatever.
[ link to this | view in chronology ]
Re: Re: Re: Re: Re: Re: Re: Re: Re: Bull!
The problem is that Verizon wants to charge too much for access to their regional monopolies. One possible solution is that we need more competition in the marketplace and that governments shouldn't grant regional monopolies. This may not be the solution you or Verizon wants but that doesn't diminish the fact that this is a good solution.
No solution will make everyone happy. You have industry interests that want government to give them an unfair and unearned monopoly on one hand and you have the public interest that benefits from competition on the other. The government should serve only the public interest. Such solutions may not make industry interests that want monopolies and invest in buying and paying for politicians happy but that doesn't change the fact that the government should only serve the public interest. A government that serves the public interest maybe a problem for those wanting laws passed in their favor but it's a solution to the public and the government should only serve the public interest. So when you speak in terms of 'problems' and 'solutions' it is important to understand the interests of the parties involved and the interest that the government is supposed to serve.
[ link to this | view in chronology ]
Re: Re: Re: Re: Re: Re: Re: Bull!
First of all why should they advertise for others. Verizon has its own marketing department and they can do their own advertising.
and, no, the fact that one company doesn't want to advertise for another doesn't cause 'problems'. That's like saying that Toyota refusing to advertise for Honda causes 'problems' for Toyota. What kinda silly argument is this. There is no problem, the only problem here is a fabrication of your imagination.
But, more importantly, the point of net neutrality is that service providers don't have to pay every regional government established monopoly what they demand just to get access to its customer base. In a competitive landscape, where governments don't set monopolies, this wouldn't be as much of an issue, but since local governments do set local monopolies if each service provider had to go through a regional monopoly to deliver its service to those customers it would be prohibitively expensive to start a service since government established monopolists would naturally abuse their monopoly position to maximize profits. This ultimately hurts consumers and so it creates a 'problem' for consumers.
"There was clearly bandwidth available into Verizon, but Level3 seemed unable or didn't have the desire to route to it. Instead, they steadfastly dumped as much traffic at the already overloaded link as they could."
So then you admit that Verizon clearly had available bandwidth to spare but they refused to route Level three traffic through those routes?
Level three's responsibility is only to route traffic to last mile providers and then it is up to those last mile providers to route the bandwidth from there.
If what you mean is that Verizon may offer more bandwidth to Netflix directly through Verizon provided that Netflix paid Verizon for such access then see my points on net neutrality and government established regional monopolies and how that hurts consumers above.
"Instead, they steadfastly dumped as much traffic at the already overloaded link as they could."
So Verizon should simply upgrade the link. It's not expensive.
[ link to this | view in chronology ]
Re: Re: Re: Re: Bull!
Lets Summarize:
Verizon has a vested interest in growing its tier 1 business, and by refusing to increase the ports connected between it's and level 3 routers, in the same building, it is trying to force Netflix into buying tier 1 service from it.
There is nothing wrong with Level 3 keeping traffic from its customers on its own network all the way to the final mile service providers. There is a lot wrong with Verizon trying to force companies to use its tier 1 services rather than that of a competitor by controlling access to the final mile delivery. Level 3 has offered to provide the hardware and engineers to fix the problem, Verizon is refusing because it wants to take over Level 3 customers, and/or increase the costs to competitor to ins content business.
[ link to this | view in chronology ]
Re: Re: Re: Re: Re: Bull!
I don't see them trying to sell tier 1 transit services at all. They are selling "over the top" network access for a service that uses an incredibly high amount of bandwidth, and mostly at peak times.
I don't see them trying to take over level3 customers, I don't think that is the plan at all.
[ link to this | view in chronology ]
Re: Re: Re: Re: Re: Re: Bull!
[ link to this | view in chronology ]
Re: Re: Re: Re: Re: Re: Bull!
That is asking Netflix to use their tier 1 services, rather than Level 3's.
P.S. How much is Verizon paying you to lie on their behalf?
[ link to this | view in chronology ]
Minor fix
'Hey, that's a nice business you got there. I see it requires a nice, steady signal, would be a shame if something were to happen to it at the last mile. Now, if such a hypothetical problem were to occur while you were signed with another company, why, there's nothing we could do about it. On the other hand, if you were doing business with us, instead of that other company, I'm sure a problem like that would be much less likely to crop up...'
[ link to this | view in chronology ]
Re: Minor fix
[ link to this | view in chronology ]
Re: Re: Re: Re: Re: Re: Bull!
Because you refuse to acknowledge that Verizon is doing anything wrong here.
[ link to this | view in chronology ]
Re: Re: Re: Re: Re: Re: Re: Bull!
[ link to this | view in chronology ]
Re: Re: Re: Re: Bull!
[ link to this | view in chronology ]
[ link to this | view in chronology ]
I already said it was Verizons fault!
[ link to this | view in chronology ]
objective view
[ link to this | view in chronology ]
Let's be fair here
Peering arrangements are critical but very fragile. Having a peering partner suddenly start dumping huge amounts of traffic onto you might be an easy fix at that connection, but it's a huge imposition to distribute it down the line. If traffic is not equal, then somebody is being taken advantage of. Period.
And to the consumers who think they pay for an "X" MB pipe and expect that BW to be delivered to any destination, --that's exactly like claiming your two-lane driveway should guarantee you two unobstructed lanes anywhere you choose to drive.
[ link to this | view in chronology ]
Re: Let's be fair here
It's Verizon claiming they have plenty of bandwidth (except for that one saturated link).
If traffic is not equal, then somebody is being taken advantage of.
No, if traffic is not equal, then one party has to pay.
[ link to this | view in chronology ]
Comcast?
I use comcast, for the past 2 years the cheapest cable modem plan they offer - and have never had any problem streaming to multiple devices at once. Highest quality option available for the video settings.
[ link to this | view in chronology ]