Detailed Report Shows How ISPs Are Making 'Business Choice' To Make Your Internet Connection Terrible
from the consumers-lose dept
A couple of years ago, we wrote about an effort by the big broadband players to push the FCC away from using M-Lab to measure basic network diagnostics on the internet. M-Lab is a very interesting project, focused on collecting a huge amount of data about internet performance, and making that data widely available. In the past, for example, we've highlighted an M-Lab project showing which ISPs were throttling BitTorrent.Now, M-Lab has released a new report, along with all of the data and a very nice tool to analyze it all, called the Internet Observatory, that looks at ISP interconnection and, most importantly, its impact on consumer internet performance. The end result? You guessed it. M-Lab found that the various ISPs have plenty of capacity, but it appears that they make the business decision to let your internet get clogged. No wonder the big telcos wanted the FCC to move away from supporting M-Lab!
The idea that broadband players would leverage interconnection bottlenecks for a business advantage (at the expense of subscriber connections) isn't a new idea. But now there's (lots and lots of) data to confirm that.
The report looks at New York City, where M-Lab was able to collect a ton of data from a variety of different vantage points. What it found was that starting in the spring of 2013, customers on Comcast, Time Warner Cable and Verizon basically choked off content coming from Cogent (one of the major transit providers) every day from the early afternoon until late at night. Those big ISPs would like to try to blame the transit providers for the clogging, but M-Lab's data shows that's not the case. That's because a fourth ISP in the area, Cablevision, showed no similar degradation at all -- showing that Cogent appeared to have plenty of bandwidth available. Rather, as we've suggested in the past, the big broadband players were purposely letting the connection between Cogent (and other transit ISPs) wither on the vine, rather than opening up new ports (a trivially simple process).
Similarly, M-Lab's data shows that the problem wasn't a lack of bandwidth on the part of the ISPs (i.e., no actual technical congestion), because those same ISPs had no problem connecting to a different transit provider, Internap. So the only logical culprit was the interconnection points. There was more than enough bandwidth to go around. There's just the single bottleneck of the interconnection border router (which, again, is trivially simple to get rid of by opening up more ports). Here are the charts showing all of this. First up, you can see the rather notable sluggishness found between Comcast, Time Warner Cable and Verizon on downloading content via Cogent:
And the impact of this business decision by those ISPs to let the border router get clogged is massive for end users. Basically, everyone's broadband connections slowed down to a level no sane person would describe as broadband.
While daily median download throughput overall hovered around 4 Mbps, performance degradation was much worse during peak use hours. For much of the time between Spring 2013 and March 2014, download speeds during peak use hours remained well below 4 Mbps. By January 2014, the download throughput rate during peak use hours for Comcast and Verizon traffic over Cogent’s network was less than 0.5 Mbps, the minimum rate necessary for web browsing and email according to the FCC (Figure 4). Note that only between 2:00 AM and 1:00 PM were the three affected Access ISPs (Comcast, Time Warner Cable, and Verizon) able to attain speeds above 4 Mbps across the Transit ISP Cogent. During peak use, only Cablevision achieved sustained rates above 4 Mbps across Cogent, while all Access ISPs achieved sustained rates above 4 Mbps across Internap throughout the day.
The M-Lab report shows nearly identical situations with Cogent and broadband access providers in other areas like Dallas, LA and Seattle. While the issue with Cogent is the most obvious (perhaps because Netflix is using it and that was the target of the clogging), the report also found those same broadband providers clogging the other major transit provider, Level 3, though not quite as much as with Cogent. As the report notes, there are indications that the degradation of Level 3 was also for business reasons, not any technical reasons. The degradation with Level 3 happened in a nearly simultaneous manner across many different sites "irrespective of their relative baselines." In other words, even when there was plenty of capacity on all sides, it didn't matter. Verizon decided to start clogging Level 3, and boom. Suddenly performance drops. The report found similar "simultaneous" downgrades in download speeds with another large transit provider, XO as well.
The report's conclusions lay things out pretty clearly:
In conducting this research we find a consistent theme across multiple Access ISPs and Transit ISPs: the interconnection relationships between network operators matter. The quality of service that millions of consumers experience on a daily basis for periods of multiple months can be directly tied to these relationships. Further, these relationships are not simply technical. We see the same patterns of degradation manifest in disparate locations across the US. Locations that it would be hard to imagine share any significant infrastructure (Los Angeles and New York City, for example). We thus conclude that the business relationships between impacted Access ISP/Transit ISP pairs is a factor in the repeated patterns of performance degradation observed throughout this research.Now let's see the various broadband providers and their friends try to spin this as well. And lets see if the FCC finally realizes that interconnection points absolutely are a major issue for an open internet, and that the big broadband players appear to be purposely letting those connections clog for the sake of business negotiations.
Thank you for reading this Techdirt post. With so many things competing for everyone’s attention these days, we really appreciate you giving us your time. We work hard every day to put quality content out there for our community.
Techdirt is one of the few remaining truly independent media outlets. We do not have a giant corporation behind us, and we rely heavily on our community to support us, in an age when advertisers are increasingly uninterested in sponsoring small, independent sites — especially a site like ours that is unwilling to pull punches in its reporting and analysis.
While other websites have resorted to paywalls, registration requirements, and increasingly annoying/intrusive advertising, we have always kept Techdirt open and available to anyone. But in order to continue doing so, we need your support. We offer a variety of ways for our readers to support us, from direct donations to special subscriptions and cool merchandise — and every little bit helps. Thank you.
–The Techdirt Team
Filed Under: broadband, business decision, clogging, interconnection, ports clogged
Companies: cablevision, cogent, comcast, level 3, m-lab, time warner cable, verizon
Reader Comments
The First Word
“Subscribe: RSS
View by: Time | Thread
[ link to this | view in thread ]
Re:
[ link to this | view in thread ]
The weak and corrupt FCC won't do anything about it, and broadband providers are never going to willingly open up on their own. At this point we really only have one, last hope: Google Fiber.
[ link to this | view in thread ]
[ link to this | view in thread ]
Re:
The ISPs wouldn't abuse their monopoly position like this would they?
[ link to this | view in thread ]
Re: Re:
[ link to this | view in thread ]
[ link to this | view in thread ]
Re:
The business decision is usually in the form of ISP A saying that Cogent is breaking the initial agreement and wants some sort of kick back, while Cogent is saying that Comcast is the one breaking the agreement and needs more paid links. (usually an ISP will have both even to the same provider)
At the end of the day however it's still the ISP's job to deliver the services that their customers are paying for, so they have to either pony up or route around it.
[ link to this | view in thread ]
Re:
[ link to this | view in thread ]
Re:
[ link to this | view in thread ]
The ONLY thing that would work
But the speed during peak hours is worth much more than speed at night, so there would have to be some calculation done as to worth of connection compared to time of day. With these numbers I would think some place between 60-70% refund.
It would be billions and that is without the MAJOR fine on top. They would listen to that.
But alas, this is but an impossible dream that can never come true.
[ link to this | view in thread ]
Not that it isn't necessarily a reasonable minimum nowadays - but if it is, that says something unfortunate about the footprint of modern Websites... to say nothing of what people think of as reasonable in E-mail nowadays.
[ link to this | view in thread ]
Re:
Meet the new boss, same as the old boss.
Why is the "hope" a Corporation? If land is going to be grabbed for right of ways, why not a co-op?
[ link to this | view in thread ]
Re: Re:
[ link to this | view in thread ]
Re:
[ link to this | view in thread ]
Re:
Damn whippersnappers just don't know what good is.
Seriously though WordPress, Drupal, and other content management systems usually add so much fluff to the webpages and added calls to other sites like JSON usually without compression that things really have gotten out of hand. Check out this site, http://www.webpagetest.org, for some good optimization tips.
[ link to this | view in thread ]
Re: Re:
[ link to this | view in thread ]
Amusing as in ha-ha, interesting, or some other form of amusement?
The 53Kb/s sites lacked javascript, heavy graphics, or sound/motion pictures.
Today's web sites allow the "marketers" who want movement and noise to have that on a web site.
[ link to this | view in thread ]
Re: Re:
[ link to this | view in thread ]
It affected users outside of those areas too
Graph
As you can see, from about 6pm - 2am my connection to my server was basically unusable. It was so bad it would affect ssh! I would get very noticeable typing lag in my ssh sessions.
Once Netflix caved and paid Comcast, it got better immediately and is now back to normal.
[ link to this | view in thread ]
Re: Re:
Why not both? Realistically, Google seem to be making more headway with local governments than smaller municipal schemes, and they're the ones capable of installing new infrastructure. Therefore, they're best placed to break existing monopolies. Once the monopolies start to crumble, then there's room for others to move in, but you may need the elephant to get the ball rolling.
[ link to this | view in thread ]
Re:
[ link to this | view in thread ]
[ link to this | view in thread ]
[ link to this | view in thread ]
Re: Re:
[ link to this | view in thread ]
Re:
[ link to this | view in thread ]
Re: Re:
[ link to this | view in thread ]
Re:
When you comment on this site, that is counted as browsing (according to EE, at least), so of course you wouldn't have any issue with Comcast. Comment again once you've failed in watching Nicki Minaj's latest video on YouTube.
[ link to this | view in thread ]
Re: diatribe
[ link to this | view in thread ]
No support for allegations Netflix was deliberately throttled
[ link to this | view in thread ]
Re: No support for allegations Netflix was deliberately throttled
As you can see, after about nine months -- right as Netflix gave in and started paying broadband providers to unclog their Cogent connections, things miraculously picked up.
Nine months of 'network congestion', and then, as though through a miracle, just as Netflix starts paying them more, suddenly the network cleared right up. You'd have to be blind, either accidentally or intentionally, not to see the link there.
[ link to this | view in thread ]
The Last Word
“