Detailed Report Shows How ISPs Are Making 'Business Choice' To Make Your Internet Connection Terrible

from the consumers-lose dept

A couple of years ago, we wrote about an effort by the big broadband players to push the FCC away from using M-Lab to measure basic network diagnostics on the internet. M-Lab is a very interesting project, focused on collecting a huge amount of data about internet performance, and making that data widely available. In the past, for example, we've highlighted an M-Lab project showing which ISPs were throttling BitTorrent.

Now, M-Lab has released a new report, along with all of the data and a very nice tool to analyze it all, called the Internet Observatory, that looks at ISP interconnection and, most importantly, its impact on consumer internet performance. The end result? You guessed it. M-Lab found that the various ISPs have plenty of capacity, but it appears that they make the business decision to let your internet get clogged. No wonder the big telcos wanted the FCC to move away from supporting M-Lab!

The idea that broadband players would leverage interconnection bottlenecks for a business advantage (at the expense of subscriber connections) isn't a new idea. But now there's (lots and lots of) data to confirm that.

The report looks at New York City, where M-Lab was able to collect a ton of data from a variety of different vantage points. What it found was that starting in the spring of 2013, customers on Comcast, Time Warner Cable and Verizon basically choked off content coming from Cogent (one of the major transit providers) every day from the early afternoon until late at night. Those big ISPs would like to try to blame the transit providers for the clogging, but M-Lab's data shows that's not the case. That's because a fourth ISP in the area, Cablevision, showed no similar degradation at all -- showing that Cogent appeared to have plenty of bandwidth available. Rather, as we've suggested in the past, the big broadband players were purposely letting the connection between Cogent (and other transit ISPs) wither on the vine, rather than opening up new ports (a trivially simple process).

Similarly, M-Lab's data shows that the problem wasn't a lack of bandwidth on the part of the ISPs (i.e., no actual technical congestion), because those same ISPs had no problem connecting to a different transit provider, Internap. So the only logical culprit was the interconnection points. There was more than enough bandwidth to go around. There's just the single bottleneck of the interconnection border router (which, again, is trivially simple to get rid of by opening up more ports). Here are the charts showing all of this. First up, you can see the rather notable sluggishness found between Comcast, Time Warner Cable and Verizon on downloading content via Cogent: And here's what happens from Cablevision users get content via Cogent over that same period. Notice anything different? Oops. Looks like the problem isn't with Cogent's capacity: And, finally, here's the connection of all four broadband providers to another transit provider, Internap, showing that it's not a problem of capacity with the broadband providers themselves. They all seem fine: From this, it seems pretty darn clear that the location of the problem is the interconnection with Cogent between Comcast, Time Warner Cable and Verizon.

And the impact of this business decision by those ISPs to let the border router get clogged is massive for end users. Basically, everyone's broadband connections slowed down to a level no sane person would describe as broadband.

While daily median download throughput overall hovered around 4 Mbps, performance degradation was much worse during peak use hours. For much of the time between Spring 2013 and March 2014, download speeds during peak use hours remained well below 4 Mbps. By January 2014, the download throughput rate during peak use hours for Comcast and Verizon traffic over Cogent’s network was less than 0.5 Mbps, the minimum rate necessary for web browsing and email according to the FCC (Figure 4). Note that only between 2:00 AM and 1:00 PM were the three affected Access ISPs (Comcast, Time Warner Cable, and Verizon) able to attain speeds above 4 Mbps across the Transit ISP Cogent. During peak use, only Cablevision achieved sustained rates above 4 Mbps across Cogent, while all Access ISPs achieved sustained rates above 4 Mbps across Internap throughout the day.

As you can see, after about nine months -- right as Netflix gave in and started paying broadband providers to unclog their Cogent connections, things miraculously picked up.

The M-Lab report shows nearly identical situations with Cogent and broadband access providers in other areas like Dallas, LA and Seattle. While the issue with Cogent is the most obvious (perhaps because Netflix is using it and that was the target of the clogging), the report also found those same broadband providers clogging the other major transit provider, Level 3, though not quite as much as with Cogent. As the report notes, there are indications that the degradation of Level 3 was also for business reasons, not any technical reasons. The degradation with Level 3 happened in a nearly simultaneous manner across many different sites "irrespective of their relative baselines." In other words, even when there was plenty of capacity on all sides, it didn't matter. Verizon decided to start clogging Level 3, and boom. Suddenly performance drops. The report found similar "simultaneous" downgrades in download speeds with another large transit provider, XO as well.

The report's conclusions lay things out pretty clearly:

In conducting this research we find a consistent theme across multiple Access ISPs and Transit ISPs: the interconnection relationships between network operators matter. The quality of service that millions of consumers experience on a daily basis for periods of multiple months can be directly tied to these relationships. Further, these relationships are not simply technical. We see the same patterns of degradation manifest in disparate locations across the US. Locations that it would be hard to imagine share any significant infrastructure (Los Angeles and New York City, for example). We thus conclude that the business relationships between impacted Access ISP/Transit ISP pairs is a factor in the repeated patterns of performance degradation observed throughout this research.

Now let's see the various broadband providers and their friends try to spin this as well. And lets see if the FCC finally realizes that interconnection points absolutely are a major issue for an open internet, and that the big broadband players appear to be purposely letting those connections clog for the sake of business negotiations.

Filed Under: broadband, business decision, clogging, interconnection, ports clogged
Companies: cablevision, cogent, comcast, level 3, m-lab, time warner cable, verizon