New Report Debunks The 'Bandwidth Hog' Myth
from the and-data-caps-are-improving-infrastructure-how? dept
ISPs and cell phone companies have long been known for raising the spectre of the "bandwidth hog" in order to justify data caps or bandwidth throttling. Considering most ISPs are also television providers, throttling and capping data usage helps push some customers back towards their TV sets. (Depending on how many other eggs are in the company's basket, this also helps push them back into theaters and tilts their heads back to regular old radio, as these caps make streaming services very unattractive.)Karl Bode at DSL Reports has finally obtained some data which confirms what many of us had suspected all along: the Bandwidth Hog is a convenient bogeyman rather than an actual being. Analysts Benoit Felton and Herman Wagter managed to talk an "anonymous mid size DSL company from North America" into sharing its data on customer usage. Not surprisingly, despite their open invitation to the ISPs to contest their "disruptive user" argument, no other broadband/DSL service offered to provide any data, suggesting they're already well aware of what usage numbers actually show.
In a blog post, Felten notes that the pair took real user data for all customers connected to a single aggregation link and analyzed the network statistics on data consumption -- in five minute time increments -- over a whole day. What they found is that capping ISPs often don't really understand customer usage patterns, and are confusing data consumption (how much data was downloaded over a whole period) and bandwidth usage (how much bandwidth capacity was used at any given point in time).That's a lot of percentages and percentages of percentages. Fortunately, a commenter at DSL Reports was able to use the "dreaded" highway analogy to simplify things:
What they discovered is data that runs in stark contrast to a lot of the claims put out there by some familiar, larger ISPs when justifying caps and overages. Among the pair's findings is that the top 1% of data consumers (which they call "very heavy consumers," instead of the already adversarial "hog") account for 20% of the overall consumption.
Looking deeper into the data, they also found that about 61% of very heavy data consumers download 95% of the time or more, but only 5% of those who download at least 95% of the time are very heavy data consumers. While 83% of very heavy data consumers are amongst the top 1% of bandwidth users during at least one five minute time window at peak hours, they only represent 14.3% of said Top 1% of users at those times.
1% of vehicle drivers on the road travel a disproportionate amount of miles compared to the average driver. But they are on the road all the time. Most of the time they are on the road there is no rush hour congestion.The heavy drivers are likely to be involved in rush hour traffic jams, but only represent a small, not terribly relevant, fraction of total drivers in the traffic jam.Limiting the amount of miles a driver can drive, does nothing to widen the roads and little to keep people off the roads during traffic jams, thus does not help with congestion.In other words, internet usage tends to be heaviest at certain points of the day, and installing caps or throttling supposedly heavy users does nothing to relieve that congestion. Instead, it punishes users across the board by hitting some of them with additional fees and offering very little in the way of improving connection or speed for the rest of the users online during these "traffic jams."
Felten concludes that ISPs themselves need to better understand the difference between data usage and bandwidth consumption, or face driving their customers to more reasonable competitors. That's assuming consumers have a choice, given caps exist in many markets largely due to no competition.That's the real problem. For many people, there are few options. And most of the ISPs are more than happy to install caps and overage fees, especially if someone in the market is already doing just that. Bode also notes the adversarial relationship with their customers that the ISPs are creating through the usage of terms like "bandwidth hog" or "disruptive user." Rather than look into improving infrastructure, they'd much rather vilify certain paying customers in order to deflect attention away from their service limitations.
Karl Bode expands on this:
It would also be naive to assume many of the larger ISPs -- stocked with number crunchers and network analysts -- don't already know everything Felten stated. However, there's a reason that ISPs don't like bandying real, raw data about -- and it's because there's a few large carriers that like to use bogus science to justify anti-consumer behavior, most recently with AT&T's announcement of caps and overages for DSL and U-Verse users. When asked to prove that these caps and overages were necessary, AT&T couldn't -- something ignored by general tech press coverage of the move.While it's nice to finally have some data on hand to debunk the "bandwidth hog" myth, most ISPs will be able to dismiss it as not being representative of their customers' usage patterns. After all, they're still refusing to provide any data to back up their claims. In areas without competition, this myth will still be used as a scapegoat for everything from hard caps to lousy connections.
As we've noted repeatedly, most carriers impose caps and overages claiming it's due to either network congestion or financial necessity. In realty, caps and overages are implemented by carriers that simply want to jack up the cost of bandwidth so they can protect TV revenues from Internet video by making Internet video more costly and less appealing. The financial "necessity" of moving away from the flat-rate pricing model is proven false quarterly by earnings reports.
Thank you for reading this Techdirt post. With so many things competing for everyone’s attention these days, we really appreciate you giving us your time. We work hard every day to put quality content out there for our community.
Techdirt is one of the few remaining truly independent media outlets. We do not have a giant corporation behind us, and we rely heavily on our community to support us, in an age when advertisers are increasingly uninterested in sponsoring small, independent sites — especially a site like ours that is unwilling to pull punches in its reporting and analysis.
While other websites have resorted to paywalls, registration requirements, and increasingly annoying/intrusive advertising, we have always kept Techdirt open and available to anyone. But in order to continue doing so, we need your support. We offer a variety of ways for our readers to support us, from direct donations to special subscriptions and cool merchandise — and every little bit helps. Thank you.
–The Techdirt Team
Filed Under: bandwidth, bandwidth hog, broadband
Reader Comments
Subscribe: RSS
View by: Time | Thread
Clear is a perfect example of Boneheadedness
I dropped to less than 56k speeds for more than a day anytime I tried to watch a Netflix video at the same time as I downloaded a large file after I moved to a new house in a more affluent area. Even though my monthly usage never changed it averaged the same for ever month for the last year.
Well you say they did it because I was sucking up too much bandwidth during peak times. I say false. The bandwidth I used (85% of it, by their charts) was used during the slow period during the business day. (I do IT support so I would be doing support and watch Netflix during the business day.)
I wasn't a heavy user by any measure. 3.6GB a month.
That "throttling" pushed me away from Clear and also stopped me from recommending them. Before I saw that I was making around $1250 a month recommending them.
In the year when my service was good I had over 600 people and companies sign up for Clear.
[ link to this | view in chronology ]
It's a problem with higher priced "services".
What I discovered was a blatant rip off of consumers gullible to fall for "faster" speeds. Everything at peak time dropped down to speeds my system was already seeing before the upgrade.
Here's the kicker: When I asked if they were throttling during peak time, they said no. So I asked, "Basically what you're saying is I'm paying for the privilege of driving in the fast lane but because your networks allow for slower cars to be in my lane, I have to be slowed by them?"
I dropped the extra speed and the associated charge and went right back to the speeds I can never surpass, all for the wonderful price of $Suck it, customer (tax not included, as well as "fees").
I contacted my local government regarding a data hub and bandwidth service as a means to offset their inability to allow competition to enter the marketplace, restricting us only to Brighthouse.
Hear that laughter? Yeah, it's been over two years. I don't think they'll ever stop until someone's bold enough to help finance most of the cost to get it done.
Sometimes, I truly despise Corporate America.
[ link to this | view in chronology ]
Re: It's a problem with higher priced "services".
Cities have already tried that, then corporate lobbyists come in and ban the whole projects.
What we need is someone willing to fund political campaigns that will allow competition.
[ link to this | view in chronology ]
Re: It's a problem with higher priced "services".
only sometimes?
[ link to this | view in chronology ]
[ link to this | view in chronology ]
Re:
ISP claim: 1% of users are using too much bandwidth
ISP Solution: Set limits that affect 80% of users
Real world cure: Set limit just below that which the 1% are using that is causing the problem. You only affect the problem people.
Or, you know, actually market and sell what you can reasonably supply rather than unlimited data for 25Mb+ down for everybody that WILL overload your networks when people actually use that much.
[ link to this | view in chronology ]
I'm sure if I did nothing but stream movies, I could get pretty high, but that's a little unreal. About 90% of the people will NEVER go above 1gb a month.
[ link to this | view in chronology ]
Re:
[ link to this | view in chronology ]
Re:
[ link to this | view in chronology ]
Re: Re:
[ link to this | view in chronology ]
Re: Re: Re:
[ link to this | view in chronology ]
Re: Re: Re: Re:
[ link to this | view in chronology ]
Re: Re: Re: Re: Re:
[ link to this | view in chronology ]
Re: Re: Re: Re: Re: Re:
[ link to this | view in chronology ]
Re:
Yeah, I'd find it difficult to use more the 3GB on my Evo, but I can very, very easily exceed 3GB in under a day on my home DSL connection by just buying a couple of games on Steam.
This post, the article it points to, and the research discussed there is about broadband service, not mobile service, which is a whole other kettle of fish. Conflating the two is at best confusing and at worse intellectually dishonest, muddies the discussion, and ultimately works to the advantage of the entrenched telecoms that trying to extract monopoly rents from their customers in exchange for actually lowering the quality of service.
[ link to this | view in chronology ]
[ link to this | view in chronology ]
Brown Sugar
1. As soon as a business get's a taste, they're hooked. They can rarely transition back to a competitive environment.
2. They'll spend millions lobbying and influencing policy to make sure they can keep getting high.
3. When people try to give them an intervention, they lash out with anger, hostility, and complete denial.
4. They'll systematically destroy their customer relationships, waste their future opportunities, and eventually wind up bankrupt in the gutter.
[ link to this | view in chronology ]
Re: Brown Sugar
[ link to this | view in chronology ]
Re: Re: Brown Sugar
[ link to this | view in chronology ]
Re: Brown Sugar
I just got a $10 credit for calling Verizon's FiOS bullsh!t onto the carpet.
This stuff gets logged and come renewal time, they have to answer for it.
[ link to this | view in chronology ]
Re: Re: Brown Sugar
[ link to this | view in chronology ]
Re: Re: Brown Sugar
10 dollars. 10 DOLLARS! Wow, you're rolling in it. That's just over .5% of the average annual cost of service. I bet I could get my bill permanently reduced by 10 dollars a month, all you have to do is bitch at a bunch of customer reps.
The point is that people want a high quality of service when paying a high price; no one is really concerned with saving 10 dollars, if it means crappy service.
[ link to this | view in chronology ]
It's designed to withstand short bursts of increased (i.e. hogging) traffic and survive it quite nicely, thanks. It's also designed to be unnoticeable to the customer. (Poorly designed or rural and suburban cable subscribers can and do still notice that bandwidth decreases between 4 and 8pm when people come home and flip in the TV and computer. The bandwidth goes back to normal after 8pm. A great deal of that is the "shared" aspect of coaxial cabling vs twisted pair.)
It's largely been a false argument from the start. Where there is fibre to the house it's a completely false argument as, theoretically, the bandwidth/bitstream in fibre is infinite. And before any screaming starts that there is still copper out there, I know that. The closer your bit of copper is to a cross connect that itself is fed by fibre the less the copper restricts bandwidth/bitstream transmission. (though if you wire your house with cat 3 [cheap] tel wire you deserve what you get.)
It's a money grab, pure and simple. There's little or no technical validity to it at all.
PS: And as AT&T well knows 99% of all data and voice switching is not done externally and internally by light not copper or relays or any of that junk in the central offices.
[ link to this | view in chronology ]
Re:
PS: And as AT&T well knows 99% of all data and voice switching is done externally and internally by light NOT copper or relays or any of that junk in the central offices.
[ link to this | view in chronology ]
Ostensibly this throttling is to ensure heavy users don’t hog bandwidth while there are a lot of users online. I see two huge problems with this:
First, the providers have oversold their network. They have sold more bandwidth than they are capable of providing at any one time. This is blatantly dishonest. We pay for an 3.5MB BT connection (the fastest available in my postcode) but rarely see speeds of even a third of that. A recent OFCOM study found that most UK internet users get speeds well below what they signed up to.
Second, the result of this throttling is that at the time when most people are online, the experience is at it’s worst. In the evenings on my connection it is often not possible to stream a 1 minute YouTube video in its entirety. If other users are similarly affected, the ISPs will have a lot of peeved off customers.
The rate at which the big ISP networks in the UK are being updated is also abysmal. BT have only recently started offering 20MB (though in reality this is closer to 8MB) fibre connections in heavily built up areas.
Meanwhile the smaller Virgin, whose customers actually do get the advertised speeds, and sometimes even higher ones, are rolling out 100MB connections in cities. Four years ago, when I lived in the centre of Belfast, we had a 20MB Virgin connection. It will be another two years before I can get a 20MB BT connection the large town I live in now.
Even here where we have greater competition, there is very little to differentiate between any of the providers. The prices are much the same, the speeds are much the same and the throttling and caps are all much the same. I’m not sure how this situation came about, but I find it problematic.
The only exception is Virgin. Their service and support seem to be pretty great but they are not widely available outside urban residential centres.
[ link to this | view in chronology ]
Re:
[ link to this | view in chronology ]
Re: Re:
[ link to this | view in chronology ]
two ways to avoid the problem
1. At night (1am to 7am), I assume their network is almost completely idle. So introduce a modified BitTorrent client that only downloads during that period (or price bandwidth based on day/night use). If you want a movie or TV program, get it at night for viewing the next day. They wouldn't see any strain from bandwidth hogs at all.
2. Clone the NetFlix streaming library inside their walls. At 500 meg an hour for video (350 meg is more typical), a $50 terabyte drive holds 2000 hours. If NetFlix has 100,000 hours of streaming video, that's 50 drives ($2500). Even hardened and replicated, it would cost nothing to put that much data inside the cable system servers. Then all the bandwidth currently going outside the system to NetFlix disappears with no difference to the customer. Kick some of the money saved to NetFlix. Problem solved.
[ link to this | view in chronology ]
Re: two ways to avoid the problem
Better experience for all.
[ link to this | view in chronology ]
Re: Re: two ways to avoid the problem
Forget you Autodesk and Adobe. I'll just grab the install media later.
[ link to this | view in chronology ]
Re: Re: Re: two ways to avoid the problem
[ link to this | view in chronology ]
Re: two ways to avoid the problem
The differences in areas with fiber buildouts are telling. You see 2 major things that you won't see in your typical broadband monopoly/duopoly market: Competition and No Caps or Throttling.
[ link to this | view in chronology ]
Re: two ways to avoid the problem
Ooooh, but you would be making a copy. The content providers would go apeshit.
[ link to this | view in chronology ]
Re: two ways to avoid the problem
I don't think you can find a 1TB for under $100 these days. (there is one one pricewatch, all others start at $105)
so your 2500 figure goes up to $5000 but not bad still.
Either way, Content Delivery Networks already do this sort of thing.
[ link to this | view in chronology ]
Re: two ways to avoid the problem
If the ISP won't do it, I sure will LoL
[ link to this | view in chronology ]
[ link to this | view in chronology ]
Re:
[ link to this | view in chronology ]
Most users take data in short and peaky ways, usually loading a webpage with images and / or a video clip in short order, and then sitting reading it / watching the video with no additional bandwidth usage for a period of time. A great example is writing a comment here, downloading the page was a short burst, followed by the time it takes the write a comment as "dead" time.
Constant users "raise the floor" and leave less space for the burst users to get their data. If you have a 10 gig connection, but 5 or 6 gig of it is constantly in use by downloaders, you effectively only have 4 or 5 gig to support all your other burst users. It becomes more critical as you get down to the ends of the network (last mile), where you might have only a 100 meg network connection providing service to a group of end users. You get a few people downloading constantly, and much of that bandwidth disappears, slowing the burst users access.
It's why things like Netflix are such a problem for many ISPs, because they represent a data flow that is not in keeping with their network designs or bandwidth plans.
[ link to this | view in chronology ]
Re:
[ link to this | view in chronology ]
Re: Re:
When you consider that network hardware is typically purchase on anywhere between a 3 year and 10 year depreciation cycle, you will understand that changes just don't happen overnight.
It should also be said that if they were working their networks to be able to handle the P2P onslaught (which most seem to agree is slowly subsiding), they may find themselves once again with a network that isn't optimized for the current trends - and by the time they change it again to be there, the target will have moved again.
Just as importantly, the ISPs continue to face downward pricing pressures in their businesses, and yet the consumers are expecting them to spend more money, to raise connectivity speeds, and to provide an endless fountain of bandwidth - all at that lower price.
You have to remember it is a real world business, and that business doesn't move at the speed of the internet.
[ link to this | view in chronology ]
Why can other parts of the world provide better bandwidth at a cheaper price?
[ link to this | view in chronology ]
My experiences
I recently switched from Rogers (the incumbent cable provider in my neck of Canuckistan) to Teksavvy a recent startup.
I couldn't be happier. No caps, no throttling (which was hitting among other things my encrypted uploads of said 100s of megs asset files). We cancelled all cable tv, and I pay substantially less for a better connection.
The idea of caps and throttling is lame, I was paying ~$65/mo +tax for a cap of 100 gigs/mo, which, in this day and age is ludicrous. So I was paying on avg another $20-$40/mo in overage charges.
That plus throttling affecting my work, and Rogers contant billing errors (all in their favour), left me cranky as hell.
I am thrilled with the new service and will never go back to capped and throttled. Anyone who is able to switch from Bell/Rogers to Teksavvy, I HIGHLY recommend.
[ link to this | view in chronology ]
Re: My experiences
[ link to this | view in chronology ]
My solution to being capped
[ link to this | view in chronology ]
Re: My solution to being capped
Look to the generational differences with regards to answering the phone.
I always answer the phone unless it just would not be appropriate. Because I grew up with the idea that all calls were worth answering.
[ link to this | view in chronology ]
Bad analogy
[ link to this | view in chronology ]
This is complete bullshit! What about the ISPs who are not involved in marketing television, the local telcos, or local wireless ISPs. They do it in an attempt to limit the usage.
Highways will only support so much traffic, and it isn't just the number of cars on the highway that causes issues, but the size of the cars (the number of packets). If you imagine the bandwidth as the highway system, you need to refine the analogy to get an acurate picture. Each service running on each actively communicating device in a home transfers some data, some more than others. Instant messaging clients typically use very little data, email without attachments is also very little (those can be SMART cars), Web sites and emails with photo attachments we will call sub compacts, music streaming services would be buses, and video streaming services would be caravans of 18 wheelers. Video files are huge, enormously huge compared to other network traffic.
Lets say you can get 1000 sedans driving down the highway at 65mph, if you throw a bunch of 18 wheelers onto that same highway the congestion will slow the entire road down substantially. Drivers complain about the slow traffic and no one wants to pay for a larger highway so you limit traffic by putting up throttling red lights at highway entrace ramps. You limit the number of people that can get on the highway to improve the efficiency of they highway system.
This used to happen back in the days of dial up when you would attempt to connect and get a busy signal. Now that everyone has always on connections its just a flood of traffic.
[ link to this | view in chronology ]
Re:
When trying to make the case of saying video files are like semi's, these would have to be semi's that maintain the same speed as the SMART car at all times, it's just the end of the SMART car would cross the finish line before the end of the semi would.
[ link to this | view in chronology ]
"We need caps to offer better service for all."
What they mean:
"We are too cheap to upgrade the network in your area so we are going to lie about usage instead and tell you to spend more money"
[ link to this | view in chronology ]
Re:
I think you meant, "We, the customers, are too cheap to pay the additional fees to that would required to upgrade the network in our area..."
[ link to this | view in chronology ]
Re: Re:
It's like if you hired a housekeeper, and every year, your house got messier and messier because the quality of her service degraded......that's kinda how it feels....
[ link to this | view in chronology ]
Re: Re:
Talk about blaming the victim.
[ link to this | view in chronology ]
Re: Re:
Bulk rates for internet bandwidth is $1/mbit/month for dedicated bandwidth with a SLA.
So maybe a nice 900% profit, so $10/mbit/month?
Hey, I'm willing to pay $100/month for 10mbit of symmetrical dedicated bandwidth, even though I would be getting price gouged.
[ link to this | view in chronology ]
Oh they understand alright....
[ link to this | view in chronology ]
Bandwidth Hog Myth
[ link to this | view in chronology ]
First, some caveats:
- I love unlimited data caps (I have one, and I'm a "data hog")
- Competition kills data caps eventually (I'm in Australia: 5 years ago my cap was 10GB, the highest cap was around 50GB. Now there are caps of 350, 500, 1000GB, and even unlimited.
But there is simply no denying that caps DO reduce congestion (even if, as this data shows, it's only really relevant at peak times). 10-15% at peak times is nothing to be sneezed at.
I have a degree in computer science, if that makes the slightest difference to the credibility of my opinion.
[ link to this | view in chronology ]
Re:
They have shown time and time again, that one average, most people don't even know there are caps, yet alone change their data usage because of caps.
Most people can't link what they do with data consumption. As far as they care, reading an email takes as much bandwidth as P2P or Netflix.
Most people rank their data consumption in time spent on the internet, not on types of web services they use.
[ link to this | view in chronology ]
Download 95% of the time? Huh?
61% of very heavy data consumers download 95% of the time"
Does that mean download non-stop for 23 hours a day? I imagine not, but I can't guess what it does mean.
[ link to this | view in chronology ]
Re: Download 95% of the time? Huh?
[ link to this | view in chronology ]