DailyDirt: Big Data Isn't Necessarily Better

from the urls-we-dig-up dept

The old -- Garbage In, Garbage Out -- GIGO principle originated during the early days of computing, but it may be even more applicable today. With the explosion of data available that can be collected, there's a temptation to assume that analyses and meta-analyses can make sense of all that data and produce incredible insights. However, we should probably have some skepticism before we jump into the deep end of data and expect miraculous results.

• Microsoft researchers report that they think they can diagnose internet users with pancreatic cancer -- just by analyzing a large number of search requests. It's not clear what can be done with this research since the data was anonymized (and therefore no one can be contacted), and if users know their searches are being monitored for serious health issues -- will they continue to search using search engines that might creepily diagnose them? [url]
• Google Flu Trends attempted to predict flu seasons based on people's searches, but it didn't end up doing such a great job. Future versions of this project could be better, but the initial success of predicting flu trends suffered acutely from "big data hubris" -- a treatable (hopefully) ailment. [url]
• A widely-told parable warning of the mistakes using neural nets tells how the US Army once tried to train software to detect camouflaged tanks from various images. The complex software didn't learn how to detect tanks at all, but instead focused on the clouds that the algorithms determined correlated well with tanks. Oops. Who wants to trust AI to make life-or-death decisions if we can't understand what the machines are thinking? [url]

After you've finished checking out those links, take a look at our Daily Deals for cool gadgets and other awesome stuff.

Filed Under: ai, artificial intelligence, big data, big data hubris, cancer, ed fredkin, flu trends, gigo
Companies: google, microsoft

DailyDirt: Correlation Is Not Causation

from the urls-we-dig-up dept

Big data is a term that's been getting some buzz as the next thing that's going to change everyone's lives (for better or worse, depending on how you look at it). Having a lot of data doesn't necessarily mean you also have a lot of useful knowledge. Garbage in, garbage out, so they say. And making correlations is easy compared to finding a direct causal relationship. However, that hasn't stopped (so-called) journalists from writing misleading headlines. If you hate correlations being mistaken for causation, submit examples you've seen in the comments below. Here are just a few to start off.

• Likes for curly fries on Facebook might correlate with high IQ scores, but don't click that like button just yet. Maybe there are more social experiments being performed on Facebook users than can be accurately counted. [url]
• Former high school athletes seem to get higher paying jobs (at least for the self-reporting men in this study). A lot of skills correlate with various forms of success. Perhaps enjoying the things you do (learned skill or not) is a reward unto itself. [url]
• Measuring the size of brain features can correlate with all kinds of activities, and people have been trying to measure brain sizes for a long time... because there are instruments that can measure the size of various brain parts. The interpretation of these measurements can lead to a lot of faulty conclusions. However, you won't often see the headline: "Watching moderate amounts of porn won’t hurt your brain." [url]

If you'd like to read more awesome and interesting stuff, check out this unrelated (but not entirely random!) Techdirt post via StumbleUpon.

Filed Under: big data, brain, causation, correlation, gigo, iq, journalism, pet peeves, success
Companies: facebook

DailyDirt: Making Money The Old-Fashioned Way... By Algorithms

from the urls-we-dig-up dept

People are changing the way they make decisions now that technology can help them crunch more numbers than ever before. Instead of just going with a gut instinct, decisions can be based on all kinds of random data analysis (for better or worse). Big data is a popular trend, and more and more successful examples of data mining for profit seem to get publicized every day. But are we only looking at the winning combinations and ignoring the losers? Here are just a few examples of algorithms that might be making some money.

• If you don't think your cellphone metadata matters to anyone, venture capitalists might not want to fund your new venture. Apparently, some VCs fund entrepreneurs based partially on an unconventional algorithm that includes things such as the age of the founder's cellphone number and the average time of his/her first call in the morning. [url]
• Can an algorithm pick stocks better than human financial analysts? Sure, but a monkey throwing darts can, too, sometimes. The wisdom of a crowd of analysts might not be a bad algorithm to use, but it still relies on a crowd of humans. [url]
• Poker players can bluff to win, but now that more players are practicing against algorithms and using simulations -- it might be harder for those bluffs to work. It's not so reliable to try to guess when a player is bluffing, but a simulation of thousands of poker hands can give you some statistical confidence.... [url]

If you'd like to read more awesome and interesting stuff, check out this unrelated (but not entirely random!) Techdirt post via StumbleUpon.

Filed Under: algorithm, artificial intelligence, big data, data mining, gigo, poker, simulation, venture capital