As data scientists, a lot of people ask us if data science is better than human intuition. We’ve never been comfortable with the question and here’s why: it’s a false choice. Yet since this question comes up so often, let’s take a minute to flesh it out a bit.

For starters, non-data science businesspeople often look at relatively small samples, but in more depth. A retired CEO may have worked for 10 companies during her career (tiny sample), but she knows each story in rich detail. She knows the tangible facts (ex. financial outcomes) and intangible tidbits (ex. how a drunk Vice President nearly ruined everything at the holiday party).

Data scientists, on the other hand, tend to look at larger samples, but often in less depth. They may examine 10,000 corporate holiday parties to accurately predict when VPs will get drunk and cause a ruckus, but they may not know what soothing words were used to talk a specific drunkard down from the radio antenna.

Both perspectives are important. For example, knowing cases in greater depth ensures that quantitative research is grounded in real-world causality and is less likely to be spurious. Intuition can also help data scientists figure out what variables to consider and how to build an appropriate model.

Meanwhile there’s something to be said about statistics and larger samples. Nobody with a terminal illness wants to hear the doctor say “I’ve only given out this pill twice. Both patients died, but I knew each of them really well.”  It would be a lot better to hear “this pill has been given out 10,000 times and only two people have ever died.  I never met those two people, but there’s a 99.98% chance the pill will cure you.”

The best of both worlds is, of course, large samples that are understood in great depth. That’s the holy-grail.

Non-data science approaches also tend to be faster at first, but slower later on. For example, a lot of business decisions have to be made quickly. There isn’t time to build a predictive model or to even glance around for patterns. If your customer threatens to walk out the door unless you say “yes or no” in the next three seconds, you’d better say something… quick.

Relying on your wits is part of doing business. However if there are big problems that keep resurfacing, it’s a lot slower to go on guessing. If you don’t bring data science or some other form of rigor to the table you may never get a grip on what the underlying problem is. In the long term it’s a lot slower to treat the symptoms (keep guessing) than to cure the heart of the disease (ex. use data science).

In contrast, data science is often slower at first, but faster later on. It can take hours, months or even generations for data science to model, predict or solve some of business’s toughest riddles. However once you’ve built a robust tool, it’s relatively fast and easy to handle or even prevent those problems if they threaten to pop up again.

A final difference is the “intuitive-ness” of the answer.  If you’re facing a problem where the answer is intuitive, then human intuition may be a good tool for solving it. Yet if you’re facing a problem where the answer may be counter-intuitive, data science is a good approach since human intuition is more likely to lead you astray.

While these are generalizations (exceptions certainly exist), data science and non-data-science-based approaches tend to differ in the following ways:

data science vs non data science

Seen this way, the guiding question becomes: what circumstances are you in? What kind of problem are you trying to solve?

If you need a quick decision for a highly intuitive problem that isn’t likely to come up again, a non-data-science-based approach may be good enough. However if you have a big problem that keeps coming up and the answer may be counter-intuitive, data science is probably the way to go if you’re willing to do some extra diligence up-front.

Perhaps the most exciting thing about intuition and data science is how well they complement each other. Like an old married couple, each one’s strengths complement the others’ weaknesses. They also help each other grow. Intuition can give data science a place to start looking for patterns and add depth to what patterns are revealed. Meanwhile the lessons learned through data science can become part of people’s intuitive understanding of the world and how it works.

So we beg you; resist temptation to pit one against the other. We’re not being diplomatic. The reality is, intuition and data science are good at different things. Like a shovel and a garden hose – you need them both, but for different tasks. It isn’t us versus them. It’s us plus them.

Share →

25 Responses to Data science vs intuition: which is better?

  1. Ken says:

    You need intuition to make the connections data science may provide. You can have the greatest data base in the world…but it’s the question…that’s. most important…Hitchhikers guide.

  2. Peter says:

    There’s a great chapter on this very topic in Thinking, Fast and Slow, by Daniel Kahneman, an excellent review of current understanding on how we think.

    Intuition by experts in fields with fairly immediate feedback (fire fighting, surgery, etc.), largely turns out to be memory: our brains have recognized a situation from either our own experience or stories we have heard from other people, and then presented us with the memory of what happened next, that we take as intuition.

    Intuition by experts in fields without fairly immediate feedback (say, politics, human behavior, etc.) is largely hocum, and is better replaced by simple algorithms based on observations (i.e., Data Science). We don’t know what we what know, so fools rush in!

  3. Julie says:

    OH OH! Why are we back to either or?

  4. Mohan says:

    Data Science, obviously! Both Data and Science (meaning scientific method) are both preeminent in preeminent in decision making (which is at the heart of business). That is why companies who know how tap the power od data and analytics outperform those that do otherwise, consistently.

  5. Doug says:

    Nicely put. There’s a connection between the two which is important to note:

    Quite often, it’s intuition and small sample research that’s required to point us into the highly profitable ways to use data science. In other words, data reveals nothing on its own. And tends only to reveal what the data scientist chooses to investigate. Those choices are driven by (drum roll please) intuition or learnings from small sample research.

    If you will, it’s the classic research approach of digging deep in qualitative research to uncover the universe of issues to be explored. Then crafting quantitative research to learn which issues are most/least important and further quality populations (for example) among which they are important.

  6. Ted says:

    Einstein has said: “The intuitive mind is a sacred gift and the rational mind is a faithful servant. We have created a society that honors the servant and has forgotten the gift.”

  7. Mimi says:

    YES – data science tells you the difference between A / B / C. We still need people with brains to determine how to apply that information generated and if it’s valid.

  8. Eunika says:

    Intuition is necessary to take a right direction in research. Vision, intuition and feeling are the muses of the scientific.

  9. Bob says:

    This question is like – “Which is best for survival, air or water?”. Intuition is your model of how things work, data determines how far your model can be used to predict in advance of hard data. Ultimately data directs modifications to your model which may be simple or subtle – talking understanding in a whole new direction.
    Water with no air is called drowning and humans can only live a few days in air without water. Similarly there is no advance without a symbiosis of data and intuition.

  10. Joseph says:

    Both!

    Data is just that: data. Without intuition leading you to ask the best questions from the data, it does not yield innovation.

    However, as an electronics design engineer, I have seen strokes of genius waylaid by the trickiest limitations of physics and technology.

    But data never says “NO!”. “Will never work!” “Can’t be done!” is just LACK OF VISION.

    Intuition – the VISION to see what could be, what should be what will be.
    Data – the map of HOW it will come to be.

    “Engineer’s don’t get smarter as they advance; they just learn to ask better and better questions.” – unknown

  11. Doug says:

    Nicely put. There’s a connection between the two which is important to note: That it’s important to use both in their roles to support each other.

    Quite often, it’s intuition and small sample research that’s required to point us into the highly profitable ways to use data science. In other words, data reveals nothing on its own. And tends only to reveal what the data scientist chooses to investigate. Those choices are driven by (drum roll please) intuition or learnings from small sample research.

    If you will, it’s the classic research approach of digging deep in qualitative research to uncover the universe of issues to be explored. Then crafting quantitative research to learn which issues are most/least important and further quality populations (for example) among which they are important.

    The quantitative results then feed back into more intuitive or qualitative work. And so on…

  12. Michael says:

    One quote from that paper always stands out when I hear the word ‘intuition’ ( which, IMHO, is used all too often interchangeably for ‘assumption’ :-) :

    “Remarkably, the intuitive judgments of these experts did not conform to statistical prin-
    ciples with which they were thoroughly familiar. In particular, their intuitive statistical inferences and their estimates of statistical power showed a striking lack of sensitivity to the effects of sample size. We were impressed by the persistence of discrepancies between statistical intuition and statistical knowledge, which we observed both in ourselves and in our colleagues. We were also impressed by the fact that significant research decisions, such as the choice of sample size for an experiment, are routinely
    guided by the flawed intuitions of people who know better.”

  13. Marcos says:

    The article does in fact advocate for both and provides a nice rubric for thinking about the strengths and weaknesses of each approach. I think the word ‘intuition’ is a little misleading here. ‘Intuition’ implies some kind of snap judgement based on a mysterious gut feeling. It sounds like what we’re talking about here is the difference between qualitative and quantitative approaches – between understanding built on relatively fewer cases but with greater depth and nuance (e.g. experience, case studies) and understanding built on lots of data/cases (e.g. statistics, pattern analysis) but without deeper understanding of context..

  14. Buddy says:

    Intuition is a spark to ignite the data in determining a theory. However, if the foundation – the data, is flawed or a poor sample, then borderline statistics could lead to erroneous theories. I found that a least squares fit on non linear data is not as good as trial and error for y=ax^b for a & b. Science proves the fallacy of science but will merge to the truth when science isn’t flawed by interpretation.

  15. Fouad says:

    Well – Analytics vs. Intuition and/or Academic vs. Intuition!
    The interplay between analytics and intuition in managerial decision making started over several conferences actually in Europe – A kind of golden wildcard !
    On the reverse way – we discover many primary school were academic + Intuition were learned as Win-Win combination …
    So – my wish would be to not dissociate them as such and start thinking out-of-the-box – hence the bonus 1+1=3

  16. Murilo says:

    I would say 1+1= X…
    It’s a real adventure!

  17. Joe says:

    I agree with the article, it is a false choice, they are two skills, one learned and one honed.

    The other key item to note is that they are not mutually exclusive, as anyone that has ever set up a Naive Bayes classifier knows well;^)

  18. Mihel says:

    inductive science doesn’t exclude deduction or inductive reasoning. Even analogies are acceptable. As long as basis is sound.

  19. Rebecca says:

    These two views of “data” and “intuition” need not be seen as contradictory at all… these are just different levels of knowledge each having a different purpose and scope.

    The following link to a video talk [Entering the realm of reflexive knowledge] clarifies the different levels of knowledge, the roles and scope of each –
    http://srinivasv.com/becoming-a-true-blue-knowledge-practitionerentering-the-realm-of-reflexive-knowledge/

    The video talk also introduces 2 journeys of knowledge and clarifies the various levels within each.
    “The first journey of knowledge is from data to information to interpretation/ analysis (Analysis of various sorts is what they call knowledge – reasoning, analysis, etc.)…”

    “The second journey of knowledge is where you start recognizing that knowledge has an inner dimension – i.e., there is knowledge within you. Then knowledge starts taking a completely different form. This is what they call in knowledge management – the wisdom space….”

  20. Haga information…

    I’m more than happy to uncover this great site. I want to to thank you for your time due to this wonderful read!! I definitely really liked every little bit of it and i also have you book-marked to see new stuff on your blog….

  21. jacqueline friedberg…

    Saved as a favorite, I really like your blog!…

  22. Della House says:

    A picture may or may not be worth a thousand words, but a picture is certainly worth a thousand numbers. The problem with most data analysis algorithms is that they generate a set of numbers. To understand what the numbers mean, the stories they are really telling, you need to generate a graph. Edward Tufte’s Visual Display of Quantitative Information is the classic for data visualization, and a foundational text for anyone practicing data science. But that’s not really what concerns us here. Visualization is crucial to each stage of the data scientist. According to Martin Wattenberg ( @wattenberg , founder of Flowing Media) , visualization is key to data conditioning: if you want to find out just how bad your data is, try plotting it. Visualization is also frequently the first step in analysis. Hilary Mason says that when she gets a new data set, she starts by making a dozen or more scatter plots, trying to get a sense of what might be interesting. Once you’ve gotten some hints at what the data might be saying, you can follow it up with more detailed analysis.

  23. Data science is changing the way we look at business, innovation and intuition. It challenges our subconscious decisions, helps us find patterns and empowers us to ask better questions. Hear from thought leaders at the forefront including Growth Science, IBM, Intel, Inside-BigData.com and the National Center for Supercomputing Applications. This video is an excellent source of information for those that have struggled trying to understanding data science and its value.

  24. Janell Vacha says:

    Read More…

    I discovered your blog site on google and check a few of your early posts. Continue to keep up the very good operate. I just additional up your Rss feed to my MSN News Reader. Seeking forward to reading more from you later on! ……

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>