Everybody Lies | Seth Stephens-Davidowitz

Summary of: Everybody Lies: Big Data, New Data, and What the Internet Can Tell Us About Who We Really Are
By: Seth Stephens-Davidowitz


Embark on an insightful journey into the world of big data with Seth Stephens-Davidowitz’s ‘Everybody Lies: Big Data, New Data, and What the Internet Can Tell Us About Who We Really Are’. The book highlights the power of big data and provides a plethora of examples, including how Google’s efficient use of collected data made it into the giant it is today. Understand the difference between correlation and causality, and how big data can assist in conducting randomized controlled experiments, also known as A/B tests. However, big data is not without its limitations and ethical concerns, and Stephens-Davidowitz thoughtfully explores these issues as well.

Intuition Meets Data Science

Big data refers to an enormous volume of information requiring computational power to recognize patterns. It’s interesting to note that data science has an intuitive aspect, as we are all, in a way, data scientists. Our experiences allow us to gather information that we use to spot patterns and make predictions. However, relying solely on intuition isn’t scientific; incorporating data leads to a more refined and accurate worldview. Even though an intuitive perspective can take us far, data can refine and enhance our understanding of various aspects of life, including relationships.

Take, for example, the author’s grandmother. At 88 years old, she had witnessed countless relationships come and go, gathering information on what made them successful. One Thanksgiving, she began suggesting the qualities the author should seek in an ideal partner. From her years of observing relationships, she knew which characteristics would guarantee a lasting bond.

Grandma was convinced that relationships with mutual friends endured longer. Her belief was based on personal experience, as she and her husband often spent evenings with their close friends in Queens, New York. But despite her intuition and years of experience, hard data proved otherwise. A 2014 study by Lars Backstrom and Jon Kleinberg, utilizing Facebook data, showed that couples with more friends in common were more likely to change their relationship status from “in a relationship” to “single.”

This example illustrates the importance of blending intuition with data science. Data provides the material necessary to confirm or disprove our initial gut feelings, helping us recognize more accurate patterns and predictions than personal experience alone could offer. In short, while intuition can be a valuable asset, incorporating data-driven insights contributes to a more comprehensive and precise understanding of the world around us.

Unveiling Big Data’s Power

Data science becomes truly exceptional when it not only collects vast amounts of data but also utilizes it to reveal patterns and make predictions. Google exemplifies this remarkable potential of big data. Founders Larry Page and Sergey Brin revolutionized search by creating an efficient algorithm that analyzed website links to improve relevance and user experience. Big data provides a continuous flow of novel information, inaccessible prior to technological advancements, thus allowing real-time monitoring and insights on various phenomena, such as global health patterns and unemployment rates.

Data science holds immense potential, yet its true charm lies not in the quantity of data it amasses but the usefulness of that data in discerning patterns and assisting in predictions. Google’s story perfectly demonstrates this notion. Co-founders Larry Page and Sergey Brin didn’t make their search engine wildly successful by merely gathering large amounts of data – they ensured such data could be employed effectively.

Before Google’s arrival, search engines produced results based on the frequency a phrase appeared on a webpage, often yielding irrelevant outcomes. Page and Brin’s innovative algorithm transformed the search landscape by considering the number of links directing users to a particular website, which indicated its relevance to the searcher. Their method helped connect users with pertinent information based on link patterns and usage behavior.

Big data’s incredible power comes from four key principles, the first of which is its novelty. It opens up a relentless stream of fresh information never before available. In the past, people waited for agencies such as the Bureau of Labor Statistics or the Centers for Disease Control and Prevention to release reports on unemployment rates or disease infection rates, respectively.

Today, Google’s big data capabilities allow real-time tracking of previously untapped patterns and trends. For instance, engineer Jeremy Ginsberg showcased how flu-related Google searches could be used to monitor the spread of the illness across geographic areas and time, reflecting big data’s transformative potential in various domains.

Unveiling Truths Through Big Data

A study from the University of Maryland revealed that people tend to lie on surveys to project a favorable image of themselves. This phenomenon, known as social desirability bias, makes such results unreliable in understanding human behavior, thoughts, and beliefs. Big data, on the other hand, offers a powerful alternative, as it provides unfiltered insights into people’s online behavior – which is less likely to be skewed – unveiling surprising truths about many aspects of our lives.

The findings of a survey conducted among graduates at the University of Maryland revealed a shocking discrepancy between self-reported GPAs and official records. A mere 2% of respondents claimed to have a GPA below 2.5, while the actual number stood at 11%. The phenomenon observed here is a testament to a universal truth about surveys – people often misrepresent the truth to paint themselves in a better light. This behavioral trait is known as social desirability bias.

Apart from this bias, another factor contributing to dishonest responses in surveys is the desire to impress the person administering the survey. Our tendency to provide untrue or exaggerated answers is heightened when we feel like we need to make a good impression, regardless of anonymity.

These untruths render surveys unreliable for understanding genuine behavior, thoughts, desires, and beliefs of individuals. In contrast, big data offers an exciting alternative due to its inherent honesty. Extracted from our online behavior, big data is less likely to be clouded by lies or skewed results since our search engine queries do not involve a human questioner.

Take, for example, the topic of unconventional sexual fantasies involving fruit. While most people wouldn’t readily admit to such preferences in a survey or interview, a data analysis of the search terms on the adult website PornHub revealed an eye-opening search – “anal apple.” This highlights the potential of big data to uncover truths that people might be hesitant to share with another person directly.

By utilizing big data, we can bypass the limitations of social desirability bias and access a more accurate understanding of human behavior, desires, and beliefs.

Big Data’s A/B Testing Impact

In a world full of correlations, establishing causality can be challenging. However, big data simplifies this process by making A/B tests more accessible, revealing the actual cause-and-effect relationship. Previously, conducting A/B testing was a tedious and demanding endeavor, requiring recruitment, surveys, and in-depth analysis. Nowadays, data scientists can write programs that effortlessly analyze data from A/B tests. A prime example of this powerful application is Barack Obama’s 2008 presidential campaign. By utilizing various website layouts and analyzing their performance, Obama’s campaign was able to determine the most successful design, boosting sign-ups and donations. In summary, big data’s ability to streamline A/B testing has significantly contributed to our understanding of causal relationships.

Want to read the full book summary?

Leave a Reply

Your email address will not be published. Required fields are marked *

Fill out this field
Fill out this field
Please enter a valid email address.
You need to agree with the terms to proceed