How we became psychic by harnessing the power of data

Did you know you have the power to tell the future? No really. It’s true! Data gives you the power to peer past the veil of the now and into the future of what could be, with a reasonable amount of predictable accuracy.

A little too metaphysical for you? Let me explain.

Back in 1654, two French mathematicians sent letters back and forth discussing what happens in a game of chance. They wanted to figure out the chances of winning a game in a best 2/3 scenario, and what rewards would be deemed fair(1). For instance, if you and a friend both had a score of 2:2 in a 5 turn game, should you be dividing the majority of the prize pot equally, with a small bonus for the winner in the final round?

If you felt you were more likely to win the final round, you might say “NO” as you’d want the entire pot. Scenarios of chance such as this, fascinated mathematicians throughout the ages, and still remain a staple to the fundamentals of statistics.

Understanding chances, or probabilities, is the backbone of flexing your psychic data muscles. The more you can wrap your head around the likelihood of something happening, the more accurately you can predict it.

Now, stepping back, initially statistics was used as a way to describe the world around us. Historical evidences suggests that statistics was born out of government. For example a 1794 German book, Statistik, described the analysis of demographic and economic data about the state(2). The Latin, statisticum collegium, refers to the “council of state”. Statistics was used in census-taking initiatives, to give a snapshot of the current community. Numbers described what was happening, hence the early uses for staticts were descriptive statistics.

Descriptive statistics are what we leverage the most to describe what happened in the past. For example, your phone’s pedometer tracks the number of steps you took in a day, and over time, can show you trends such as how often you walked in the past month, your most active days, how fast you went etc. All of these descriptions, are based on your past performance. Businesses leverage descriptive statistics in a variety of ways including campaign analysis, manufacturing performance, product KPI’s and more.

So how did we get to a state of prediction?

Well, as much as machine learning has been a hot topic of the past decade, predictive analysis has been alive and well for the past 75 years. Once again, government needs can be credited with bringing this tool to the mainstream(3). More accurately, war drove the need for predictive analysis. In a race to nuclear arms, and decoding German communications, knowing what was likely to come next was key to success. Innovations that came out of that time include(3):

• Linear programming: the foundation of everything we have today, used to predict the best possible outcome from a given set of parameters (Told you data can make you psychic.)
• Monte Carlo Simulation: variable estimation technique… this was used to predict atomic behaviour in a chain reaction
• Computational Modelling: using computers, math and physics to study the behaviour of complex systems. (Did someone say foundations of machine learning?!)

While the above are over-simplified descriptions, these definitions give us an understanding of the foundation required to build the super-psychic simulators of today. As society moved past the threats of WW2, we had a new problem to tackle. This is a classic question many business-savvy individuals have asked at one point or another How do we monetize this?

The answer: optimization.

In the 1950’s and 60’s humans looked to improve and optimize many things in their lives. One of the first to tackle: weather. In 1950 the ENIAC computer was able to generate the first models to forecast weather3. The forces of nature were not safe from being interpreted as 1’s and 0’s. Although the initial calculation took 24 hours to complete, ENIAC proved that with a faster machine, weather prediction was indeed possible(4).

From here, the biggest breakthrough was in the solving of the “shortest path” problem in 1956. This problem, still leveraged almost daily, was first applied to air travel and logistics, finding optimal travel routes(3). One of the most exciting applications (in my opinion) was in 1958 when FICO applied predictive analytics to credit risk. Think about it, today, our credit score is one of our most closely guarded, and individually influential numbers in first world society.

Looking for a loan? How about a home? Your credit score matters. As cool as it is to know about population growth, nothing impacts us on an individual basis quite like the credit score(5). Some employers even use your score as a measure of trustworthiness when deciding between job candidates! Before credit scoring, getting a loan was a lot more about who you knew. If you had the right relationships, you could get the money you required for a fair deal. This left many marginalized populations out of the credit market, as often, business banking was conducted based on “gut feel”(5). Today, credit scoring uses payment history, amounts owed, length of history, new credit, and credit mix6, to determine your likelihood of defaulting on a loan. Fast forward from the first scoring in 1958, to 1992, and in under 40 years, FICO was deploying real-time data analytics (thanks to the power of better computers) to fight credit card fraud(3).

Other highlights of the 70’s — 90’s included the foundation of the stock market, the Black Scholes model, model driven Decision Support Systems, and algorithmically designed online experiences via Amazon, eBay, and Google(3). Jobs, and entire industries, rest on the back of our ability to leverage data for predictions.

Today, in combination with access to more talent, a staggeringly exponential growth in computational power (see Moore’s law), and consumer demand for faster, better, stronger, and more personalized experiences humans rely on predictive analytics more than ever before.

Without predictive analytics not only would hot topics like my favourite — Big Data — not be on the table for discussion, but machine learning as an entire industry would cease to exist. No chat bots responding to customer inquiries. No driverless cars. No dynamic ticket pricing. No virtual (robot) influencers. No earthquake prediction. I could go on.

You see, we are able to predict SO MANY things and it is all thanks to statistics. Even though the first dabbles into machine learning occurred in 1952 (the first computer game that could learn while it ran… it played checkers, and was created by Arthur Samuel(6)), they would have not been possible without our realization of the power of patterns, prediction, and probability. They would not have been possible, without first realizing that with data- we are psychic.

— -

Emily is a self-proclaimed data nerd, passionate about using the insights we gain from data to produce incredible products and experiences. Follow her journey into the world of data, tech, product, and user/customer experience here on LinkedIn!

— -

Citations:

1. Theodore M. Porter Professor of History, University of California, Los Angeles. Author of The Rise of Statistical Thinking, 1820–1900; Trust in Numbers.
2. .A Brief History of Statistics (Selected Topics) ALPHA Seminar August 29, 2017, University of Iowa http://homepage.divms.uiowa.edu/~dzimmer/alphaseminar/Statistics-history.pdf
3. Dr. van Rijmenam, M., The History of Predictive Analytics Infographic, June 24 2013https://datafloq.com/read/history-predictive-analytics-infographic/438
4. Easterbrook, S. The first Numerical Weather Prediction on ENIAC, January 25 2011 https://www.easterbrook.ca/steve/2011/01/the-first-numerical-weather-prediction-on-eniac/#:~:text=The%20original%20forecast%20took%20about,useful%20weather%20prediction%20was%20possible.
5. Hill, A., A Brief History of the Credit Score, April 22 2014 https://www.marketplace.org/2014/04/22/brief-history-credit-score/
6. History of Machine Learning https://www.doc.ic.ac.uk/~jce317/history-machine-learning.html#:~:text=1952%20saw%20the%20first%20computer,network%20in%201958%2C%20called%20Perceptron.

Originally published at https://www.linkedin.com.