This post is part of an ongoing series of research updates by Bambu, a financial technology company I started in 2016. The theme of these posts is predicting the future. You can find part one here, sharing the results of a survey we ran about people’s attitude towards their future.
So, you think I’m kidding about predicting the future? Just hear me out. Predicting the future is not only possible but even simple if you stack the probabilities on your side by making precise statements about the object and time of the prediction.
Example 1: I predict everyone alive today will die. Certainly, there’s a non-zero chance I’m wrong, but historically that seems a pretty safe bet.
Example 2: Similarly, I can predict that for the next two seconds, you will continue to read this article, or at least finish this sentence. Got you!
So clearly, you can predict many things as a party trick by picking the right granularity of events and time horizon for the prediction.
The question is, where is the line between what’s defensible mathematically and what’s new and valuable information? If you’re too conservative, you end up with tautologies, i.e. statements that are true but add no value or information besides a tired chuckle from the audience. If you’re too aggressive, then you’ll end up with highly interesting information that simply has no connection to reality, or at best is just a coin toss, and get dismissed as a charlatan.
Is there a sweet spot in between? Well, that’s what we’re going to find out! To help us digest and compare various approaches, I’m introducing a simple chart that splits into four quadrants based on the aforementioned concepts of utility and relevance.
Historical approaches to predicting the future
It should come as no surprise to you that we’ve been doing this as far as history can shine a light. Whether reading weather, entrails, stars, or palms, man has always sought to interpret what nature has in store for us. Why? Well, nature has always been a cruel master, and perhaps hope placed in omens is better than no hope at all. On the other hand, the randomness of events around us has always been tough to accept. We chose to believe in divine intervention or deterministic fate to fill gaps in our understanding of natural phenomena. Either way, when we didn’t yet have things down to a science, we simply made up weird and random ways to predict until new science came along to show us the way forward.
Shamans
Basis: wisdom, intuition, subconscious
Content: high utility, low relevance
Possibly the oldest form of predicting the future is the tribal Shaman. Historical sources from archaeology show that this was a common practice amongst many ancient cultures, from the Native American “Medicine Man” to the Rishis of prehistoric India. These cultures relied heavily on the priest class, or even individual members of the tribe, that often had secretive rituals to access the ancestors’ wisdom.
An illustration from 17th century Russia showing a village Shaman in action. Source: Wikipedia.
Often these rituals would include mental practices such as meditation, ascetic practices such as fasting, and even medication in the form of hallucinogens derived from herbs or mushrooms. Given recent research into meditation and psychedelics and their applications in treating various psychological disorders, it doesn’t seem too far-fetched to say these shamans may have accessed genuine latent information through altered states of consciousness.
The problem with these predictions is a high degree of subjective interpretation; it left a lot to the imagination, whether reading pig entrails or describing psychedelic dreams. Thus life-and-death questions about when to pick the year’s harvest or whether to attack a neighbouring tribe were based on somewhat suspect information, resulting in a morbid and fateful toss of the coin.
Oracles
Basis: wisdom, intuition, history
Content: high utility, medium relevance
One might think of the Oracle as a more refined, institutionalized version of the Shaman. While the term originates from Ancient Greece, we might include the Egyptian priests in this class, given their similarities. While very little is documented for us to know about their specific practices and methodology, there are some second-hand accounts in Greek literature. What is clear is that the interpretation is still a part of the show, but the expectations have evolved from outright miracles to sage advice to the rulers of the day. In that regard, Oracles could be seen as a precursor to the specialist advisory teams that surround modern-day world leaders, just with a bit of extra fanfare and mystique.
Astrology
Basis: positions of planets and stars at specific times
Content: medium utility, no relevance
I won’t spend much time here, rather than point out that the futility of “reading the stars” is well refuted by science. Frankly, the existence of twins without identical life paths is already a big problem for the methodology. Besides, astrology already scores lower on utility, as it’s relegated more to gossip-column level speculation about your love life.
Astrological birth chart for Emperor Nero of Rome. Source: Wikimedia
Fortune tellers
Basis: psychology, common sense
Content: medium utility, low relevance
The only difference between fortune-telling and astrology is that fortune-telling does have some scientific basis. No, it’s not your palm or the cards. Fundamentally, the fortune teller is reading you as a person. How you respond to questions, your choice of words, emotional state, body language, and appearance.
The fortune-teller has no new information to give, but they can tease out subtle truths that you refuse to acknowledge due to your own biases and emotional charge. So in some sense, using some homespun psychology and simple common sense, a good fortune teller could be more beneficial than a bad psychiatrist! Then again, it’s equally possible the fortune teller is specifically taking advantage of your biases to tell you precisely what you want to hear, reinforcing bad decisions. So maybe don’t cancel your shrink just yet.
Current approaches to predicting the future
Moving forward from things that don’t work, we arrive in the domain of modern science and mathematics. Let’s examine what can be done with the best of today’s technologies?
The Human Brain
Basis: sensory data, memory, hormones
Content: low utility, high relevance
I didn’t see this one coming, did you? Well, as it turns out, none of us would be around if we weren’t pretty good at predicting the future. Let me explain. There are several ways to predict the future inside our brain that we know of, at least.
Firstly, think of catching a ball with your hand. The only reason you can see it is that your eyes and hands collaborate, facilitated by the brain, to predict where the ball will be at a specific moment in time. We forget how hard such a thing is until we have to create it ourselves with robots. Robots suck at picking items up, let alone playing catch with dad.
One such mechanism in our brain that allows this is called anticipatory timing. Without it, you couldn’t walk because you need to (mostly subconsciously) predict a complicated series of events to know how body mechanics, friction, and gravity will impact whether you slip or faceplant. This explains why babies do many falling and face planting, as they refine their motor skills algorithms just like a miniature robot.
Source: Darici, O., Temeltas, H. & Kuo, A.D. Anticipatory Control of Momentum for Bipedal Walking on Uneven Terrain. Sci Rep 10, 540 (2020). https://doi.org/10.1038/s41598-019-57156-6
The second kind of prediction feels pretty useless, almost harmful sometimes. We can imagine potential scenarios or ruminate on them endlessly lying in bed at night. What happens if you miss the bus in the morning? Why did I drop the glass earlier? What could have been if you went on that second date after all? We can conjure up entire lifetimes of complicated events with real or imaginary people in them.
Other types of prediction happen in our brains, such as how our eyes and brain interpret the world around us based on what we want to see. The prediction seems to be a fundamental capability, and we may have only scratched the surface of what’s going on with things like consciousness. I’ve covered that whole chestnut before, so bookmark that post for later.
CIA “Super Forecasters”
Basis: data, intuition, common sense
Content: high utility, medium relevance
Following on from our ability to rationalize and simulate different scenarios in our mind, it’s logical to think that there might be certain innate abilities or skill development that determine how good or bad one is at this type of thinking. In fact, in 2011, the CIA tried to answer that very question. They set up a competition of sorts to pick out people from different walks of life against their best analysts.
Lots of models. To set the scene for context, what an intelligence analyst would do to predict outcomes like presidential voting in Venezuela, or chance of war in the Middle East next year, is build models. The models would be based largely on existing data, either public or extracted by the CIA themselves, and try to extract various statistical metrics, relying on rational hypotheses, to judge the overall probability of specific outcomes. The problem is that there’s a lot of human judgement in that process in terms of picking good reference data, the treatment of data, framing hypotheses, and the metrics used to extract predictions.
Shockingly at the time, the hobbyist “Super Forecasters” beat the analysts. It wasn’t even close. There’s quite a lot of good material and podcasts on the topic if interested, but the short of it is this. These people had much simpler models than the analysts, who had the best tech and large teams of mathematicians.
Instead, the hobbyists had better intuition about choosing good reference data, and most importantly, they were focused on the art of being a good forecaster. In some sense, they weren’t overthinking the problems, but we’re very focused on a consistent process honing their prediction skills, which frequently resulted in better results. Mind you; we aren’t talking 90% accuracy. More like going from 51% to 55%, so overall, just marginally better than tossing a coin!
More recently, we’ve seen the very same phenomenon as hobbyists have regularly outscored government agencies in predicting the spread of COVID-19.
Insurance Actuaries
Basis: data, logic, common sense
Content: medium utility, high relevance
Isn’t it peculiar that the most uninteresting industry is probably the best at predicting the future? That seems like an incredible ability, possibly justifying the use of a cape. Well, the issue is that the scope of those predictions is extremely narrow. By definition, insurers only focus on things that drive risk, because they have to foot the bills if they get it wrong. Hence this work is carried out in cubicles, not secret underground lairs.
Here’s an example of how actuarial tables can predict whether people of various ages (x-axis) will live another 5 to 25 years. It’s interesting to note how the curves slow down if you make it past 75. Source: Wikipedia
While the obvious datasets relate to things like health and longevity that determine how much we pay for health and life insurance premiums, the origin story of these predictions was ensuring ocean voyages for ships. Sadly, it was quite common for ships not to return, and so money was put up almost as a bet against success, which would then be paid out to widows of sailors and the owner of the ship. Nowadays, more complex models can calculate probabilities for your iPhone screen cracking or your flight gets delayed.
Fundamentally, the probability is a prediction. Insurers bet your life on those odds.
Hedge Fund Quants
Basis: data, logic, common sense
Content: medium utility, medium relevance
Besides insurers, hedge funds also make money with predictions. That’s the only thing they care about. They purely focus on maximizing investment returns, by any means necessary, while mostly staying within legal and regulatory limitations. As far as we can know, that is. The methods vary, but fundamentally these companies are trying to find an information edge.
For example, hedge funds were paying for satellite imagery to calculate precise crop yields and count cars in parking lots of shopping malls. Why? Because they wanted to know information on how various commodities and businesses were performing before that data was published in the next quarterly reports. If you knew in advance, you could make a high probability bet on the asset going up or down when that data became public. Yes, it’s effectively legalized cheating.
While making lots of money is useful, that is a pretty narrow use-case. Knowing the price of copper tomorrow won’t help with your life problems; other than that, you’ll have more money to spend on worrying about them.
Artificial Intelligence
Basis: data alone
Content: medium utility, high relevance
If you weren’t aware, the Artificial Neural Networks powering most modern A.I. solutions were directly inspired by the brain. While we don’t understand the brain fully or particularly well, we’ve had good progress in using a few basic building blocks to train computers in learning useful tasks. This began with identifying handwriting in the nineties and has evolved into beating humans at games like Chess and Go and driving autonomous cars in urban traffic.
Much like the human brain we covered earlier, some applications of decision making involve the ability to predict the future. Driving cars is pretty self-explanatory in that sense. Firstly, the A.I. needs to know what various car systems are doing every millisecond and how they will coordinate something like an emergency braking situation on an icy road, should that be necessary. But even more complicated is the requirement to predict what humans will do. Remember, other cars are being driven by erratic, emotional hairless apes, us. Not to mention pedestrians doing random things like jaywalking. Don’t get me started on cyclists.
No wonder Tesla’s Autopilot is one of the most powerful A.I. systems on the planet. It comes with 8 cameras giving it a complete 360-degree view, plus sonar. Yes, like whales. The computer hardware running their neural nets is also now built by Tesla from scratch. Most importantly, though, is the data. That explains why they continue to dominate the category, as no one else has a decade’s worth of on-road data from thousands of cars across billions of miles. No data, no A.I., no self-driving. It’s that simple. Read more about all that stuff here.
Board games are another good example because a key part of successful gameplay is to know what your opponent can and will do. Given the latest generation, AlphaZero has a 99% win rate in both Chess and Go against humans; I’d say they’ve got us figured out pretty good. That’s how big of a deal it was when Deepmind crushed humanity’s last hope in Lee Sedol in 2016. Before it happened, most experts thought it might never happen.
Visualizations of the decision-making processes of AlphaGo in choosing the next move (orange circle) with numbers indicating overall win probabilities. Source: Silver, David et al. (2016). Mastering the game of Go with deep neural networks and tree search. Nature. 529. 484–489. 10.1038/nature16961.
The thing to note here is that A.I. models are purely and entirely based on the data. So it matters what data is used. Random data will result in no learning, and biased data will result in biased decision-making.
Academic approaches to predicting the future
While the current approaches all have various real-world applications, several fields of academic study examine human populations to understand them. In some sense, the highest level of understanding is predicting because that gives you a model that functionally represents the data. Seeing is believing, after all.
Social Studies
Basis: data, logic, common sense
Content: high utility, high relevance
I never studied the humanities, so I’ll be brief here. Not only have we studied ourselves in fields like psychology and biology, but there are whole fields of study around human behaviour in groups large and small. At the same time, we all think of ourselves as highly individualistic, with free will (*cough*) to do as we please, the reality we also admit to is that routines largely dictate our lives. We all eat and sleep, and some of our work is in between those mandatory activities.
When we look at populations of hundreds or thousands of people in communities, we can start to model, and therefore predict, how people spend their time and when they spend it. The specificity of those models increases when you group people by types of unemployment, say, shift workers. Pretty quickly, and somewhat unnervingly, I might be able to tell with high accuracy what you’re doing at any given time of day, between doing laundry and watching T.V. The only question is, is that useful to know or just creepy?
Population studies
Basis: data, logic, common sense
Content: high utility, high relevance
If we move up to a higher level of examination, we can look at how portions of the population behave over their lifetimes. It’s just another way to fix our parameters to make it feasible to predict: what are the questions and over what period.
For example, it will be impossible to predict if you will die tomorrow, or not accurately. As we discussed with insurers, there are probabilities for that kind of thing, but those are based on other people, not you specifically. Whereas, with those same models, I can pretty precisely predict which decade you will die in. Again, picking the proper battle is a big part of the challenge here.
Sociological research is interested in the mechanics and causality of how populations evolve. Think of questions such as what happens to kids from divorced parents; are they more likely to divorce themselves? What about kids from low-income families? How likely are they to go to college? This is where it gets interesting.
One of the methods to tease out these kinds of questions is fundamentally the same technique used under the hood of AlphaGo. Yes, we are playing the game of life with real people. So-called Hidden Markov Models take sequential datasets, such as moves on a board or events in real life, and model the relationships between them. These relationships are modelled in complex graphs, which give you the probabilities of the next move or next life event. Different states of the graph model different events that can happen each with a probability adding up to 100%. Between moves, which can be sequential in board games, but years in life course data, there is also a probability of moving to a different state.
With AlphaGo, such graphs represent the rules and flow of the board game. Some parts of the graph (much larger than above) might represent opening sequences or final moves to find a checkmate. With life-course data, the same graphs represent… the rules of life. How things play out. Who gets married, who gets divorced. Which one of the two has kids, which has none. Even things like who retires earlier, the married or the divorced, or those who stayed single. Most importantly, these events are not disconnected but form a continuum of life events that we can examine from start to finish, allowing us to predict forward from any given scenario. That seems pretty useful!
Global studies
Basis: data, logic, common sense
Content: high utility, high relevance
Another level up, and we’re looking at what happens on a global scale. Things like climate change come to mind. Techniques tend to be traditional data gathering and analysis, but as we know from climate change denialists, the science doesn’t often speak for itself.
Another more pressing example is the spread of COVID-19. Below is an animation put together by the World Economic Forum that shows the first three months of spread day-by-day. It highlights exponential growth dynamics, as in two weeks in March, cases ballooned everywhere at once like a bad case of the measles had attacked the map of the world.
More broadly, global predictions are very useful as evidence for policymaking but have relatively little to say about individuals.
Theoretical approaches to predicting the future
If we allow ourselves to take a few steps beyond what is already here, we can look at a few sci-fi inspired methods, including where A.I. could take us eventually. Because… why not? It might show us what’ll be possible soon, as it has many times before. Chances are, if you take any modern technology, someone thought and wrote about it fifty years ago.
Mathematics (“Foundation“)
Basis: data, logic, common sense
Content: high utility, high relevance
Suppose you’re not already familiar with Foundation. In that case, you will be, as a high-profile T.V. version streams soon on Apple T.V. It is considered one of the great sci-fi works of the 20th century, from the godfather of sci-fi Isaac Asimov. The premise of the original trilogy is a theory of “psychohistory”, which is a mathematical formulation of how humans behave at macro scales. Think planetary or galactic studies, rather than mere global studies—statistical biases and trends from trillions, not millions of people.
“Since emotions are few and reason many, the behavior or a crowd can be more easily predicted than the behavior of one person can. And that, in turn, means that if laws are to be developed that enable the current of history to be predicted, then one must deal with large populations, the larger, the better.”
― Isaac Asimov, Robots and Empire
One of the fun aspects of this approach is that it only holds if the predictions are not made public without giving away the plot. If people know the prediction, they will change their natural behaviour, thus affecting history.
While nothing like psychohistory exists as a serious field of study, our most recent pandemic is a wake-up call. A better model is needed to understand certain aspects of global dynamics. Such models could be used to enforce both more reasonable and effective strategies against the next virus. Also, fun fact, Elon has named Foundation as one of his most significant influences. He also seems to be doing things about the future. Maybe he’s figured out psychohistory alongside cryptocurrencies.
Quantum Mechanics (“Devs“)
Basis: data, logic, common sense
Content: high utility, high relevance
Few technologies have sparked the imagination, and utter confusion, of people like quantum mechanics. At least when Einstein rocked the foundations of science with his relativity theories, we could use analogies and images to make sense of it all. We can feel and see gravity all around us, giving us some valuable intuitions to start from. We can’t see any quantum events, though. If relativity was shocking, then quantum mechanics is just … strange. Even worse, while there are few open questions at the edges of relativity, we haven’t made any meaningful progress on core questions in quantum mechanics since…well, since it was conceived by Einstein, Bohr, and others nearly 100 years ago.
One of these core questions is whether the universe itself is fundamentally random or not. There are a few broad camps, starting with determinists who believe randomness is an illusion and everything can still be calculated precisely in a classical sense. We just need to know the underlying “wave function” that generates these illusions. Meaning if we have all the information about a system, we could use quantum mechanics to predict forward and play it back indefinitely.
Others believe there is fundamental randomness. The same mysterious wave function collapses seeming alternative paths into a singular choice, implying somehow that “god throws dice”, which Einstein didn’t like very much. Most bizarrely, there is a third, equally serious camp that believes the universe splits in two with every random event. That both things happen, but… in separate worlds. No doubt something Dr Strange will explain to us in his upcoming Marvel sequel.
(link
)
Here’s a short video that displays one of the only macroscopic visual experiments that show quantum mechanical properties, including isolation of a “pilot wave” that pushes the water droplet along the surface.
The T.V. show “Devs” does a pretty decent job playing around with these concepts, tempting us with the prospect of knowing all. One thing is clear, though. No scientist alive thinks it’s possible to collect ALL information about any system because that implies measurement which itself affects the system, as dictated by Heisenberg’s uncertainty principle. So the whole thing breaks apart before you can get to anything useful at macroscopic human scales. So don’t hold out on this one happening.
Superintelligence (” Westworld“)
Basis: data, logic, common sense
Content: high utility, high relevance
The last stop on this particular tour puts it all together. We know current A.I. systems can be incredibly accurate at predicting specialized limited scenarios. However, we do not yet know how to vastly expand those abilities into General Artificial Intelligence, which is more like a human brain works. We can learn one skill in one domain and naturally apply it across new skills and domains. Kids who are good at one sport tend to be good at all sports. Einstein famously used playing the violin as a way to engage deeply with abstract mathematical concepts. We don’t know how that works.
But what if we did? It would open doors we didn’t even know existed. Such as system capable of human-level thinking would achieve superhuman abilities in thinking. Even if you had exactly human brain functionality, you could still extend it with near-infinite computation. While that story could go in many directions, most quite extreme, the portrayal of such a superintelligence in Westworld’s season 3 was quite effective.
Again, without spoiling much, the A.I. had a lot of data on many things and much intelligence to ponder on it. The result was an ability to predict outcomes on both individual and global levels and therefore actively interject. A kind of manifest destiny powered by computation.
Today, the rapid ongoing advances in A.I. research promise to both deepen and broaden the scope of predictions that can be made on existing data. Most of the people doing that researchers say it’s only a matter of time before we approach true superintelligence, with many pointing to mere decades until it arrives. That means some of us will be around to watch the birth of something that might truly know our next move before we even think of it. Possibly all the other moves after that, too. That makes our future beyond such a point beyond prediction itself, as we hand the yoke of our destiny into the hands of a higher being. Read more about such a technological singularity here.
Can we predict the future, then?
So, where does all that leave us in terms of predicting the future? Well, if we look at the prime candidates, there is one thing in common. While the universe may never bow to quantum information or pure mathematics, probabilistic models based on lots of data already work in many scenarios. There’s little reason to assume that won’t continue to improve quickly, and the limits are yet unknown. That’s what we’re here for.
Simple problems: MAYBE
The past few years have seen an explosion in the A.I. community regarding the availability and open-sourcing of powerful models for a wide range of data and problems. Whether you’re working with time-series, image, video, or language data, there are tools you can try on your data. The deciding factor on whether you can predict accurately is going to be the quality of your data. This cannot be emphasized enough. Some math and coding skills are required for now. Leave the cutting edge stuff to the tech giants, though, as the entry price is going up with bigger models that require more extensive datasets and much bigger hardware. All of which are very expensive.
Individual life events: YES
Moving on from simple datasets, a different type of complexity is involved in structuring life events. Generally, this class of problems can be described as gameplay scenarios and solved with A.I. methods such as Hiden Markov Models and Reinforcement Learning developed to make humans obsolete as opponents in board games like Chess, Shogi, and Go. Some day, only Wall-E will get a driving license, and we’ll be passengers from there.
As with the previous category, it comes down to the availability of data. Not just any data, but specifically life course histories. Luckily, at least for developed countries, such data is compiled by academia and government in longitudinal studies following a large group of individuals over several decades. Developing countries, not so much.
Population trends: YES
What we can do for individuals applies even more for populations and is already widely used in academia, government, and policymaking. Further, insurance models apply very well at this level, where health and life outcomes can be predicted based on group characteristics. This is the basis for the prices we all pay for health and life insurance, regardless of where you live.
Global statistics: YES
While the weather continues to elude us, due to the chaotic nature of how climates react locally to moment-by-moment conditions, there is an increasing number of things we have a better grip on. Yes, COVID-19 wasn’t a good showing, but it also highlighted how much technology is at our disposal if appropriately used. You can be sure that we’ll be better prepared for the next one. Similarly, climate change models increase their predictive power, pointing out the decision boundaries policymakers will have to contend with in managing quality of life, productivity, and sustainability.
Stay tuned for the next one, where we’ll dive into individual life event predictions in the next post in much more detail, as this is the ideal basis for financial planning, the specific topic of interest for Bambu’s research.
The author, Aki Ranin is the founder of two startups based in Singapore. Bambu is a Fintech company that provides digital wealth solutions for financial services companies. Healthzilla is a health-tech company and creator of the Healthzilla health analytics app for iOS and Android smartphones.
