Holy grail or poisoned chalice?
TL;DR Nobody has cracked it. Period.*
Machine Learning has always fueled the fantasies of Wall Street. After all, AI detects faces, drives cars, beats the World Champions at Chess, Go, and now Starcraft 2. Its application to trading seems natural, doesn’t it?
Totally! What if we could throw a treasure trove of data at an AI and let it predict future market prices?
Don’t bother with that…
Prices cannot be predicted, they are mostly random. They are not predictable on average, only on occasions but nobody knows when.
The era of the magic ‘black box’ automating alpha hasn’t come yet.
Nobody has cracked automated trading using Machine Learning-based predictions*.
That’s right. Sorry to disappoint. Nobody knows whether their bot will behave well next month.
Wait a minute… there must be someone Einsteinly smart out there who has figured it out.
How about the Renaissance and Two Sigmas’ of the world? With their Billions of AUM and their armies of PhDs, they must have sealed the deal.
There is no real evidence they have. Those guys have made a habit of keeping things secret, letting outsiders speculate. That is their brand after all, and in the game of FOMO, that’s a pretty lucrative move.
Don’t get me wrong, quant trading strategies exist, ranging from arbitrage to high-frequency trading, they manage north of $1.5 trillion worth of assets according to Morgan Stanley. But they remain relatively simple in the grand scheme of things. They exploit sequences of predictable behaviors and biases.
There’s still no evidence that we’ve reached the algorithmic trading singularity, with an end-to-end automated black-box consistently generating alpha.
And this is why funds experimenting with more complex strategies spend fortunes on execution and safety measures to protect their back: cross-signal confirmations, alerts, stop-losses, crash-recoveries, roll-backs…
Dad, I have a good feeling about that one.
You’re telling me that this thing is the ultimate home-run, and nobody has ever cracked it… I’m in!
I know what you’re thinking. What if you were the exception, the missing link, the chosen one?
Right! Someone must fulfill the prophecy, and bring the balance in the force.
Easy, Anakin… Before you get all carried away, let’s talk about the most common traps we all fall into.
1–The answer is 42
Have you ever read Hitchhiker’s Guide to the Galaxy?
So good!
The ‘Answer to the Ultimate Question of Life, the Universe, and Everything’, calculated by an enormous supercomputer named Deep Thought over a period of 7.5 million years is… 42.
Hilarious! :’)
Finally, the one and only truth.
Right, and Deep Thought points out that the answer is meaningless because the beings who instructed it never knew what the Question actually was.
The funny part is that quants do it all the time. They set up GPUs and train complex algorithms for days to ask what the price will be tomorrow.
What’s wrong with that? You can’t feed-in historical prices to predict future ones?
Don’t forget, markets are unpredictable and you’re breaking one of the most essential rules of trading:
Past Performances are not indicative of Future Results.
Not convinced yet?
Here is a great reality-check for you with some price predictions of Bitcoin using an LSTM. The web is full of disillusioned traders attempting ML-based price predictions.
Ouch! Party with backtest, hangover with live trading…
When something looks too good to be true, it probably is.
Now, let’s discuss the second most frequent mistake quants make.
2-The Rube Goldberg Syndrome
Have you heard of the Rube Goldberg Machine?
The machine that uses a complex chain reaction of events to achieve a final, trivial goal?
Correct. Or metaphorically, the tale of an over-engineered useless machine.
Quant: “I think we’re getting really close!”
Come here… Can you read me out loud my notes from that year-long experiment I told you about?
Sure. Let’s see…
“Normalizing data. Check.
Eliminating noise. Done.
Reducing overfitting… I almost lost my sleep over it, but I think we are in a pretty good shape.
Using gradient descent to find global minima. Eh eh! that’s it… I’m a genius.
…
Wait a minute… Why is the live agent still not behaving like in backtest?
Either I’m going mad, or I’m missing something.”
We dug really deep but never ended up finding the gold mine.
Could you have used more features, run a better data cleaning, or further tweaked hyper-parameters?
Smart suggestions… but you’re now catching the Rube Goldberg syndrome!
Eventually, it became clear that we were approaching it from the wrong angle.
Your machine will deliver on what it is designed to do. You’re not trying to land a rover on the moon, so no need to build a rocket ship.
As often in life, less is more.
Last but not least, let’s unwrap the sneakiest misunderstanding that tricks us all.
3-’Buy low, Sell high’ they say
Aaah, I know that one! Must be the single most famous adage in investing.
Well, this law is misleading at best in algorithmic terms. Lows and highs only become clear in retrospect, and what looks high one day may look low another day.
And we, humans, are good at exercising ‘common sense’. We employ our judgment in universal ways without thinking expansively or requiring large data sets. Machines are in their relative infancy in this field.
Take a market price for example.
Right, it’s just a number.
The number is a shell. But underneath the shell is a complex derivative of the underlying business, its capital structure, its macro-economic fundamentals, as well as human emotions and buyers/sellers’ intentions.
So the only way for a machine to precisely predict the market price, you would need to feed all those elements that could potentially affect the price. Which is practically impossible to obtain and train an algorithm on.
Gotcha! And if you only input price history, there is a whole lot of information your algo is missing about the underlying factors that affect the price.
Price is the consequence of everything that is happening on the market, not the cause.
Alright, Morpheus, you’ve enlightened me. I won’t take the blue pill.
But what are your suggestions to trade using AI?
1-Ask The Right Questions
It becomes important to ask questions differently. Instead of asking “what will the price be” knowing that price movements are random walks, ask questions like:
- What other sources of independent information evidently drive market dynamics?
- What mix of indicators, trends, and past events most accurately generate the right signals?
I see! You need to operate a mental shift and focus on the factors that drive proven wins.
Exactly. The key resides in developing empirical evidence from correlations between data events and the corresponding market responses, then ask the machine learning model to find patterns in the data that precede that trade.
Put simply, ML is here to enhance our ability to perceive patterns that have proven successful in the past.
You have to set the game as a ‘winnable’ game by taking advantage of demonstrated correlations.
To detect real correlations you need to be proficient in many independent disciplines. Let’s uncover how you can develop these skills.
2-Develop Transverse Thinking
Have you ever watched a decathlon race?
Run, Throw, Jump — Decathletes are considered the ‘World’s greatest athletes’.
From what I read, decathletes don’t have to be the World’s bests at any particular discipline but consistent across ten of them.
Precisely.
In algorithmic trading, you need to be technically savvy, develop a battle-tested understanding of Machine Learning, decipher the odd nature of time series data, and grasp the fundamentals of markets, finance and trading.
Makes sense. The well-balanced quants with the ability to think transversally will take the prize home.
And until AI reaches higher-intelligence and comprehends every humans’ intricacies, you will have to meet it half-way and understand the machines’ logic.
That’s good news! That combination of skills must be really scarce. I can see how it’s only developed organically.
With a brand new discipline come brand new opportunities.
Unfortunately, developing transverse thinking to spot the right market correlations isn’t enough. There’s an external factor that is unavoidable in the world of innovation and that you need on your side: timing.
3-Timing is Everything
Did you hear about the recently published Five stages of Autonomy of Self-driving cars?
Fascinating stuff! Level 5 autonomy is where drivers simply plug in their destination and leave the rest up to the vehicle itself.
In the realm of autonomous trading, we can realistically estimate that trade execution has reached Level 3 to 4 while generating profitable and reliable trade signals remains between Level 1 and 2.
Let’s face it, the fully automated trading engine is still far off. We still don’t have a clue when the inflection point will be?
At the current point in time, Humans and machines complement each other, excelling at different types of skills. Human Augmentation is what’s here and now.
“Someday, yes, we will have flying cars. It does not mean that we need to be making flying parking lots at this moment.” — Scott Belsky
Conclusion
Did you notice anything peculiar about our dialogue?
Hmm…
I used a series of metaphors, illustrating complex concepts with simple analogies. Why is that?
Easier to read? :)
Because it talks directly to the human brain’s core strength. Building bridges between the known and the unknown through associations. Learning something new rarely starts from a blank sheet. Neurons have to unwire in order to re-wire to update our brain’s chart. Our very own neural network is a living map of experience-based rules (be it conscious or unconscious).
Same goes for ML-based trading, I assume.
You cannot start from a blank sheet, hoping the neural net will come up with something logical.
Spot on!
You have to start from a map of simple, evidence-based rules that have been carefully crafted and have withstood the test of time.
That’s 90% of the work, right there.
And then chug-in massive amounts of data into the ML system and let it evaluate, weigh, and toss out information to come up with Alpha!
*’Nobody has cracked AI-based trading’: by now you’ll have understood that this initial assertion was purposely provocative. The reality is always more contrasted. There are lots of brilliant people exploring this field and their work shouldn’t be under-appreciated. Some projects reveal promising results, but quants remain the #1 responsible for profitable signals. AI comes second.
I hope this article contributed to demystifying AI-based trading and re-aligning our short to mid-term expectations with the brutal and unpredictable reality of markets.
If you found it helpful, please clap (up to 50 times) and share to get it in front of smart people like yourself.
If you are developing your own models, seek external expertise to guide you in the right direction, or simply want to nerd-it-out on quant strategies, come say hi! yoann@gosupernova.co.
At Supernova, we also offer the following Service Packages to actively managed crypto funds: automated rebalancing, backtesting, alpha creation, and automated execution. Get in touch at services@gosupernova.co.