AI News, Why is machine learning in finance so hard?

Why is machine learning in finance so hard?

Financial markets have been one of the earliest adopters of machine learning (ML).

Even though ML has had enormous successes in predicting the market outcomes in the past, the recent advances in deep learning haven’t helped financial market predictions much.

Even though there are a number of papers claiming the successful application of deep learning models, I view those results with skepticism.

The issue of data distribution is crucial - almost all research papers doing financial predictions miss this point.

We expect the distribution of pixel weights in the training set for the dog class to be similar to the distribution in the test set for the dog class.

In addition to making sure the test and train sets have similar distributions, you also have to make sure the trained model is used in production only when the future data adheres to the train/validation distribution.

While most researchers have been mindful not to incorporate look-ahead bias into their research, almost everyone fails to acknowledge the issue of evolving data distributions.

For example, even if we have a complete understanding of what happened during the great depression of the 1930s, it’s hard to convert it to a form that makes it usable for an automated learning process.

(Please note that mixture of experts is a very common technique to combine the models from the same scale - almost all quant asset management firms employ this technique.) I

If there is one thing you take away from this post, let it be this: Financial time-series is a partial information game (POMDP) that’s really hard even for humans - we shouldn’t expect machines and algorithms to suddenly surpass human ability there.

What these algorithms are good at is the ability to unemotionally spot a hardcoded pattern and act on it - this unemotionality is a double-edged sword though - sometimes it helps and other times it doesn’t.

Entity Relationship Diagram (ERD) Training Video

Stanford Certificate - Data, Models and Optimization

Preview Stanford's online graduate certificate: Data, Models and Optimization Info: ...

Stanford Seminar - Topological Data Analysis: How Ayasdi used TDA to Solve Complex Problems

"Topological Data Analysis: How Ayasdi used TDA to Solve Complex Problems" -Anthony Bak, Ayasdi Colloquium on Computer Systems Seminar Series ...

How can Digital Agriculture Feed Nine Billion People | Jim Ethington | TEDxUCDavisSF

Jim shows us how to use big data to help solve world hunger problems. Jim Ethington technologist and entrepreneur with 15 years of experience building data ...

Predicting the Winning Team with Machine Learning

Can we predict the outcome of a football game given a dataset of past games? That's the question that we'll answer in this episode by using the scikit-learn ...

Professor Gunnar Carlsson Introduces Topological Data Analysis

An Introduction to Topological Data Analysis by Ayasdi's Gunnar Carlsson.

Panel Data Models with Individual and Time Fixed Effects

An introduction to basic panel data econometrics. Also watch my video on "Fixed Effects vs Random Effects". As always, I am using R for data analysis, which is ...

Choosing which statistical test to use - statistics help

Seven different statistical tests and a process by which you can decide which to use. If this video helps you, please donate by clicking on: ...

Markov Chains - Part 1

Thanks to all of you who support me on Patreon. You da real mvps! $1 per month helps!! :) !! Part 2: ..

Statistics with R (1) - Linear regression

In this video, I show how to use R to fit a linear regression model using the lm() command. I also introduce how to plot the regression line and the overall ...