diagnosing ML models for noob modelers
all models are wrong, but some are hot
Welcome back friends.
Sometimes I have to fit models. And turns out, I'm quite shit at it. But don't worry; with this one trick (practice), you can get better at something. As a continuation of previous article about regime-robust model, I wanna write about ML models a bit more.
actually no I take it back. here I wasn’t doing much modelling — I was doing more ad-hoc stuffs for the team. now I spend 90% of my time fitting shitty models models at work I have 0 interest in fitting shitty models at home lmao I just wanna go back home and watch Netflix https://t.co/HdJ8qeJ81J
— qm (@quantymacro) January 29, 2025
So lately besides running my 30 Sharpe brypto strategy, I've been trying to learn how to fit ML models better. Throughout my practice I stumbled upon a few questions; why do people love SHAP values so much? Why when I read stuffs online, there seems to be 100x more focus on explaining predictions of ML model, than diagnosing the errors of ML models, which arguably is just as important? How do we spot non-linearity in errors? At the end I came up with (which I think are) simple ways that I find intuitive to help me fit better models, and I hope it's useful for some of you too. Let's get it.
Important to note that this article (like all of my articles), are not a guide on how to do things. Rather it's just my flawed and unrefined thoughts of a topic that I choose to talk about. Treat it like an exploration of my thoughts.