MAMBA 2024-25
Im posting this from my google docs draft that was too long
I was curious how it would hold up compared to last time in the win projections contest, but also in terms of practically how it did versus vegas, in real world testing.
For a reminder of what I did to measure its accuracy in the “real world”, i came up with a super simple win projections from it and posted it to the APBR win projections contest before the season. This was a very simple projection as I really wanted to just test the metric, no aging curves, no preseason adjustment, and it was quite literally only lm(cumalativeMamba ~ Wins, data = mambaData). Along with not including aging curves (so my projections would not take into account if players improve or decline, so it assumed year 2 wemby = year 1 wemby), I also learned after entering many people in the contest (and in general for these projections) regress results to 41 wins somewhat, especially some of the metric projections in the contest which likely artificially inflated their results when I last posted (This only applies to a few entries in the contest. With the trade deadline about to end I wanted to see where it was at, at this point, as obviously i wound’t incorporate things like possible trades
It was joint tied for first in win projections last time, and since then has done better. It has gone from tied for first, to basically in a tier of its own in first place. The gap in accuracy (Measured by MAE) between this and 2nd place, is equal to the gap between 2nd place and 10th place. 2nd place is, by the way, an average of all the predictions rather than an actual entry. It is still outpacing the multiyear proprietary predictive versions of LEBRON and EPM from BBI as well in overall projections, although they both also beat out vegas and all three had similar success against the spread
In terms of practicality, the projections right now are 21-9 against the vegas spread. If i had incorporated aging curves, it likely would better than this (and it also bet the “Over” for the mavericks and 76ers, who have had horrible injury luck so far). I was able to see DARKO’s projections vs the spread as well, DARKO is the leading all in one solely projective metric in the NBA, and it is 16-14 against the spread. Its average error is slightly worse and RMSE is better (as results are more regressed to 41 wins in most projections than mine given the nature of how I did it). Note that DARKO has preseason adjustments built in with in, aging curves, and complex projected data used as well. (I want to look into doing the last one at some point)
Since Synergy data now is private and I have to use the NBA API for it, a few very low volume players might have somewhat inaccurate priors. By end of the season most people like this are likely under the “replacement level” minute mark cutoff, but it is pretty annoying for midseason projections, so a few strange offensive players might be high up.
It isn’t necessarily inconsequential to get them out midseason as since RAPM type data is way more noisy i personally think the weight you put on the prior and things like that are likely different or should be fluid. I was frankly just trying to get it out in a day, and will likely maintain it weekly and try to edit it along as I go, but while I thought the results looked off at first I saw there was just similarity to LEBRON and EPM in some players being rated higher or lower than expected, but as a whole I definitely would say the whole year is more reliable than any midseason data for these kinds of metrics in general
Despite it performing extremely well compared to other public leading metrics and vegas consistently and doing even better than expected in real world testing, and the results (or at least by the season end) looking generally coherent for something that tests well, I still consider this more of a proof of concept as the “RAPM” portion of the model was pretty vanilla compared to some of the fancy stuff some people include like rubber band adjustments and all of that, although im not sure if that necessarily improves it in the context of an all in one or if I will implement that or just make a online RAPM tool or something instead. Despite it ranking really well I think its an open question whether it is overrating certain archetypes or how it ranks role players in general compared to top end guys, but my main goal was to see if I could make a genuinely good all in one metric that could outdo the public ones and overall the results were quite good! Depending on what happens the next few weeks I’ll probably transition to more projective data stuff or tools with regards to basketball instead, as honestly even though I think this is at the very least on par (or the full season versions at least) with the metrics on the public right now, I think within the context of creating a real “ultimate impact metric” you probably need a really multilayered approach versus a one size fits all one that the ones in the sphere use right now, and i just don’t have that data with me
Also a strange comment someone told me about:
So before, when I reworked the metric, I said I did it to improve and because I saw some issues in the metric (I think). This is still true, I do not think the original version was good, at least compared to the current one, and had some pretty blatant misses here and there and the Prior itself was not as good. Then I got sent this message
“> The creator of MAMBA specifically said he thought the original version underrated LeBron’s post-Miami defense, and then created a revised version that improved LeBron’s standing, while noting that changes were made because LeBron “w[as] underrated in the Prior” (albeit that particular comment came specifically in a bullet point about offense, but it certainly further indicates the sentiment about how the prior was revised). Improving LeBron’s standing of course wasn’t the only reason changes were made, but I don’t think anyone could fairly read through both lengthy write ups the creator made and fail to come to the conclusion that improving LeBron’s standing was a major purpose of the revision. There’s no particular reason to prefer new MAMBA for these purposes compared to old MAMBA, except that the creator happened to like the output more (which, again, was obviously in part a result of it placing LeBron higher).”
Just wanted to respond to this because, I mean, it’s just a bit peculiar and kind of an annoying accusation? I guess I’ll go point by point:
The creator of MAMBA specifically said he thought the original version underrated LeBron’s post-Miami defense, and then created a revised version that improved LeBron’s standing, while noting that changes were made because LeBron “w[as] underrated in the Prior”
I just don’t really know how to react to this just because the idea that I “revised MAMBA because it clearly underrated Post-Miami Lebron’s Defense” is a bit of a peculiar point, I don’t really know how to respond to that because I don’t recall ever saying that? this message might be referring to how I thought all in one metrics like LEBRON generally underrated Lebron’s defense, and generally led me to my general criticisms of all in ones as players have variable non-box score impact. This was a general thought for multiple players, but I brought up Lebron becuase that was the conversation I had with someone that inspired the metric, and because like in the context of a write up using the name Lebron hits harder than Caruso
while noting that changes were made because LeBron “w[as] underrated in the Prior” (albeit that particular comment came specifically in a bullet point about offense, but it certainly further indicates the sentiment about how the prior was revised). Improving LeBron’s standing of course wasn’t the only reason changes were made, but I don’t think anyone could fairly read through both lengthy write ups the creator made and fail to come to the conclusion that improving LeBron’s standing was a major purpose of the revision. There’s no particular reason to prefer new MAMBA for these purposes compared to old MAMBA, except that the creator happened to like the output more (which, again, was obviously in part a result of it placing LeBron higher).”
This is just very peculiar, but I guess I’ll try to respond to it. The only real change I made on offense that impacted Lebron was an adjustment of how I used synergy data and adding in transition points over expectation, which ended up causing a huge boost in terms of accuracy. Outside of that, I mainly fixed some minor mistakes I had in the prior when compiling the box score data, and then did what EPM did in terms of setting certain limits to some data that could be extremely high or low based upon noise.
I mean, I think the name Lebron when referring the the player came up once outside of when I read out the results where I would be like: Yeah well now the guy that everyone knew was the best player in the world is now higher than George Hill so, i mean, its just a bit of a weird accusation
So to make something clear, I adjusted how I tested the metrics in terms of how I dealt with replacement players, so the testing metrics are different between posts (as you can see with EPM for example when you look at it). I had the Old MAMBA data when I was testing the projections, and the improvement between the old MAMBA data and the new MAMBA data was roughly the same as how much better old MAMBA tested compared to EPM when I did the projections the old way.
But I mean, I guess my response to this would be
I had issue in data and in the Priors, I fixed those issues, the metric tested much better, Lebron did better in a better metric.
My understanding is this was in some sort of argument about players, and its pretty funny seeing a number I made be brought up there when I have realized im just not really someone that ever markets anything so I kind of just have this as a “hey this is a cool thing I did” type of thing that “woah I made this crazy metric that beats vegas and everything” or whatever.
But like, the idea that I made a metric and then revised it to solely try to push an agenda for Lebron is kind of absurd? There were a ton of different weightings for priors and different priors I tried out, and there were certainly a large portion of them that tested very well, and better than the old one, where Lebron was 1st most years from 2015-2020, but for obvious reasons I picked the one that tested the best. Furthermore, I made this a RS only metric, which would be strange to push an agenda for a player who blatantly coasted in that timeframe for literally the entire time period this data covers. Honestly I have to think anyone that has tried running RAPM or general all in one metrics have a hard time imagining them being used as a case against Lebron, because his dominance in terms of on court impact really becomes more apparent when you try to make one of these things.
I recall I was curious to see how “playoff mode” bron looked in this type of data from 2015-2017, so I tried compiling that data and it truly was completely ludicrous, I don’t remember. It was fairly noisy, but I remember the constant was Lebron was something insane like a +14 or +15 in a data set where no one generally exceeded +12, so that was crazy, obviously playoff data is too noisy to post an all in one playoff metric or something like that, but him being that far ahead of everyone in the dataset by a country mile was pretty funny, most people’s results had been someone muted compared to their regular season results.
My reasoning for creating an all in one was I thought my approach could help somewhat offset the bias you get from priors in all in one metrics. I think it did so a little bit but certainly not completely and I think in hindsight a multilayered approach might make more sense than this, but it definitely proved to me that the bias there exists. It was somewhat shocking to see how much player’s would move the less I weighted the box score (and thus weighted box score impact more). You genuinely could get to points where Lebron was just first place all the way through the entire warriors cavs era aside from 2018 instead of Steph being the highest one in general, without even including playoffs, with a still reasonable weight on the box score but just relatively more weight than normal on on court impact for an all in one metric. Based upon it being such a consistent trend and the sheer magnitude of it, he’s very clearly someone underrated by this kind of thing. With how much of Curry’s impact comes from outside of the box score, I expected him to be somewhat underrated by the box score priors relative to his impact, but his priors were consistently fantastic with the ones I made even in years his raw RAPM impact were not as strong, so that was very interesting, to see he didn’t have that issue of the model potentially underrated him the same way.
But yeah, pretty weird message I got from someone, about me there. Not what I expected to be emailed, wonder where it was from.
https://timothywij123.shinyapps.io/MAMBAData/
