@lysergic said:

This is all fabulous. Thanks for doing this. Some scattered thoughts:

1. Dropping raters with exceptionally high deviation scores will mechanically increase R2 and decrease RMSE, so I would be careful to only use those metrics when comparing weighting schemes that treat high-deviators the same.

I've been using a combination of deviation, entropy, and # of shows rated, which seemed most closely aligned with the RYM system. Entropy is the metric that drives most of the weights, followed by deviation and then # of shows. Typically, anomalous raters are identified by more than one metric, i.e., about half of all people identified with deviation also have low (zero, or near zero) entropy. But your point is well-taken.

3. I would be curious to see a graph of avg rating by year, comparing the different weighting schemes. Which years win? Which lose? (Ideally extend back to 1.0)

I haven't done this for every year, but here's a couple of things that aim in that direction. For all of the weighted averages (two of which are depicted below), the distribution remains left-skewed, but is squished from the top. There's a bit more mass in each tail, and the overall mean rating (the mean for all shows) actually increases. This graph includes shows from all eras.



As for the year-by -year thing, I've done it for selected years--one from each era (Modern, 2.0., and 2 x 1.0). There was no real rhyme or reason as to which years were selected; mostly they seemed like interesting years, and I had the Phish Studies Conference deadline bearing down on me.

4. You've probably thought of this, but a middle ground option between doing nothing and doing full-on real-time adjusted weights would be to generate the weights at regular intervals, e.g. monthly. (...any new accounts created in between weighting updates would get an initial weight of zero or close to zero.)

This is exactly how RYM handles its weights.