@Diptchip

https://en.wikipedia.org/wiki/Turtles_all_the_way_down

@Science

@freemo @Science That's great and all but gravity can be measured, while these turtles cannot.

@Diptchip

Yea the point here isnt that gravity is as valid as a world turtle. The point in my eyes is more so that for all the absurdity the ideas of old may have, and summarily dismissed on their absurdity. The real world has notions that appear just as absurd (albeit with stronger evidence however).

@Science

@freemo @Diptchip @Science
Love the parable.
I read something similar the other day.
https://lukesmith.xyz/articles/chess.html

@torresjrjr @freemo @Diptchip @Science Thought provoking in the best sense!

@torresjrjr @freemo @Diptchip @Science

Hmm.. I don't get the parable. Models' accuracies are evaluated not by comparing their average prediction with average result, but by something like comparing the "leftover surprise" the model leaves us with (KL divergence of the probability distribution of the world with respect to probability distribution that the model predicts). On that count, the model that models the intricacies of chess is obviously more accurate, and the question of whether the increased complexity is "worth it" is not answered obviously in the negative.

Consider the following example: let's say that we have a clock that has a dot that blinks, so that it's on during every even second and off during every odd second. The article, if I extrapolate correctly, would say that the model that says "at every point in time the probability that the light is on is 1/2" is just as _accurate_ (ignoring the question of its complexity) as the model that says "at every point in time the light is on iff it was off a second ago". I don't think that any model evaluation method that would claim that is useful. Thus, I see that article as, in large part, a strawman against a model evaluation method that isn't really used nor is intuitive. Is there something I'm missing?

@robryk

The parable has many issues in my eyes though im not sure that is so much an indication the undelying realization is wrong, or perhaps just the way it is framed is lacking. I need to think about that

That said I either do not follow your criticism, or perhaps I do and simply disagree, I am not sure yet.

but by something like comparing the “leftover surprise” the model leaves us with

So this part is either confusing or wrong, perhaps you can explain. What you seem to be saying is that left over surprise would be how much surprise one would expiernce if they previously only knew the model and then had the truth revealed to them. The “leftover surprise” indicating how close the model is to the truth.

Presuming that is, in fact, what you meant the failure here is that you can only determine how “true” a model is if you are omnipotent with regards to the problem, you already know the outcome. So in any practical sense it wouldn’t be a valuable way to interpret real world models.

Back to my original statement…

With all that said I don’t think a models likeness to the underlying system is at all relevant or important, it is only through a “trick” of wording in the parable that it is constructed in a way, counter to real world, to make it seem that way.

The parable leads in with statements that implies a 50/50 coin toss model is most accurate over the most number of iterations. This is where the fundamental error comes in IMO.

While it is true that if you randomly select players for games, without being able to track who is the same player from one game to the next, and sampled randomly over a large set you’d see 50/50 results. If that is your model then your model is telling you is the chance a randomly selected person would win against another randomly selected person. As a model it tells you that answer perfectly. However what the parable doesnt state, and at best hints at, is that it does nothing to tell you the likelihood of a specific person winning.

Thats all well and good, but what is the better model… well models are not and never were about providing you a realistic mental image of what is happening. A model is simply a system of interdependent components that produce some output. Generally the point of a scientific model is to determine the likely outcome, given some set of conditions, of a process. If a model can predict the real world event accurately given various conditions, within its scope, then its a good model. It doent even matter if it resembles the underlying system at all.

So to answer the question…

So to answer the question of if the coinflip model is a good one… Its an amazing model, if you are trying to model the chance one of two randomly selected people might win, without any other knowledge.

It is a very poor model at predicting other things however, to use the example from the parable it is a very poor model at determine the chance a player has to win if their opening move is the kings gambit. The second model would predict that best. In fact this too is where the parable goes wrong. The second model that categorizes likelihood of winning from the opening move would be better than the old model in every way presuming you are trying to determine the likelihood of winning if you know the opening move. Even over very large number of iterations this model would outperform the other and give more accurate results. So while it is, clearly the better model, this is not due to it being a closer representation of the underlying system, it is merely better because it predicts the outcome more accurately.

@torresjrjr @Diptchip @Science

@freemo @torresjrjr @Diptchip @Science

Presuming that is, in fact, what you meant the failure here is that you can only determine how “true” a model is if you are omnipotent with regards to the problem, you already know the outcome. So in any practical sense it wouldn’t be a valuable way to interpret real world models.

You can approximate KL divergence by sampling from the “world” distribution (easiest way to see how: it’s essentially expected value of log(1/p) where p is the probability the model assigned to the outcome we’ve sampled). That makes KL divergence estimable (with the small exception of models that assign probability of 0 to any outcome) when comparing models against the real world (insofar any estimates can be made against the real world).

So to answer the question of if the coinflip model is a good one… Its an amazing model, if you are trying to model the chance one of two randomly selected people might win, without any other knowledge.
It is a very poor model at predicting other things however, (…)

I agree completely. I would phrase it as it being the best model over the system where the outcome (white/black wins) is the only random variable being modeled.

So while it is, clearly the better model, this is not due to it being a closer representation of the underlying system, it is merely better because it predicts the outcome more accurately.

I don’t understand the distinction, or, perhaps I should say that I don’t understand what “closer representation” means. Is it something that can be evaluated (even by an omnipotent evaluator)?

@robryk

By closer representation i simply mean that it isnt naive to the underlying mechanics. That is, it maes assumptions that chess is a game, with pieces, with a board, and with opening moves, all of which require some level of understanding the nature of the game, our earlier coin flip model is completely naive to any internal workings.

@torresjrjr @Diptchip @Science