How do 'luck' calculations work..

[dorbel]

Yes, As I have said before, the bot definition of "luck" is both flawed and incomplete.

[sixty_something]

thanks, boomslang .. that is one nice piece of work

as they say, a picture is worth a thousand words or in this case 42,000 moves .. i tried to say it before as simply and completely as i could only to be summarily shot down .. it appears your graph and comments support my perception (certainly not unique) perhaps even establishing a proof of concept .. so, pardon me while i repeat myself

the better player minimizes his error rate
in so doing he optimizes his 'luck' potential
yielding the impression the better player is luckier
than less skilled opponents

in answer to an earlier question, yes, 'luck' potential is simply another way of saying equity change, but change on a subsequent turn .. when a bot optimizes for equity on the current move, part of that optimization must directly include the careful analysis of the opponents next possible rolls and moves as well as the bot's next turn where equity based "luck" is calculated .. this look ahead analysis seeks an optimal position in subsequent moves .. this appears to effectively optimize the equity based "luck" calculation for the better player .. i think that may be what we are seeing in boomslang's graphs

by definition, the less skilled player will make decisions that yield suboptimal equity gain, thus the "luck" calculation of the less skilled player will be impacted negatively or be suboptimal .. this impact is directly seen in equity changes from move to move .. since the "luck" calculation is a calculation of equity change based on the roll, the concept that optimal versus suboptimal play may impact "luck" calculation seems almost transparent to me .. obviously, it is neither transparent of trivial

most importantly, i am not saying the better player is luckier .. i am saying that the "luck" calculation is NOT a perfect reflection of pure luck which theoretically would be the same for both players regardless of skill .. however, since the "luck" calculation is an equity based calculation only approximating the impact of pure luck, it is biased toward equity not probability .. since the better player optimizes equity, is it any surprise we see results biased toward the better player?

all i am saying is "the nature of backgammon", as boomslang says, and the nature of the "luck" calculation yield "the impression the better player is luckier" .. boomslang's graph number 4 seems to directly support this .. indeed it appears to may be more than an impression, but it is a slippery slope as we have seen to attempt to equate pure theoretical luck with "luck" calculations .. i think all of us who play bots significantly more powerful than we are have repeated first hand experience that bots just seem too damn lucky - don't we? i would contend that this perception is merely a reflection f what boomslang's experiment has shown

now, boomslang, would you consider conducting another experiment to test another aspect of of this theory?

i believe that the more moves (or games) analyzed the more the calculated "luck" differential will diverge between a better player and a less skilled player probably up to some limit .. while 42,000 moves may be a large enough sample to have reached such a limit, it may not be .. so, if it isn't too time consuming, how would that last graph look after say 100,000 moves or more? any idea where that limit may be or if there is a limit at all?

finally, has anyone yet directly addressed diane's original question and really defined how the "luck" calculation really works? now that we seem to have established that it works differently than we expected, it seems a good time to revisit that original question

[boomslang]

This figure coincides with my own shorter studies of matches played between me and gbots on fibs. I have broken the study down into games rather than matches.

In my example I used games aswell (last two graphs).

However the better player, which in my study is at least occasionally me, gets the positive luck more often.

Maybe also another thing is going on..? Being (very) lucky throughout a whole game means lots of trivial moves and therefore small errors? I think for players up to intermediate level this is the case.

The densities in the second graph should be symmetrical around zero instead of showing the biases they do.

Why do you think that? I think they should have an average of zero (because luck tends to zero in the long run), but they cannot be symmetrical because the better player will be more likely to be really unlucky than to be really lucky. (In other words, it has virtually no chance to be really lucky because its opponent gives away too many equity and it had won the game/match already. Remember, these graphs are from the situation expert vs newbie.)

So what do you think? Have you shown with this that total luck does not tend to zero in the long run if the skill difference between the players is large? Or have you shown that gnu's calculation of luck is defective? You're obviously aware of the problem or you wouldn't have included that "if" above.

Not sure what you mean... What I meant to explain with this quote is the source of the 'dip' in the graphs of Zorba and me.
I included the 'if' because of Diane's remark about bots appearing lucky. When a bg player simply looks at whether or not a bot had a positive total luck and draws his conclusion on that -- and I think a lot of players do that -- he will indeed see a lucky bot more often than an unlucky bot. It is however unfair to conclude that bots are lucky in the long run, because of the negative skewness of the distribution of luck during a game or match for the bot. And yes, that means that I think the asymmetry of the distributions are not artefacts caused by GNUbg's inconsistency in evaluating luck and skill.

[boomslang]

now, boomslang, would you consider conducting another experiment to test another aspect of of this theory?

i believe that the more moves (or games) analyzed the more the calculated "luck" differential will diverge between a better player and a less skilled player probably up to some limit .. while 42,000 moves may be a large enough sample to have reached such a limit, it may not be .. so, if it isn't too time consuming, how would that last graph look after say 100,000 moves or more? any idea where that limit may be or if there is a limit at all?

finally, has anyone yet directly addressed diane's original question and really defined how the "luck" calculation really works?

I dont expect the results to be much different. The width and location of the dip (if any) is based upon the difference in playing strengths and not on the number of rolls. Maybe match length also plays a role.
I might generate some 58000 rolls more if I get bored (prolly after July 11th, the day Holland plays Germany).


Perhaps Inim can shed some light on GNUbg's internals regarding Diane's original question.

[diane]

I included the 'if' because of Diane's remark about bots appearing lucky.

Just to be clear, I have no personal 'feeling' that bots are lucky....I only used them as an example because [excluding blunderbots], they will play the best move in all circumstances and therefore have the best chance of appearing to have a high luck factor [as calculated by the bot] if there is anything in all this.

[pck]

Why do you think that? I think they should have an average of zero (because luck tends to zero in the long run), but they cannot be symmetrical because the better player will be more likely to be really unlucky than to be really lucky. (In other words, it has virtually no chance to be really lucky because its opponent gives away too many equity and it had won the game/match already. Remember, these graphs are from the situation expert vs newbie.)

You're right. I was confused when I wrote that, mixing in my thoughts about my own experiments where both players had equal skill. Average of zero it is indeed.

Not sure what you mean... What I meant to explain with this quote is the source of the 'dip' in the graphs of Zorba and me.
I included the 'if' because of Diane's remark about bots appearing lucky. When a bg player simply looks at whether or not a bot had a positive total luck and draws his conclusion on that -- and I think a lot of players do that -- he will indeed see a lucky bot more often than an unlucky bot. It is however unfair to conclude that bots are lucky in the long run, because of the negative skewness of the distribution of luck during a game or match for the bot.

I misunderstood your remark. You said "This means that if you consider a bot 'lucky' when it has a positive luck, it not just appears lucky more often, it actually is." I didn't give due notice to the crucial "often", effectively reading "luckier" instead. ("More often" here meaning "in more games", "positive luck" meaning "more luck than opp in a particular game" (= "positive luck difference for the bot" when we talk gnu).)

And yes, that means that I think the asymmetry of the distributions are not artefacts caused by GNUbg's inconsistency in evaluating luck and skill.

I can safely return to that assumption now too, with the above confusion cleared up for me. It makes a lot more sense this way. Thanks again for the clarifications.

[pck]

all i am saying is "the nature of backgammon", as boomslang says, and the nature of the "luck" calculation yield "the impression the better player is luckier" .. boomslang's graph number 4 seems to directly support this .. indeed it appears to may be more than an impression, but it is a slippery slope as we have seen to attempt to equate pure theoretical luck with "luck" calculations .. i think all of us who play bots significantly more powerful than we are have repeated first hand experience that bots just seem too damn lucky - don't we? i would contend that this perception is merely a reflection f what boomslang's experiment has shown

I agree. As boomslang points out, the difference between "being luckier" and "being lucky more often" is very important and a potential source of confusion:

To have been lucky more often does not necessarily mean that one has been luckier.

One may have been slightly luckier than one's opp for 17 matches and hugely less lucky in the next 3. One's total luck may then well be zero or close to zero, even though one was luckier far more often. As boomslang's data shows, this is indeed what is likely to happen when the skill difference between the players is large.

The phenomenon can also be explained without looking at empirical data. http://www.bkgm.com/articles/Zare/AMeasureOfLuck.html has this:

3. Strong players are lucky more often than weak players.

Strong players still have to be lucky to win, just less so. Because they don't need to be as lucky, their good luck is spread out to win many matches and their bad luck is concentrated in a few losses. Someone who bears in and bears off efficiently doesn't roll larger numbers than someone who aims for the ace point. He/she just uses the pips received more efficiently. The same thing happens with luck.

A strong player throwing away .2 less equity than I do will still have to be lucky by +.8 to beat me (in a match). Of course, this will happen 60% of the time.


In every match I start out with a 50% chance of winning. To win, I need to generate another 50% of equity to get to 100%. Let's assume my opp plays consistently worse than I do, that is, he throws away a lot more equity than I do by making bad moves. Therefore, if I win the match, much or most of the 50% equity I needed (in order to get from my initial 50% to the final 100%) will have been generated by that differential in skill. The rest will come from a moderate to small amount of luck (I have "virtually no chance to be really lucky" as boomslang notes in #82). In the matches I lose, the skill differential will still be there, so in those, my opp will necessarily have to have had a lot more luck than me. (*)

Now let's look at a long run of many matches played between me and this opp. I know from the previous paragraph that there will be small amounts of luck for me in matches I win, and lots of bad luck in matches I lose. So for the luck total to even out to zero, the number of matches which I win must be larger than the number of matches I lose. Hence I will have the better luck in more matches as a consequence of my better skill.

The same is of course to be expected to happen when a human player plays a better skilled bot.

The empirical data which boomslang presented is consistent with this, hence it cannot be inferred from his data that the bots' luck calculations are flawed.

(*) I may win a match despite the fact that my opp is luckier than me. But for that to happen the skill differential needs to be huge (> 50%), as can be calculated from the equation 50% + luck_my - luck_opp + skill_opp - skill_my = 100% which characterizes the matches I win. (skill_my/opp = my/opp's equity *waste*). So the above argument holds for players whose skill diff. is 50% or less.

[sixty_something]

today's Writers Almanac celebrates the birthday of historian David McCullough who practices the habit of painstaking research into primary sources .. for example, his biography of Harry Truman, took him 10 years to research and write. Truman (1993) won the Pulitzer Prize.

Writing is thinking. To write well is to think clearly. That's why it's so hard.

that quote made me think of this thread  

[sixty_something]

OK, Zorba, i know you have been busy and probably a wee bit hungover watching Le Grand Orange** make the finals, but your voice is keenly missing after boomslang's excellent experiment .. for me, i am not convinced i am even close to understanding the whole concept yet .. i have been busy as well with prepping to move at the end of the month and playing chess with my grandson .. i have not even begun your suggested reading assignment, much less read this entire thread (it got a little out of control last week)

further, i am also not sure whether my simple four line statement is any better understood or acceptable by anyone including myself .. it was the best i could do at the time and i haven't had time to persue it any further

so, if perchance you find time before or after the FIFA Finals, i'd sure like to read your take on these new developments whatever they may be - pro, con, or fed up

of course, other voices are welcome too .. it seems we have come close to wrapping up this topic for now .. i suspect it will raise its head again here or in another topic .. if we are close to some kind of conclusion, let's each try to wrap it up with some kind of summary .. IMHO, i think this thread could provide very useful fodder for someone to publish some serious work helping us all and hopefully others to better understand the concepts of pure luck and calculated "luck" in backgammon

** Le Grand Orange was actually a redheaded rookie sensation named Rusty Staub with the original Houston Colt .45's (now Astros) way back in 1963 when he was only 19 and i 16 going on 17 .. without "the Google" that was as close as i could get to a nickname for the Dutch team

[dorbel]

The contributions of boomslang in particular, as well as pck and Zorba, have contributed to our understanding of what bots mean by "luck" and why superior players get the "luck" more often than not. It is important to understand that what we mean by "luck", usually winning a game with a joker, is something different and something that on average will be equal in the long run. Other than that, I am not sure how understanding what is going on will help us to play better! A good play is still a good play, same as it ever was.
However, I don't recall ever seeing this research anywhere else and kudos to these guys for adding something new to the game right here on fibs. Applause, applause.

[diane]

However, I don't recall ever seeing this research anywhere else and kudos to these guys for adding something new to the game right here on fibs. Applause, applause.

Yes, I had no idea this simple question would lead to 5 pages of interesting stuff [5 pages of whining about luck, maybe - given that it is a fibs specialist subject  ]

[sixty_something]

something i'd especially like to see are definitions of key terms and phrases .. for example,

• LUCK - pure luck such as that we see when rolling fair dice which we all agree is theoretically equal regardless of skill
• "luck" - the calculated expression of equity gain or loss for an individual player resulting directly from the roll of the dice in a given situation

are those definitions complete and acceptable?

can we concurr LUCK and "luck" are two independent entities with different properties?

can we further state that boomslang's work provides convincing evidence that in the long run a significantly better player tends to accumulate more "luck" than the less skilled player?

finally, has anyone confirmed exactly how "luck" is calculated?

the best explanation i recall of the "luck" calculation is the difference between the optimal equity for a given roll and the weighted average of the optimal equity for all possible rolls

are there other terms and phrases we need to define for the purpose of better understanding and communication?

[sixty_something]

in keeping with the definitions i suggested, i modified the quote below for improved and consistent readability (i hope)

It is important to understand that what we mean by "luck" LUCK, usually winning a game with a joker, is something different [than calculated "luck"] and something that on average will be equal in the long run. Other than that, I am not sure how understanding what is going on will help us to play better! A good play is still a good play, same as it ever was.

i agree completely, dorbel, that such understanding will not yield better play directly .. however, i do believe it leads to improved understanding of analysis tools such as GNUbg and Snowie and toward better understanding equity changes .. indirectly that may yield better analysis to us as individual human players over the board .. i would not dare suggest exactly how

The contributions of boomslang in particular, as well as pck and Zorba, have contributed to our understanding of what bots mean by "luck" and why superior players get the "luck" more often than not.
...
However, I don't recall ever seeing this research anywhere else and kudos to these guys for adding something new to the game right here on fibs. Applause, applause.

i agree wholeheartedly .. while some of what has been written may now need to be re-read and re-interpreted, the overall level of understanding of "luck" calculations, equity changes, the mathematics of LUCK, and analysis engines has benefited us all as readers

while i have contributed very little to the technical aspects of this thread, i am proud to have participated in driving the argument forward in the face of withering crossfire from Zorba and pck .. frankly, i am still watching my back and open to reconsidering my simply stated opinion which initiated much of the dialog with Zorba and pck leading to boomslang's excellent experiment

your observation, dorbel, that you have not seen this research presented before is further evidence (from a reliable source, IMO) that diane's simple little question could lead to a serious paper that could be presented and/or published in many forums including academic, computing, and gaming

finally, kudos also to stog and webrunner as well for making FIBSboard available as a community resource for we FIBSters where stuff like this can happen -- life in FIBSland is better for it

[pck]

something i'd especially like to see are definitions of key terms and phrases .. for example,

• LUCK - pure luck such as that we see when rolling fair dice which we all agree is theoretically equal regardless of skill
• "luck" - the calculated expression of equity gain or loss for an individual player resulting directly from the roll of the dice in a given situation

are those definitions complete and acceptable?

can we concurr LUCK and "luck" are two independent entities with different properties?

I'd like to understand what LUCK is supposed to be or refer to. What exactly is it that we're seeing when LUCK occurs? With LUCK, are you referring to the idea of randomness, while "luck" quantifies my chances of winning a bg match? Is LUCK a non-numerical concept while "luck" is numerical?

can we further state that boomslang's work provides convincing evidence that in the long run a significantly better player tends to accumulate more "luck" than the less skilled player?

I don't think we can. As boomslang says in #82 (last paragraph) and I tried to explain in more detail and without resorting to empirical data in #86, it is not more luck that the more skillful player gets, but instead he will be the luckier player in more games/matches.

[Zorba]

boomslang's experiment is very nice indeed! It confirms the findings of XG about my matches, but with much more data and a bigger skilll difference it shows it much better. The skew in the luck graphs for GnuBG beginner and GnuBG expert and how they compare is very interesting. I agree with boomslang's conclusions and replies, and with pck's later conclusions which are similar. It's all in agreement with theory as far as I can see, and overall luck tends to zero regardless of skill.

The definition of luck and the usage of the word keeps causing trouble, it seems. This is how the bot determines the luck value for a single roll (rolls have luck attached to them, not moves!):

1. For the position before the actual roll, go through all 21 dice permutations. For each of the rolls, analyze for the best move and use the (0-ply) equity of that. Sum them all up (weigh non-doubles twice) and divide by 36. Now you have the average equity over all the next rolls, and the (1-ply) equity of the position before the roll.

2. For the current roll, analyze for best move (already done above, actually), use the (0-ply) equity. Subtract the value found in [1]. Now you have the luck value for the current roll.

If you use the above process to determine all the luck values for all 21 possible rolls, you'll see that the various luck values cancel out over all rolls. This is a logical result from the process used!

In other words, for a certain position, all the luck values for the various rolls add up (weighted) to zero, regardless of inaccuracies in the bot's evaluations. Because of this, luck values over various positions and rolls as in a game or match, will in the long run also tend to zero, regardless of any errors by the players (the luck calculations don't even consider what moves are played; they just take the resulting position from whatever move and go from there to determine luck for the next roll).

Here's another article on the subject (this site is a real goldmine BTW): http://www.bkgm.com/rgb/rgb.cgi?view+869

It's interesting to note that these luck calculations are also used for variance reduction in rollouts. Instead of using the actual result from a simulated game (trial) in a rollout, the bot keeps track of the net luck per trial and adjusts the end result according to that. This greatly lowers the variance (standard error), and the better a bot's luck evaluations are, the more it will be reduced.

BTW, a bot programmer might wonder, why not just ask the bot directly for an equity of a position before the roll in step 1 above? In GnuBG speak: why not use the 0-ply equity of the position, then roll, use 0-ply again and subtract the previous equity? The reason is that an imperfect bot is not consistent between plies, and by rolling the dice, you move up one ply. You'd be comparing the equity of the actual roll on a 1-ply higher level than you had determined the equity of the position before the roll.

As a result, luck can be biased using this method, won't always add up to zero anymore over all 21 rolls, and not necesssarily tend to zero in the long run anymore. This is not what we want, hence the more time-consuming "1-ply" method described above.

[jools]

I know this is really old but I stumbled onto something interesting...

Yesterday I was feeling hard-done-by at my terrible luck, so I logged out and went for a bike ride.  I got about 3km before a car driver took me out. Now my left leg, right arm and, strangely, both thumbs are pretty banged up - so I'm recuperating and reading old threads on backgammon.

I have learned three things...

1. I don't need thumbs to play backgammon, although dressing and feeding myself are pretty tricky.
2. Suddenly, I'm pretty happy with my dice.
3. I have a much better grasp of what bad luck really is.

[DianeJames]

Thank you for sharing the information i have been looking for this information quite a long time ago.

[MultanTVHD]

i read all discussion of this topic and this topic clear my all issues Thank you.