Entropy and uncertainty

In research for this blog post, I came across an old favorite by Cosma Shalizi that links back to this other post. Shalizi says (in the former):

I doubt it helps matters that many statistical physicists are in the grip of sub-Bayesian ideas about maximum entropy

In the latter:

One of the ideas in physics which makes no sense to me ... is that statistical mechanics is basically an application of Bayesian ideas about statistical inference. On this view, the probabilities I calculate when I solve a stat. mech. problem --- say, the probability that all the molecules of air in this room will, in the next minute, be found at least three feet above ground level --- are not statements about how often such events occur. Rather, they are statements about the strength of my belief that such things will occur. Thermodynamic entropy, in particular, is supposed to be just the information-theoretic, Shannon entropy in my distribution over molecular states; how much uncertainty I have about the molecular state of whatever it is I'm dealing with. 

Here's an (unfair) way of putting it: water boils because I become sufficiently ignorant of its molecular state.

He also has a paper.

I've never been one to get into the Bayesian-frequentist argument, which much like the interpretations of quantum mechanics, seem to be a waste of my time (and are basically the same underlying issue). That it's a waste of my time does not imply it is a waste of your time if you happen to be a masochist who's into philosophical arguments that never go anywhere.

I would completely concur with the paper's result. Additional measurements should never cause you to become more uncertain on average. But that means entropy should decrease if you equate it with an ideal observer's uncertainty about the system.

Shalizi's solution is to abandon the idea that entropy represents an ideal observer's uncertainty about the state of the system.

And I'd agree. This doesn't really affect anything I've said on this blog since the basic mathematics is all still the same, and I've never implied Bayesian updating except when talking about belief in a theory.