Shannon didn't define it as "chaos" rather as a:

"measure of information, choice and uncertainty."

http://math.harvard.edu/~ctm/home/text/others/shannon/entrop...

It also matches what we know: the less "uncertain" some amount of the information the less it is compressible.

However he does mention "chaotic" in one sentence, in evaluating the "constraints" imposed by the language when making the crossword puzzles:

"If the redundancy is too high the language imposes too many constraints for large crossword puzzles to be possible. A more detailed analysis shows that if we assume the constraints imposed by the language are of a rather chaotic and random nature, large crossword puzzles are just possible when the redundancy is 50%."

That said, I don't under whether Gibbs entropy or Boltzmann entropy is fundamental-- ?

I have to disagree, especially given that one of the great mysteries of our generation (the Black Hole Information Paradox) hinges on what happens to information -- sure, it's information about particles, whatever that might mean. But Jacob Bekenstein (among others) argues that Thermodynamic entropy is analogous to information (Shannon) entropy[1].

[1] http://webhome.phy.duke.edu/~mkruse/Phy255/255S06_Joyce_Copp...

“As you have probably noticed, I didn’t say anything about information. That’s because really the reference to information in “black hole information loss” is entirely unnecessary and just causes confusion. The problem of black hole “information loss” really has nothing to do with just exactly what you mean by information. It’s just a term that loosely speaking says you can’t tell from the final state what was the exact initial state.“

Thermodynamics doesn't care about our ignorance of potential system configurations, but that these configurations are available for microscopic system evolution!

Thermodynamics equations are based on the macroscopic constraints on the system that we choose to measure and manipulate, implicitely admitting our ignorance about the potential microscopic configurations. Thermodynamic entropy, unlike energy, is not an intrinsic property of the physical system; it is a property of our macroscopic description of the system and the experimental processes that we decide to perform on it.

See section VI. THE "ANTHROPOMORPHIC" NATURE OF ENTROPY in https://bayes.wustl.edu/etj/articles/gibbs.vs.boltzmann.pdf

"Recognition that the "entropy of a physical system" is not meaningful without further qualifications is important in clarifying many questions concerning irreversibility and the second law. For example, I have been asked several times whether, in my opinion, a biological system, say a cat, which converts inanimate food into a highly organized structure and behavior, represents a violation of the second law. The answer I always give is that, until we specify the set of parameters which define the thermodynamic state of the cat, no definite question has been asked!"

Temperature, interestingly, is also in our heads. It isn't actually a measure of average energy, but actually is defined as a function of energy and entropy. (Entropy comes first and temperature is defined in terms of it, not the other way around).

T = dQ/dS

You can create systems in real life with infinite temperature, or even with negative temperature, but of course they certainly don't have infinite energy. One way to do so is to pump a system into having a population inversion, such that its entropy decreases as its energy increases, because the number of possible high energy microstates is small.

In general, only one microstate is ever "available" for a system to evolve into, which is the one that its current microstate will evolve into in accordance with the laws of physics. The trouble is that we don't know its current microstate, so we don't know its future microstate, so it seems to us that there are many available microstates, but that's all due to our own ignorance having only conducted summary statistical measurements and isn't an intrinsic property of the system.

It may be used to this end, but it's not what Shannon entropy directly gives you.

There is no way to reduce the entropy of a totally random system by knowing more about it, unless your hypothesis is that true randomness doesn't exist in the universe

(You need to define a sensible measure, because the state variables are continuous, but that's not important).

You can express this in terms of the message length or number of bits of information you would need to gain about that container of ideal gas in order to narrow it down to a particular microstate.

It's no different than the entropy of, say, a password.

> There is no way to reduce the entropy of a totally random system by knowing more about it

There absolutely is! For example, Maxwell's demon. Maxwell's demon will fail to reduce system entropy if it doesn't know the microstate to begin with, but otherwise it can use its knowledge to reduce the entropy of a gas by separating its parts.

> unless your hypothesis is that true randomness doesn't exist in the universe

I do also subscribe to this hypothesis, that like entropy, randomness also only exists in our heads as a reflection of our lack of information about a system, and isn't a property of any system itself. But I don't think I need to rely on this assumption.

I don't see how that connects to the ideal gas example. Care to share?

Gasses are the same in principle but harder to visualize concretely, so let's think instead of a crystal lattice with impurities -- for example a doped silicon wafer.

I hand you a 28 gram disc of 0.1% n-doped silicon. It's got about 6e23 atoms in it, of which 0.1% are phosphorus.

That's a good statistical description of the crystal, but there are still a large number of possible configurations of phosphorus and silicon atoms that match that specification, but only one of them describes what you are holding in your hand. For whatever reason, you want to know the exact atomic arrangement of this particular wafer.

Fortunately, I've imaged this disc with an advanced electron microscope that has identified the identity of each atom in the crystal lattice, whether silicon or phosphorus.

How large is the data file that I have to send you? How much can I compress my electron microscope data?

Just like "abcdefg is less likely than kfbeksn", we have some prior information we can use to do compression. For example, dopant atoms are unlikely to cluster together, so Si-Si-P-P-P-Si-Si is less likely than P-Si-Si-P-Si-Si-P. That prior information is no harder to incorporate into the entropy calculation than it is in the case of passwords.

But such subtleties aside, we can start by assuming that any configuration is equally likely, compute the total number of configurations, and take the log base 2. That's going to be the minimum file size of the electron microscope scan. (I guesstimated 6e12 Tb but don't quote me on that). To do a proper job, we'd also need to measure and transmit the state vector of the lattice vibrations, but I'm ignoring that for now.

Ok, so that's for a crystal, which I chose because the electron microscope technology is easy to concretely visualize, and the state can be expressed as a string of Si's and P's. But an ideal gas is similar in principle, except its state variable consists of a string of positions and velocity values. We will have to discretize to some resolution to get discrete states, but that's fine. There is some datafile I could send you that would (quantum uncertainty aside) let you know the exact state of the gas, and with fine manipulators you could use that data to sort the gas molecules into different containers by molecular energy and reduce its entropy to zero. The entropy of the data you'd need to do that would be at least as high as the entropy of the gas, and upon clearing that data from your memory, the second law is satisfied.

The entropy content of a situation can't be computed without knowing the physics that would have gotten it there, just as I can't compute the entropy of a message (my surprise at receiving it, or the number of bits needed to transmit it) consisting of a single symbol without some knowledge of the distribution it was drawn from (the expectations of myself, or the receiver). (If it is part of a longer message I can use that message itself as a model, but that is only a first approximation.)

It really doesn't. The little Enterprise thought-experiment and anecdotal impressions of what "looks" and "doesn't look" disordered is nonsense. From my source above:

> Shannon entropy is defined for a given discrete probability distribution; it measures how much information is required, on average, to identify random samples from that distribution.

It's carefully defined, and used in basically every piece of technology we own. Further, it's not even particularly tough to wrap your head around (a middle-schooler could understand Huffman encoding).

I think you're misreading it, the author is trying to make the point that you can't meaningfully answer that question just by looking at the dots. It's a trick question, but it's used as a part of teaching how you compute entropy (which involves knowing the distribution). Yes, this is essentially information theory, but he's a physics teacher.

Is that really the idea here? You're right: maybe I'm just not getting the point.

From an information theory perspective, consider 'apple' vs 'aaaaa'. I submit that if you have nothing besides the string itself, you'd conclude that 'aaaaa' is lower entropy than 'apple', but if your model is word frequency on the web, it'd be the other way around. There isn't anything inherent in the strings that makes one or the other true.

There are interesting cases in-between where you could derive a model with more math, e.g. the first million digits of pi has very low entropy, but in general exactly how much is a matter of the uncomputable Kolmogorov complexity.

With a physical system, if you know the configuration of the system and the laws of physics it obeys, you can compute this in a meaningful way, and it can tell you what direction the system "wants to go" to satisfy the second law of thermodynamics.

Wikipedia has a neat article on the topic which I'm just starting to read now: https://en.wikipedia.org/wiki/Entropy_in_thermodynamics_and_...

When you only know summary, average statistics of a system (such as its temperature), its entropy is nonzero, because your information about the system isn't enough to isolate a particular microstate. Instead you just have a distribution of many possibile microstates, any of which would match the measurements you know. The more possible system microstates there are that would agree with your data, the higher its entropy.