# Information Theory and Thermodynamics

Published: Last Edited:

Disclaimer: This essay has been submitted by a student. This is not an example of the work written by our professional essay writers. You can view samples of our professional work here.

Any opinions, findings, conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of UK Essays.

In order to develop better tools, machines and technology we have had to develop our understanding of the physical world. This has allowed us to construct machines that are more capable than those preceding it.

The French scientist Carnot was studying machines and was trying to understand how to make them better and more efficient. As part of his studies he calculated the maximum efficiency of any machine and was able to relate this to temperature. Carnot's idea was to simplify the machine to its simplest form (this generality that makes it universal) and analyse that - Carnot knew that machines of the time (and of today) work as a result of a temperature difference across the machine. In his time it was obvious, fire produced steam which turned a turbine that did work; today is not too different, all of our machines still need a temperature difference to make them work, however the temperature difference driving the machines may be at some distance, for example a power station producing electricity. Even wind, and solar require temperature differences to work.

Analysing these machines further lead to concepts that we, perhaps, take for granted: work, power and energy notable examples. Whilst working on these concepts Boltzmann came up ideas that grew into statistical thermodynamics. It was extended and correctly describes a whole range of phenomena.

The idea of thermodynamics is to relate various physical properties of a substance to the "bulk" behaviour of the constituent parts within.

Microstates and Macrostates:

Boltzmann realised that by knowing the number of different states that a system could be in and the number of configurations that each state would enable him to work out the probability of a particular state occurring. And that on average when something is observed it is more likely to be found in one of its more probable states. Many systems have lots of moving components and this means that over time a system will have "evolved" into a more probable state. This may now seem obvious, but it hadn't been pointed out explicitly at the time.

A Macrostate is the global state of a system. For example if we consider a box with red, blue and green balls a possible macrostate might be to find all of the red balls are in the bottom left corner, whilst all of the others are randomly distributed in the rest of the box. Another example of a macrostate might be that the total electrical charge of block could be Coulombs.

A Microstate is one particular configuration of the system that produces a macrostate. In the balls example if the balls are identical apart from colour we can permute the balls with the same colour amongst themselves and end up with different microstates. An example with the charged block might be that we have 4 particles each with charge as one microstate, and another might be to have 1 particle with and another with

Below is a figure that represents a hypothetical macrostate of three colours of balls three particular microstates that can be used to achieve it. In the leftmost diagram we have a macrostate with all of the red balls in the bottom left corner, the other diagrams show different permutations of the balls that also achieve the desired macrostate.

In order to calculate the probability of this state we would need to know how many combinations of it there are. This is a simple counting argument: we have 1 way of putting the green ball in its spot, two ways of putting the blue balls in their position and ways of arranging (we can pick any of the 5 to go in the corner, then any of the remaining 4 to go next to that, then any of the remaining 3 etc). There are a total of 8 balls and so the

In general if there are objects we have possible arrangements; if we also have different "types" and if of them are of type 1 (say red), are of type 2 and are of type then we can find the total number of permuted arrangements with:

We can use these two facts to calculate the number of "accessible" microstates of type , this is called the "weight" of the microstate and is denoted by, :

The weight of a microstate is proportional to the probability of the system being in it. So one way to calculate the probability of being in the state is via:

where the summation is over the weights of all the other possible microstates.

A handy way to view a microstate is with a pack of cards (Birks bath), in a pack of playing cards the statistical weight of a club is 13 since there are 13 of then the statistical weight of a queen is 4. The probability of selecting a club card is the chances of picking out a club are times greater than picking out a queen. The statistical weight of the queen of hearts is 1.

There is one obvious constraint that can always be imposed and that is that the total number of particles is the sum of the number of particles in each state:

We can impose other constraints on the system as they are required later. Because the particles of each type are identical it is natural to assign a probability that a randomly selected particle is of type, , as:

We are also able to define an "Expectation" value for the system. If we were interested in the average occupancy of each of the types we would have:

which would represent the average occupancy of each type. If we were interested in the charge (or energy (I shall use for either) we would similarly have:

Let us take some examples and compute the statistical weights, average occupancy and average energy (represented by the value of the type index e.g. if , the energy would be two units). I shall consider that the "atoms" are all identical apart from the energy that they have and that a macrostate is the same for each. For the first case let us assume that we have 5 atoms and the macrostate corresponds to an energy of 5 units. The table below shows that (for example) the microstate 3 has a weight of 20, this means that there are 20 microstates with the occupancy levels given that correspond to the macrostate We can tabulate the various combinations as below:

 microstate number Occupancy of type i "weight" probability n0 n1 n2 n3 n4 n5 1 4 0 0 0 0 1 5 0.0397 2 3 1 0 0 1 0 20 0.1587 3 3 0 1 1 0 0 20 0.1587 4 2 2 0 1 0 0 30 0.2381 5 2 1 2 0 0 0 30 0.2381 6 1 3 1 0 0 0 20 0.1587 7 0 5 0 0 0 0 1 0.0079 totals 15 12 4 2 1 1 126 1 Average occupancy 2.222 1.389 0.794 0.397 0.159 0.040

Table 3.1: Table showing occupancy levels for a 5 atom system with a macrostate of 5.

This table was generated by finding all of the numbers that sum (in this case) to 5 which is the macrostate. It shows the number of atoms with a particular energy in the columns headed , the statistical weight of each microstate is in the "weight" column, the probability column next to it shows the probability of randomly selecting this microstate from a given macrostate (in this case 5 atoms and a total energy of 5). The row titled average occupancy shows the expected occupancy of an energy level of type , calculated from the table. Looking at the table there are two equally most likely microstate arrangements. The first of these corresponds to and , both occurring with a probability of 0.238.

Another possible macrostate is listed below, this time we have 7 atoms and an energy of 7 units. The headings of the table are the same as in the previous example. We can see that the weight of the most probable microstate is 420 and that we have a probability of 0.245 of randomly selecting one of them. The occupancy levels are:

 microstate occupancy of type i weight probability n0 n1 n2 n3 n4 n5 n6 n7 1 6 0 0 0 0 0 0 1 7 0.004 2 5 1 0 0 0 0 1 0 42 0.024 3 5 0 1 0 0 1 0 0 42 0.024 4 5 0 0 1 1 0 0 0 42 0.024 5 4 2 0 0 0 1 0 0 105 0.061 6 4 1 1 0 1 0 0 0 210 0.122 7 4 1 0 2 0 0 0 0 105 0.061 8 4 0 2 1 0 0 0 0 105 0.061 9 3 3 0 0 1 0 0 0 140 0.082 10 3 2 1 1 0 0 0 0 420 0.245 11 3 1 3 0 0 0 0 0 140 0.082 12 2 4 0 1 0 0 0 0 105 0.061 13 2 3 2 0 0 0 0 0 210 0.122 14 1 5 1 0 0 0 0 0 42 0.024 15 0 7 0 0 0 0 0 0 1 0.001 totals 51 30 11 6 3 2 1 1 1716 1 Average occupancy 3.231 1.885 1.028 0.514 0.228 0.086 0.024 0.004

Table A3 2: a seven atom system with a total energy of seven

A final example consists of a system of 10 atoms and a total energy of 9. As will be readily seen as the number of atoms and the energy increases the number of microstates corresponding to a given macrostate increases so does the size of the table. It was quite difficult to work out the number of combinations of energy that could occur and I wouldn't want to do it again for larger tables. In the next part we shall use the method of Lagrange multipliers to massively simplify the calculations for the probabilities and expectations. For the case of 10 atoms and an energy of 9 units.

We see that the most probable microstates have the following occupancy levels:

The most probable microstate has a probability of 0.1555, but there is another microstate that is only slightly less probable (a probability of 0.1300) and this has occupancy levels of:

The two least likely microstates are the following:

Both have a probability of 0.0002 which is very small indeed. Table 3 is below:

 d occupancy of each type i "weight" probability n0 n1 n2 n3 n4 n5 n6 n7 n8 n9 1 9 0 0 0 0 0 0 0 0 1 10 0.000205677 2 8 1 0 0 0 0 0 0 1 0 90 0.00185109 3 8 0 1 0 0 0 0 1 0 0 90 0.00185109 4 8 0 0 1 0 0 1 0 0 0 90 0.00185109 5 8 0 0 0 1 1 0 0 0 0 90 0.00185109 6 7 2 0 0 0 0 0 1 0 0 360 0.00740436 7 7 1 1 0 0 0 1 0 0 0 720 0.014808721 8 7 1 0 1 0 1 0 0 0 0 720 0.014808721 9 7 1 0 0 2 0 0 0 0 0 360 0.00740436 10 7 0 2 0 0 1 0 0 0 0 360 0.00740436 11 7 0 1 1 1 0 0 0 0 0 720 0.014808721 12 7 0 0 3 0 0 0 0 0 0 120 0.00246812 13 6 3 0 0 0 0 1 0 0 0 840 0.017276841 14 6 2 1 0 0 1 0 0 0 0 2520 0.051830522 15 6 2 0 1 1 0 0 0 0 0 2520 0.051830522 16 6 1 2 0 1 0 0 0 0 0 2520 0.051830522 17 6 1 1 2 0 0 0 0 0 0 2520 0.051830522 18 6 0 3 1 0 0 0 0 0 0 840 0.017276841 19 5 4 0 0 0 1 0 0 0 0 1260 0.025915261 20 5 3 1 0 1 0 0 0 0 0 5040 0.103661045 21 5 3 0 2 0 0 0 0 0 0 2520 0.051830522 22 5 2 2 1 0 0 0 0 0 0 7560 0.155491567 23 5 1 4 0 0 0 0 0 0 0 1260 0.025915261 24 4 5 0 0 1 0 0 0 0 0 1260 0.025915261 25 4 4 1 1 0 0 0 0 0 0 6300 0.129576306 26 4 3 3 0 0 0 0 0 0 0 4200 0.086384204 27 3 6 0 1 0 0 0 0 0 0 840 0.017276841 28 3 5 2 0 0 0 0 0 0 0 2520 0.051830522 29 2 7 1 0 0 0 0 0 0 0 360 0.00740436 30 1 9 0 0 0 0 0 0 0 0 10 0.000205677 totals 172 67 26 15 8 5 3 2 1 1 48620 1 Average occupancy 5.000 2.647 1.324 0.618 0.265 0.102 0.034 0.009 0.002 0.000

Table 3.3: a ten atom system with a total energy of nine

Whilst it isn't mathematically rigorous it looks, form the tables 1, 2 and 3, that the most likely states correspond to states that have a largest number of permutations i.e. a large weighting. It also seems likely that the most probable microstates correspond to having the energy distributed fairly "evenly across" the atoms whilst maximising the weight function.

In a sense, microstates that give all of the energy to one or two atoms are less likely than states that give less energy to more atoms. One of the major problems with the method above is that it quickly becomes time consuming to calculate the various combinations by hand. For example with there are 30 microstates corresponding to the macrostate; with there are 628 and when there are 1958 different microstates. The number of microstates increases rapidly with . I think that because of these concerns it became an issue for Boltzmann to find a better calculating process.

The forgoing hopefully suggests that we need to maximise the weight function subject to the constraints that the average energy is that given by the macrostate, and we must also have the problem constrained so that the total number of atoms in each occupancy level sums to the total number of atoms . If we have occupancy levels and energy levels, Lagrange's method (appendix 2) tells us to minimise the following, but Boltzman found that it was easier to minimise . We have:

Using the normal rules for logs:

Using Stirling's approximation which says that for large , , and then differentiating with respect to the only variable we get:

The first partial derivative can be easily evaluated to give , and hence we find:

But this is a constant (since the derivative is zero) and so we get the solution:

Applying the constraints to this we find that the probability of an atom to be in a microstate is:

In thermodynamics the coefficientÂ  is re-labelled asÂ  and so we should really have

Using the script in appendix 2 we can find and use this to calculate the probabilities. The factor in the numerator, is known as the Boltzmann factor and the denominator is often just written as Z (this is from the German word zustandssumme which google translates as "conditional sum"), in thermodynamics it is called the "partition function". I have put two interesting properties of the partition function in Appendix 4.

Equation looks very similar to the dice problem that was considered in appendix 1, but the constraint and range are slightly different. We need to solve: