Harmonic entropy: Difference between revisions

It so happens that we are more or less able to analytically continue this definition to the situation where {{nowrap|''N'' {{=}} ∞}}. More precisely, we are able to analytically continue the exponential of HE, which yields the same relative interval rankings as standard HE.

Our basic approach is: rather than weighting intervals by (''nd'')^0.5, we choose a different exponent, such as (''nd'')². For an exponent which is large enough (we will show that it must be greater than 1), HE does indeed converge as {{nowrap|''N'' → ∞}}, and we show that this yields an expression related to the [[The Riemann zeta function and tuning|Riemann zeta function]]. We can then use the analytic continuation of the zeta function to obtain an analytically continued curve for the (''nd'')^0.5 weighting, which we then show empirically does indeed appear to be what HE converges on for large values of ''N''.

In short, what we will show is that the Fourier Transform of this unnormalized harmonic Shannon entropy is given by

where φ(''t'') is the characteristic function of the spreading distribution and {{overline|φ(''t'')}} is complex conjugation. Below we also give an expression for the Rényi entropy for arbitrary choice of the parameter ''a''.

=== Background: Unnormalized entropy ===

Our derivation only analytically continues the entropy function for the "unnormalized" set of probabilities, which we previously wrote as ''Q''(''j''|''c''). For this definition to be philosophically perfect, we would want to analytically continue the entropy function for the normalized sense of probabilities, previously written as ''P''(''j''|''c'').

Here are some examples for different values of ''s''. All of these are Shannon HE (<math>a=1</math>), using <math>\sqrt{nd}</math> weights, with unreduced rationals (more on this below), with the bound that {{nowrap|''nd'' &lt; 1,000,000}}, just with different values of ''s''. All have been scaled so that the minimum entropy is 0, and the maximum entropy is 1:

As you can see, the unnormalized version is extremely close to a linear function of the normalized one. A similar situation holds for larger values of ''a''. The Pearson correlation coefficient of ρ is also given, and is typically very close to 1—for example, for {{nowrap|''s'' {{=}} 1%}}, it's equal to 0.99922. The correlation also seems to get better with increasing values of ''N'', such that the correlation for N=1,000,000 (shown above) is much better than the one for {{nowrap|''N'' {{=}} 10,000}} (not pictured).

In the above examples, note that there are slightly adjusted values of ''s'' (usually by &lt;&nbsp;1{{c}}) between the normalized and unnormalized comparisons for each plot. For example, in the plot for {{nowrap|''s'' {{=}} 1%}}, corresponding to 17.2264{{c}}, we compare to a slightly adjusted UHE of 16.4764{{c}}. This is because, empirically, sometimes a very slight adjustment corresponds to a better correlation coefficient, suggesting that the UHE may be equivalent to the HE with a miniscule adjustment in the value of ''s''.

For now, we will start with a derivation of the unnormalized entropy for {{nowrap|''N'' {{=}} ∞}}, as an interesting function worthy of study in its own right—not only because it looks exactly like HE, but because it leads to an expression for unnormalized HE in terms of the [[The Riemann zeta function and tuning|Riemann zeta function]].

For now, our derivation is limited to the case of <math>\sqrt{nd}</math> Tenney-weighted rationals, although it may be possible to derive a similar result for max(''n'',&nbsp;''d'') weighting as well.

Additionally, because it simplifies the derivation, we will use ''unreduced rationals'' in our basis set, meaning that we will even allow unreduced fractions such as 4/2 so long as {{nowrap|''nd'' &lt; ''N''}} for our bound ''N''. The use of HE with unreduced rationals has previously been studied by Paul and shown to be not that much different than HE with reduced ones. However, we will easily show later unreduced and reduced rationals converge to the same thing in the limit as {{nowrap|''N'' → ∞}}, up to a constant multiplicative scaling.

==== Definition of the unnormalized harmonic Rényi entropy ====

Let's start by recalling the original definition for harmonic Rényi entropy, using simple weighted probabilities:

Remember also that the definition of ''P''(''j''|''c'') is as follows:

where the ''Q''(''j''|''c'') is the "unnormalized probability"—the raw value of the spreading function, evaluated at the ratio in question, divided by the ratio's weighting. The above equation tells us that the normalized probability is equal to the unnormalized probability, divided by the sum of all unnormalized probabilities.

where, as before, ''S''^''a'' is our spreading function, taken to the ''a''th power, and ''K''^''a'' is our convolution kernel, with the weights on the delta functions taken to the ''a''th power as described previously.

Note that if ''S'' is symmetric, as in the case of the Gaussian or Laplace distributions, then the inverted argument of (−''c'') on the end is redundant, and can be replaced by (''c'').

Lastly, it so happens that it will be much easier to understand our analytic continuation if we look at the exponential of the UHE times ({{nowrap|1 − a}}), rather than the UHE itself. The reasons for this will become clear later. If we do so, we get

where |''j''| represents the weighting of the JI basis ratio ''j''. In the particular case of Tenney weighting, we get:

where ''j''_''n'' and ''j''_''d'' are the numerator and denominator of ''j'', respectively.

Furthermore, for simplicity, we can change the units, so that rather than the argument being given in cents, it is given in "natural" units of "{{w|nepers}}", a technique often used by Martin Gough in his work on [[logarithmic approximants]]. The representation of any interval in nepers is given by simply taking is natural logarithm. Doing so, by defining the change of variables {{nowrap|''c'' {{=}} {{sfrac|1200|log(2)}}''n''}}, we obtain

Now, we note our summation is currently written simply as <math>\sum_{j \in J}</math>. For a Tenney height weighting, we typically bound by <math>\sqrt{nd} < N</math> for some ''N''. However, although it is unusual, for the sake of simplifying the derivation, we will bound by {{nowrap|max(''n'', ''d'') &lt; ''N''}} instead, despite the use of Tenney height for our weighting. This will not end up being much of a problem, as the two will converge on the same result anyway.0

Bounding by {{nowrap|max(''n'', ''d'') &lt; ''N''}} is the same as specifying that {{nowrap|''j''_''n'' &lt; ''N''}} and {{nowrap|''j''_''d'' &lt; ''N''}}. Doing so, we get

$$\displaystyle \mathcal{F}\left\{K(n)\right\}(t) = \left[ \sum_{j_n=1}^N \frac{1}{{j_n}^{0.5 -i  t}} \right] \cdot \left[ \sum_{j_d=1}^N\frac{1}{{j_d}^{0.5 + i t}} \right]$$

Now, we can see that as {{nowrap|''N'' → ∞}} above, the summations do not converge. However, incredibly enough, each of the above expressions has a very well-known analytic continuation, which is the Riemann zeta function.

To perform the analytic continuation, we temporarily change the 0.5 in the denominator to some other weight {{nowrap|''w'' &gt; 1}}. This is equivalent to changing our original <math>\sqrt{nd}</math> weighting to some other exponent, such as (''nd'')² or (''nd'')^1.5. Doing this causes both of the summations above to converge, so that we obtain

Rewriting as a function of a complex variable {{nowrap|''z'' {{=}} ''w'' + ''it''}}, and noting that the zeta function obeys the property that {{nowrap|ζ({{overline|''z''}}) {{=}} {{overline|ζ(''z'')}}}}, where {{overline|''s''}} represents complex conjugation, we get

And we have now obtained a very interesting result: if we had instead gone with something like (''nd'')² weighting on rationals, rather than <math>\sqrt{nd}</math>, that our HE setup ''would'' have converged as {{nowrap|''N'' → ∞}}, and our original HE convolution kernel would have been the Fourier transform of a particular vertical "slice" of the Riemann zeta function where {{nowrap|Re(''z'') {{=}} 2}}.

Furthermore, although the above series doesn't converge for {{nowrap|''w'' {{=}} 0.5}}, we can simply use the analytic continuation of the Riemann zeta function to obtain a meaningful function at that point, so that our original convolution kernel can be written as

Lastly, to do some cleanup, we previously went with <math>\max(n,d) < N</math> bounds, rather than <math>\sqrt{nd} < N</math> bounds, despite using <math>\sqrt{nd}</math> weighting on ratios. However, it is easy to show above that regardless of which bounds you use, both choices converge to the same function when {{nowrap|''w'' &gt; 1}} in the limit as {{nowrap|''N'' → ∞}}. Since these series agree on this right half-plane of the Riemann zeta function, they share the same analytic continuation, so that we get the same result, despite using our technique to simplify the derivation. A good explanation fo this can be found on StackExchange [https://math.stackexchange.com/questions/2593993/convergence-of-product-of-series-to-zeta-function here].

It is likewise easy to show that the function ''K''^a(''n''), taken from the numerator of our original Harmonic Rényi Entropy convolution expression, can be expressed as

==== Analytic continuation of unnormalized harmonic Rényi entropy ====

We can put this back into our equation for the unnormalized harmonic Rényi entropy. To do so, we will continue with our change of units from cents to nepers, corresponding to a change of our variable from ''c'' to ''n''. We will likewise assume the spreading probability distribution ''S'' has been scaled to reflect the new choice of units.

Using our expression for ''K''^a as {{nowrap|''N'' → ∞}}, we get

We can simplify the expression of the above if we likewise take the Fourier transform of ''S''. If we do, we obtain the [https://en.wikipedia.org/wiki/Characteristic_function_(probability_theory) characteristic function] of the distribution, which is typically denoted by φ(''t''). We will use the following definitions:

Lastly, we note that for any real function ''f''(''x''), we have {{nowrap|ℱ{{(}}''f''(−''x''){{)}} {{=}} ℱ{{(}}{{overline|''f''(''x'')}}{{)}}}}. For simplicity's sake, we can this write as ℱ{{(}}{{overline|''f''(''x'')}}{{)}}. Putting that all together, we get

where we can drop the overline on |ζ_0.5''a''|² because it is purely real, and its complex conjugate is itself.

In all these examples, the left plot is a series of plots of UHE at different ''N'', with the zeta HE being a slightly thicker line in the background. The right plot is the largest ''N'' plotted against zeta HE, being almost a perfect line.

Note also that we don't plot the UHE directly, but rather {{nowrap|''e''<sup>(1 − ''a'')UHE_''a''}}, as described previously. The units have also been converted back to cents. Each HE function has been scaled so that the minimum entropy is 0 and the maximum entropy is 1.

Lastly, note that, due to our multiplication of UHE by {{nowrap|1 − ''a''}} above, the UHE would be flipped upside down relative to what we're used to, with higher values corresponding to more concordant intervals. In the pictures below we have flipped it back upside down, for consistency with the earlier pictures.

==== ''s'' {{=}} 0.5%, ''a'' {{=}} 1.00001 ====

==== ''s'' {{=}} 1%, ''a'' {{=}} 1.00001 ====

==== ''s'' {{=}} 1.5%, ''a'' {{=}} 1.00001 ====

Note that in all these plots, the value of ''a'' is chosen to be 1.00001 rather than exactly ''1'', so as to prevent the ({{nowrap|1 − ''a''}}) term becoming 0. Similar results are seen for other choices of ''a'':

==== ''s'' {{=}} 1%, ''a'' {{=}} 2.2 ====

Note that you have to be careful if you choose to compute the analytically continued {{nowrap|''a'' {{=}} 2}} numerically: this corresponds to the vertical slice of zeta function at {{nowrap|Re(''z'') {{=}} 1}}, where there is a pole.

Now let's consider the auxiliary function {{nowrap|Ũ(''c'') {{=}} U(''c'') − U(0)}}, which gives us a "shifted" version of the UHE which has the UHE of 1/1 normalized to 0. Then we can re-express the UHE as follows:

Lastly, suppose we only care about the entropy function up to a vertical shift and scaling: in other words, we want to declare two functions (''f''(''x''),&nbsp;''g''(''x'') to be '''linearly equivalent''', and write {{nowrap|''f''(''x'') ≈ ''g''(''x'')}}, if for some ''a'',&nbsp;''b'' that don't depend on ''x'', we have {{nowrap|''f''(''x'') {{=}} ''a g''(''x'') + ''b''}}. This means we want to view two entropy functions as equivalent if one is just a scaled and shifted version of the other, so that when "normalizing" them (so that the entropy goes from 0 to 1), we get identical functions. Then we have all of the following relationships:

where we have just dropped the constants of {{sfrac|1|1 − ''a''}} and the constant vertical shift of U(0) which doesn't depend on ''c''.

Now, the main thing is that, if we are in the region where {{nowrap|''a'' &le; 2}}, then this is also the region where the U(0) term goes to infinity as ''N'' increases: the entropy doesn't converge. And in general, we have the asymptotic expansion

and, for large ''k'', '''as long as''' {{nowrap|''x'' ≪ ''k''}}, the higher-order terms become negligible. This means, for all ''c'', we would need to show that {{nowrap|Ũ(''c'') ≪ U(0)}} as {{nowrap|''N'' → ∞}}. We would then be able to rewrite the above as

This means that, as {{nowrap|''N'' → ∞}}, we would also get the following linear equivalence:

Putting this with our earlier result that {{nowrap|U(''c'') ≈ Ũ(''c'')}}, we get

We can perform the same substitution of {{nowrap|U(''c'') {{=}} U(0) + Ũ(''c'')}} again to get

Given that {{nowrap|''a'' ≠ 0}}, we have a Taylor expansion of {{nowrap|(''k'' + ''x'')^{{{sfrac|1|1 − ''a''}}}}} around {{nowrap|''x'' {{=}} 0}} of the form

where we have factored out the leading term for clarity. Now, again, we have that if {{nowrap|''x'' ≪ ''k''}}—meaning that {{nowrap|Ũ(''c'') ≪ U(0)}}—then the higher-order terms become negligible, and we simply have the asymptotic relationship of

So that, overall, as {{nowrap|''N'' → ∞}}, the following equivalence holds:

And since we have already shown that {{nowrap|UHE_''a''(''c'') ≈ Ũ(''c'')}}, we have

Now, the only missing piece needed for all of this is to show that we really do have {{nowrap|Ũ(''c'') ≪ U(0)}} in the region of interest. For now, absent mathematical proof, we will simply plot the behavior for the Shannon entropy as {{nowrap|''N'' → ∞}}.

What we see is that, while the function diverges, it diverges in a certain "uniform" sense. That is, as ''N'' increases, a constant vertical offset is added to U(''c''), so that the function blows up to infinity. However, if this vertical offset is corrected for, for example by subtracting U(0), the resulting curve doesn't seem to grow at all, but rather shrinks in height slightly until it seems to converge. We would like to prove this formally, but for now, we can at least see this from the following plot:

In other words, we can see that as ''N'' increases, the growth rate of U(0) dwarfs that of Ũ(''c'')</math>, which does not seem to grow at all.

So, this is a fairly weak conjecture to make, given that empirical evidence suggests something much stronger—that not only does it grow more slowly, but that it seems to not grow at all—it converges! In particular, it seems to converge on our analytic continuation from before. However, a strict proof of any of these things would be nice.

Note again that this does not hold for {{nowrap|''a'' &gt; 2}}, where the graph does display a very large difference between UHE and exp-UHE.

=== Why not normalized HE? ===