tag:blogger.com,1999:blog-6837159629100463303.post1948662850145490717..comments2023-06-03T07:19:32.501-07:00Comments on Information Transfer Economics: Draft paper for a talk this summerJason Smithhttp://www.blogger.com/profile/12680061127040420047noreply@blogger.comBlogger40125tag:blogger.com,1999:blog-6837159629100463303.post-5365741682464172552016-02-02T17:09:25.139-08:002016-02-02T17:09:25.139-08:00Hey, congratulations on the talk being accepted! I...Hey, congratulations on the talk being accepted! I just now noticed.Tom Brownhttps://www.blogger.com/profile/17654184190478330946noreply@blogger.comtag:blogger.com,1999:blog-6837159629100463303.post-44485768572359878722016-01-16T12:39:22.345-08:002016-01-16T12:39:22.345-08:00I think as it was written, it was ambiguous at bes...I think as it was written, it was ambiguous at best. So it's really my fault.Jason Smithhttps://www.blogger.com/profile/12680061127040420047noreply@blogger.comtag:blogger.com,1999:blog-6837159629100463303.post-67408205707707112402016-01-16T12:31:24.720-08:002016-01-16T12:31:24.720-08:00Yep, I just misinterpreted your sentence.Yep, I just misinterpreted your sentence.Tom Brownhttps://www.blogger.com/profile/17654184190478330946noreply@blogger.comtag:blogger.com,1999:blog-6837159629100463303.post-56494984631094990012016-01-16T09:29:58.790-08:002016-01-16T09:29:58.790-08:00QAM16 follows the Shannon limit (my comment about ...QAM16 follows the Shannon limit (my comment about bandwidth,SNR) better than BPSK ... that means QAM16 is more spectrally efficient.<br /><br />Maybe my last sentence seems to suggest the opposite direction in terms of channel capacity. But I was talking about what needed to be done to your BPSK signal to match up with a QAM16 without changing your bit error rate.Jason Smithhttps://www.blogger.com/profile/12680061127040420047noreply@blogger.comtag:blogger.com,1999:blog-6837159629100463303.post-9165343926571700592016-01-15T23:44:03.920-08:002016-01-15T23:44:03.920-08:00"Actually BPSK to QAM16 represents exactly th..."Actually BPSK to QAM16 represents exactly that binary to hex conversion ... in information theory, your SNR has to go up as well as your bandwidth."<br /><br />Jason, do you think <a href="http://users.ecs.soton.ac.uk/rm/resources/matlabcapacity/" rel="nofollow">this is correct</a>? If so, then QAM16 appears to always be at least as spectrally efficient as BPSK.<br /><br />It seems to match <a href="https://en.wikipedia.org/wiki/Phase-shift_keying#/media/File:Channel_capacity_for_complex_constellations.svg" rel="nofollow">this one</a> pretty closely.<br /><br />True for a given SNR, the QAM16 symbols are more difficult to resolve (i.e., more likely to be misread), but perhaps this is offset by more bits/symbol?Tom Brownhttps://www.blogger.com/profile/17654184190478330946noreply@blogger.comtag:blogger.com,1999:blog-6837159629100463303.post-31703901503618061592016-01-15T20:46:18.413-08:002016-01-15T20:46:18.413-08:00Not sure yet. Going to let it sit in my head for a...Not sure yet. Going to let it sit in my head for awhile and come back to it.Jason Smithhttps://www.blogger.com/profile/12680061127040420047noreply@blogger.comtag:blogger.com,1999:blog-6837159629100463303.post-69645907682103414742016-01-15T20:44:41.583-08:002016-01-15T20:44:41.583-08:00The difference is a matter of notation. Instead of...The difference is a matter of notation. Instead of H, we use I and instead of E[...] we would use < ... >, except we leave the ensemble averages off most of the time ... and F&B explicitly show the "thermodynamic limit" where<br /><br />V = < V ><br /><br />which really only makes sense as an ensemble average of an operator (individual atoms have no volume [in the model], individual people have no 'demand').<br /><br />In discussion with Peter F., we thought it best to leave off some of the notational clutter. I even wanted to drop the absolute value signs.<br /><br />Regarding your question about the KL divergence, the idea of a distribution becomes fuzzy since one exists over a different period of time (or at a different selection rate) than the other.<br /><br />The KL divergence makes the most sense when the two distributions are close to each other (P ~ Q) and the symbol rate is the same.<br /><br />Actually BPSK to QAM16 represents exactly that binary to hex conversion ... in information theory, your SNR has to go up as well as your bandwidth.Jason Smithhttps://www.blogger.com/profile/12680061127040420047noreply@blogger.comtag:blogger.com,1999:blog-6837159629100463303.post-40017177219216871042016-01-15T17:48:56.608-08:002016-01-15T17:48:56.608-08:00This:
E[I(d)] ≥ E[I(s)]
Is actually saying somet...This:<br /><br />E[I(d)] ≥ E[I(s)]<br /><br />Is actually saying something different than<br /><br />I(d) ≥ I(s)<br /><br />which is what the draft paper said, right? And <br /><br />E[I(d)] = E[I(s)]<br /><br />is a weaker condition than <br /><br />I(d) = I(s)<br /><br />isn't it? However, looking at latest P. Fielitz and G. Borchard paper, it appears they state this without the E[] wrapped around it.<br /><br />As we discuss above, it's possible to have two distributions with equal information entropy which nonetheless represent an information loss because the distributions for source and destination themselves differ in a way which guarantees a non-zero expected loss of information (i.e. there's a KL divergence).<br /><br />BTW, it seems that distributions which differ by only a scale factor do not necessarily entail a loss of information (i.e. uniform over the set of binary symbols on one side of the channel and uniform over the set of hex symbols on the other side of the channel).<br /><br />So how would one apply the KL-divergence formula in that case (binary on one side and hex on the other) if the channel/system itself was bundling up sets of four contiguous binary symbols (say) and replacing them with hex symbols? It seems that when presented with such a system, we may be justified in using the value of k to modify the one distribution in relation to the other prior to calculating D?<br /><br />For the non-uniform Bernoulli case (above) it appears that<br /><br />D(P||Q) = 0.1*log2(0.1/0.9) + 0.9*log2(0.9/0.1) = 2.5359 bits lost due to distribution mismatches?<br /><br />We know that k = 1 here, so scaling one distribution relative to the other isn't going to help. Like you point out, we could redefine heads and tails on one side and fix that, but if we don't do that, then this is a lossy system.Tom Brownhttps://www.blogger.com/profile/17654184190478330946noreply@blogger.comtag:blogger.com,1999:blog-6837159629100463303.post-19804996221324351802016-01-15T16:33:16.257-08:002016-01-15T16:33:16.257-08:00BTW, is this the end of your draft presentation? A...BTW, is this the end of your draft presentation? Are you planning on adding any more? For 30 to 40 minutes I'd think you have plenty.Tom Brownhttps://www.blogger.com/profile/17654184190478330946noreply@blogger.comtag:blogger.com,1999:blog-6837159629100463303.post-9894190148185857872016-01-15T16:31:41.460-08:002016-01-15T16:31:41.460-08:00I did a quick scan through your latest draft paper...I did a quick scan through your latest draft paper to see if you used the information entropy (H) concept (at least early on, when first introducing the ideas), but I didn't see it. I found that a helpful addition here. Would it make sense to introduce it there as well? I like seeing the H = E{I} development here.Tom Brownhttps://www.blogger.com/profile/17654184190478330946noreply@blogger.comtag:blogger.com,1999:blog-6837159629100463303.post-46834010025513638322016-01-12T17:18:36.170-08:002016-01-12T17:18:36.170-08:00Yes, of course: k = 4. I confused myself: 16 is th...Yes, of course: k = 4. I confused myself: 16 is the size of the hex alphabet NOT the number of binary symbols required to make one hex symbol (i.e. not n1 vs n2). So n1 = n2/4 in that case, and that I was almost correct for the apples vs pies case (I just got it turned around): n1 = n2/5 for the apples vs pies case I think.<br /><br />This latter implies k = log(sigma1)/log(sigma2) = 5, or that sigma1 = sigma2^5. So if there are two "apple symbols" (whatever those could be), then there are 32 "pie symbols."Tom Brownhttps://www.blogger.com/profile/17654184190478330946noreply@blogger.comtag:blogger.com,1999:blog-6837159629100463303.post-59084022737854056122016-01-12T12:09:12.106-08:002016-01-12T12:09:12.106-08:00I guess it's not really information loss -- it...I guess it's not really information loss -- it's just information loss if you look at single events. But k kind of adjusts for that so in the binary/hex case above, k = 1/4 (or k = 4, the other way) and you need a sequence with 4 times as many events (or 1/4 as many) to get the same amount of information.<br /><br />The noise in Shannon's diagram would be measurement error as well as any other source of information loss (poor market design, coordination, etc).Jason Smithhttps://www.blogger.com/profile/12680061127040420047noreply@blogger.comtag:blogger.com,1999:blog-6837159629100463303.post-76531629855190226052016-01-11T16:29:26.532-08:002016-01-11T16:29:26.532-08:00"then n₁ = 16*n₂, and if supply is apples and..."then n₁ = 16*n₂, and if supply is apples and demand is pies, then n₁ = 5*n₂. Is that right?" ... Hmmm, no, let me retract that one and think about it a bit more.Tom Brownhttps://www.blogger.com/profile/17654184190478330946noreply@blogger.comtag:blogger.com,1999:blog-6837159629100463303.post-58765505621640289442016-01-11T16:27:25.827-08:002016-01-11T16:27:25.827-08:00Ah!... so with your binary vs hex or your apples v...Ah!... so with your binary vs hex or your apples vs apple pies example, that sounds like a case (if the distributions for supply and demand are both uniform) where (if H(d) = H(s)) n₁ =/= n₂, true? In fact if supply is binary and demand hex, then n₁ = 16*n₂, and if supply is apples and demand is pies, then n₁ = 5*n₂. Is that right? Also, k =/= 1 for these cases, right?<br /><br />BTW, this is great! Thanks very much!!... great examples. I guess I "fooled myself" into thinking I was comfortable with this bit before, but there were some bits I never fully grasped.<br /><br />Also, you write:<br /><br />"I think a better way to say it is that k accounts for a constant information loss between the distributions"<br /><br />So, there can be information loss but H(d) still equal to H(s)? (i.e. still in information equilibrium?)<br /><br />BTW, in your Shannon channel diagram, what corresponds to the additive noise in the economics case? Does that correspond to (overwhelming likely destructive) coordination (i.e. group think, and panics)? <br /><br />In a radio/electronic communications channel, I can't think of a case where additive thermal noise could possibly improve the information content (improve the constellation): at best I can see it not disturbing anything, and that, of course, is highly unlikely.Tom Brownhttps://www.blogger.com/profile/17654184190478330946noreply@blogger.comtag:blogger.com,1999:blog-6837159629100463303.post-24660893622643883952016-01-11T15:30:48.149-08:002016-01-11T15:30:48.149-08:00But there is another ambiguity: same information w...But there is another ambiguity: same information with just more instances of symbols. For example, the strings:<br /><br />1111111100010001<br /><br />and <br /><br />FF11<br /><br />contain the same information (binary, hexadecimal). We may consider that several supply widgets are required to satisfy a single demand widget (I need 5 apples to make a pie).Jason Smithhttps://www.blogger.com/profile/12680061127040420047noreply@blogger.comtag:blogger.com,1999:blog-6837159629100463303.post-17637736675429528222016-01-11T15:13:34.895-08:002016-01-11T15:13:34.895-08:00I think a better way to say it is that k accounts ...I think a better way to say it is that k accounts for a constant information loss between the distributions if they are different but k = 1 is ambiguous and could mean either the distributions are the same or that they are complements of each other with the same information. I.e. k = 1 means that each flip of two unfair coins comes up heads equally often or one comes up tails and the other heads equally often.<br /><br />Essentially, your two distributions with p1=0.1 and p2=0.9 are the same distribution, just with different labels (i.e. switch heads with tails and p2 becomes 0.1).Jason Smithhttps://www.blogger.com/profile/12680061127040420047noreply@blogger.comtag:blogger.com,1999:blog-6837159629100463303.post-64417498232630111352016-01-11T14:58:46.424-08:002016-01-11T14:58:46.424-08:00And if σ₁ =/= σ₂, even though the distributions ar...And if σ₁ =/= σ₂, even though the distributions are both uniform, they're not the same.Tom Brownhttps://www.blogger.com/profile/17654184190478330946noreply@blogger.comtag:blogger.com,1999:blog-6837159629100463303.post-1451159740887193932016-01-11T14:55:48.402-08:002016-01-11T14:55:48.402-08:00"The distributions have to be the same (just ..."The distributions have to be the same (just not necessarily uniform, as was your original question) otherwise there is a non-zero KL divergence (information loss)"<br /><br />If they have to always be the same (for H(d) = H(s)?), then how could k ever be any other value than 1? Isn't it possible that H(d) = H(s) and k =/= 1? <br /><br />If k =/= 1, then n₁ and n₂ can be such that H(d) is still equal to H(s) (i.e. for some n₁ =/= n₂). <br /><br />Even if both distributions are uniform, σ₁ may not equal σ₂ (which would again require n₁ =/= n₂ if H(d) = H(s), correct?).Tom Brownhttps://www.blogger.com/profile/17654184190478330946noreply@blogger.comtag:blogger.com,1999:blog-6837159629100463303.post-5189733392272952892016-01-11T11:15:47.821-08:002016-01-11T11:15:47.821-08:00I probably should have just written p instead of p...I probably should have just written p instead of p1 and p2. The distributions have to be the same (just not necessarily uniform, as was your original question) otherwise there is a non-zero KL divergence (information loss).<br /><br />With the choices you selected for p1 and p2, you've matched up no-demand with yes-supply and yes-demand with no-supply. These sequences do have the same amount of information (like two binary sequences determined from a coin flip where the tails are 0 for one versus the heads are 0 for the other).Jason Smithhttps://www.blogger.com/profile/12680061127040420047noreply@blogger.comtag:blogger.com,1999:blog-6837159629100463303.post-51517979125598688762016-01-11T00:48:50.589-08:002016-01-11T00:48:50.589-08:00You write:
"k = (- p1 log p1 - (1 - p1) log ...You write:<br /><br />"k = (- p1 log p1 - (1 - p1) log (1 - p1))/(- p2 log p2 - (1 - p2) log (1 - p2))"<br /><br />So if p1 = 0.1 and p2 = 0.9 and p1 represents the probability for demand for a widget at location 0 (as opposed to location 1), and p2 represents the probability for supply of a widget at location 0 (as opposed to location 1), then k = 1. However probability function P1 =/= probability function P2.<br /><br />Now given this k (i.e. k=1) (and suppose n₁, n₂ >> 1 as well), then for there to be information equilibrium (i.e. H(d) = H(s)), it's necessary that n₁ = n₂. But is that likely given that P1 =/= P2?<br /><br />I'm not sure what I'm saying/asking here: I'm just trying to make it all a bit more concrete in my brain. It somehow seems that unmet demand for widgets is going to pile up at location 1 and/or unneeded supply for widgets is going to pile up at location 0. Is that consistent with n₁ = n₂? It seems like I took a wrong turn here somewhere. You may be a person of the concrete steppes Jason, but I'm a proud member of the concrete craniums!Tom Brownhttps://www.blogger.com/profile/17654184190478330946noreply@blogger.comtag:blogger.com,1999:blog-6837159629100463303.post-37650995481993962752016-01-09T10:37:12.886-08:002016-01-09T10:37:12.886-08:00No idea. Probably 30-40 minutes plus time for ques...No idea. Probably 30-40 minutes plus time for questions as is typical for conferences. The paper length doesn't necessarily correlate with the talk length.Jason Smithhttps://www.blogger.com/profile/12680061127040420047noreply@blogger.comtag:blogger.com,1999:blog-6837159629100463303.post-13439040952617465542016-01-09T10:34:40.839-08:002016-01-09T10:34:40.839-08:00Hi Tom,
Messages are just strings of symbols. In ...Hi Tom,<br /><br />Messages are just strings of symbols. In our formulation, supply and demand are just messages (strings of symbols). I imagine those messages are like grid points like f6 or b2 on a Chess board (one dimension being time and the other space if you'd like). The demand is then some string like<br /><br />c8d3c2d4a6a3h7d4h4a6<br /><br />and the supply is some string like<br /><br />g4d2d5b7c8g2d4e5a4c1<br /><br />As the strings get long enough, there is a match (transaction) for every chess board location listed.<br /><br />The X in H(X) is a random variable; it doesn't have a value. Specific probabilities are frequently written as P(X = x) where X is a random variable and x is a specific value. For H(X), however, the notation H(X = x) is nonsense as H is a property of the distribution of X (i.e. X itself).<br /><br />It is not actually necessary to assume uniform distribution of symbols for the derivation; it just changes the definition of k such that it depends on more than one probability value ... e.g.<br /><br />k = (- p1 log p1 - (1 - p1) log (1 - p1))/(- p2 log p2 - (1 - p2) log (1 - p2))<br /><br />for two <a href="https://en.wikipedia.org/wiki/Binary_entropy_function" rel="nofollow">Bernoulli processes</a><br /><br />The s and d are random variables, the S and D are real variables related to the n's ... n1 = D/dD.Jason Smithhttps://www.blogger.com/profile/12680061127040420047noreply@blogger.comtag:blogger.com,1999:blog-6837159629100463303.post-24365697097755476052016-01-08T19:04:25.057-08:002016-01-08T19:04:25.057-08:00When I ask "are messages = symbols" I...When I ask "are messages = symbols" I'm asking can these 2D distributions just as well be labeled "Transmitted distribution of symbols" and "Received distribution of messages?" Or perhaps "Distribution symbols transmitted" and "Distribution of symbols received?" Just off hand it seems like a message might consist of a sequence of transmitted or received symbols, but is it important which is used in the labels in that figure?Tom Brownhttps://www.blogger.com/profile/17654184190478330946noreply@blogger.comtag:blogger.com,1999:blog-6837159629100463303.post-66144451488910174682016-01-08T18:55:33.555-08:002016-01-08T18:55:33.555-08:00Is the notion H(X) meant to be a scalar for any ra...Is the notion H(X) meant to be a scalar for any random variable X? (i.e. not taking on different values for specific values of a single random variable X, but rather a scalar value for any one X defined by distribution P)?<br /><br />Are d and s in H(d) and H(s) quantity demanded and quantity supplied? What's the significance of switching to the upper case with D and S later?<br /><br />Also, to derive:<br /><br />dD/dS = k D/S<br /><br />it was necessary to assume a uniform distribution over all σ₁ demand 'symbols' and over all σ₂ supply 'symbols' (I'm not exactly sure how you define a symbol here), but then you discuss a spatial/temporal distribution which in <a href="http://2.bp.blogspot.com/-H0ohO_s8v04/VpAVsKgoy3I/AAAAAAAAIeo/1ajUaZomquM/s1600/shannon.PNG" rel="nofollow">this figure</a> (labeled "Transmitted distribution of messages" and "Received distributions of messages" resp. [are messages = symbols?]), appear to be non-uniform. Is there some relationship between these uniform and non-uniform probability distributions?<br /><br />Also, you have two derivations of demand curves with your approach (<a href="http://1.bp.blogspot.com/-r5taxvfkM2A/Vo7CkaXNWuI/AAAAAAAAIdA/JB92fdxVcRw/s1600/itm%2Bmonkey%2B1.png" rel="nofollow">here</a> and <a href="http://3.bp.blogspot.com/-6xIhdMxWWZE/VpAbo6HADRI/AAAAAAAAIe4/RmvxGlDQaCE/s1600/sandd1.png" rel="nofollow">here</a>). What is the relationship between these derivations?Tom Brownhttps://www.blogger.com/profile/17654184190478330946noreply@blogger.comtag:blogger.com,1999:blog-6837159629100463303.post-87976383593115195052016-01-08T16:15:10.290-08:002016-01-08T16:15:10.290-08:00How long do you anticipate they'll give you (s...How long do you anticipate they'll give you (should it happen)? Does that include time for Q&A?Tom Brownhttps://www.blogger.com/profile/17654184190478330946noreply@blogger.com