Received: by alpheratz.cpm.aca.mmu.ac.uk id PAA20532 (8.6.9/5.3[ref pg@gmsl.co.uk] for cpm.aca.mmu.ac.uk from fmb-majordomo@mmu.ac.uk); Mon, 18 Mar 2002 15:32:24 GMT Message-ID: <004f01c1ce91$0066c740$cf867018@bloor.phub.net.cable.rogers.com> From: "Ray Haraf" <rayharaf@rogers.com> To: <entropy@mmu.ac.uk> Subject: ENTROPY: Calculating Information Loss in Four-Way Tables Date: Mon, 18 Mar 2002 10:24:30 -0500 Content-Type: multipart/alternative; boundary="----=_NextPart_000_004C_01C1CE67.1756C380" X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2600.0000 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2600.0000 X-Authentication-Info: Submitted using SMTP AUTH LOGIN at fep01-mail.bloor.is.net.cable.rogers.com from [24.112.134.207] using ID <rayharaf@rogers.com> at Mon, 18 Mar 2002 10:26:34 -0500 Sender: fmb-majordomo@mmu.ac.uk Precedence: bulk Reply-To: entropy@mmu.ac.uk
Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable
Dear All,
I'll be very appreciative of your help to deal with the following. Given a RxCxLxS table A, I merge either two rows, two columns, two layers or two strata according to merge criteria. Resulting table B, say, is still a RxCxLxS i.e. four-way table. I would like to assess information loss relative to move from A to B. I claim Information Loss = Entropy(A) - Entropy(B). However, Freitas (http://www.ppgia.pucpr.br/~alex/thesis.html) defined Information loss as Info_After_Merge - Info_Before_Merge where Info_After_Merge = -SUMj(C+j/C++)log(C+j/C++) and Info_Before_Merge = -SUMi(Ci+/C++)SUMj(Cij/Ci+)logCij/Ci+). This turns out to be, in two-way contingency table, Column(entropy) - Column(entropy given row). (According to definitions & formulae in Statistical Decomposition Analysis, Theil, 1972).
What is the rational for considering Column(entropy after merge) vs. Column(entropy given row before merge)? I would go for either Column(entropy after merge) - Column(entropy before merge) or Column(entropy given row after merge) - Column(entropy given row before merge). I might ignore that Column(entropy given row) = Column(entropy) though! Most importantly, how can this be extended to a four-way table?
Next, in case of two-way table Theil defines the expected mutual information as the amount to be subtracted from the total marginal entropy, Column(entropy) + Row(entropy), in order to obtain the joint entropy (or coentropy). Applied to three-way XYZ table this definition should give XYY(coentropy) = XY(marginal entropy) + XZ(marginal entropy) + YZ(marginal entropy) + Column(entropy) + Row(entropy) + Layer(entropy) - XYZ(mutual information). Virtanen and Astola (http://www.uwasa.fi/~itv/publicat/entropy.html) established I(XYZ) = H(XYZ) - H(XY) - H(YZ) - H(ZX) + H(X) + H(Y) + H(Z) or H(XYZ) = I(XYZ) + H(XY) + H(YZ) + H(ZX) - H(X) - H(Y) - H(Z). i.e. mutual information is not subtracted; it is added instead as H(XYZ) = XYZ(coentropy), H(XY) = XY(marginal entropy), H(XZ) = XZ(marginal entropy), H(YZ) = YZ(marginal entropy), H(Y) = Column(entropy), H(X) = Row(entropy), H(Z) = Layer(entropy), and
I(XYZ) = XYZ(mutual information). But the formula I(XYZ) = -SUMiSUMjSUMk(Pijk*log(Pij.*Pi.k*P.jk/Pijk*Pi..*P.j.*P..k) (in Virtanen and Astola) leads to I(XYZ) = H(XY) + H(YZ) + H(ZX) - H(XYZ) - H(X) - H(Y) - H(Z). Here mutual information and one-dimensional entropies are subtracted from the total two-dimensional marginal entropy. Question is what the right answer.
Generalizing the formula I(XYZ) = -SUMiSUMjSUMk(Pijk*log(Pij.*Pi.k*P.jk/Pijk*Pi..*P.j.*P..k)) to four-way case, gives
I(WXYZ) = -SUMiSUMjSUMkSUMm(Pijkm*log(Pijk.*Pij.m*Pi.km*P.jkm/Pijkm*Pi...*P.j..*P..k.*P...m)) whose expansion does not involve two-dimensional marginal entropies. Could someone gives the right formula?
What would be good recent references to consult for entropy concept applied to multiway tables?
Thanks in advance for your valuable help and suggestions,
Ray Haraf.
This archive was generated by hypermail 2b29 : Mon Mar 18 2002 - 15:32:27 GMT