Go to:Table of Contents

Mutual Information

Let us consider the following differences

$$I(X\wedgeY)=H(X)-H(X\vert Y)=\sum_{i=1}^n{\sum_{j=1}^m{p(x_i,y_j)\log\Big({p(x_i,y_j)\over p(x_i)}\Big)}},$$ $$I(X\wedgeY\vert Z)=H(X\vert Z)-H(X\vert Y,Z)=\sum_{i=1}^n{\su......l{p(x_i,y_j,z_k)\log\Big({p(x_i,y_j\vert z_k)\over p(x_i\vert z_k)}\Big)}}},$$ and $$I\big((X,Y)\wedge Z \big)=H(X,Y)-H(X,Y\vert Z)=\sum_{i=1}^n{\su......k=1}^l{p(x_i,y_j,z_k)\log\Big({p(x_i,y_j\vert z_k)\over p(x_i,y_j)}\Big)}}}.$$

In view of the property 1.40, the following property holds.

Property 1.47. We have

(i) $I(X\wedge Y) \geq 0$ with equality iff $X$ and $Y$ are independent.

(ii) $I(X\wedge Y\vert Z) \geq 0$ with equality iff $X$ and $Y$ are conditionally independent given $Z$.

(iii) $I\big((X,Y)\wedge Z \big) \geq 0$ with equality iff $(X,Y)$ and $Z$ are independent.

The above measures are famous in the literature as

$I(X\wedge Y)$ : mutual information among the random variables $X$ and $Y$;

$I(X\wedge Y\vert Z)$ : mutual information among the random variables $X$ and $Y$ given $Z$;

$I\big((X,Y)\wedge Z \big)$ : mutual information among the random variables $(X,Y)$ and $Z$.

We can write 

$$I(X\wedge Y)=E_Y \big[I(X\wedge Y=y_j)\big] = E_{XY} \big[ I(X=x_i \wedge Y=y_j) \big],$$ where $$I(X\wedge Y=y_j)=\sum_{i=1}^n{p(x_i\vert y_j)\log\Big({p(x_i\vert y_j)\over p(x_i)} \Big)}, \ \forall \ j$$ and $$I(X=x_i \wedge Y=y_j) = \log \Big({p(x_i,y_j)\over p(x_i)p(y_j)}\Big), \ \forall \ i,j$$ In view of the above representation, $I(X\wedge Y)$ sometimes understood as the average or the conditional mutual information of $I(X \wedge Y=y_j)$ or of$I(X=x_i \wedge Y=y_j)$. The expression $I(X=x_i \wedge Y=y_j)$ is known as per-letter information, and it may be negative. It is nonnegative iff $p(x_i,y_j) \geq p(x_i)p(y_j), \ \forall \i,j.$ Similarly, we can write expressions for $I(X\wedge Y\vert Z)$ and$I\big((X,Y)\wedge Z \big)$. In fact, the following properties hold.

Property 1.48. We have

(i) $E_X \big[ I(X=x_i\wedge Y)\big] = E_Y\big[ I(X\wedge Y=y_j)\big] .$

(ii) $E_{YZ} \big[ I(X\wedge Y=y_j\vert Z=z_k)\big] = E_{XZ}\big[I(X=x_i\wedge Y\vert Z=z_k)\big] .$

(iii) $E_Z \big[ I\big((X,Y)\wedge Z=z_k\big)\big] = E_{XY}\big[ I\big((X=x_i,Y=y_j)\wedge Z\big) \big] .$

Property 1.49. We have

(i) $H(X)+H(Y)-H(X,Y)=I(X\wedge Y).$

(ii) $H(X)+H(Y)+H(X,Y\vert Z)=I(X\wedge Y)+I\big((X,Y)\wedge Z \big).$

(iii) $I(X\wedge Z)+I(X\wedge Y\vert Z)=I(X\wedge Y)+I(X\wedge Z\vert Y)=I\big((X,Y)\wedge Z \big).$

Property 1.50. We have

(i) $I(X\wedge Y)$ is a convex $\cap$ function of the probability distribution $\{p(x_i)\}\ \in\ \Delta_n$.

(ii) $I(X\wedge Y)$ is a convex $\cup$ function of the probability distribution $\{p(y_j\vert x_i)\}\ \in\ \Delta_m$.

Property 1.51. We have

(i) $I\big((X,Y)\wedge Z\big) \geq I(Y\wedge Z)\ \big({\rm or}\I(X\wedge Z)\big)$, with equality iff $X$ (or resp. $Y$) and $Z$ are conditionally independent given Y.

(ii) $I\big((X,Y)\wedge Z\big) \geq I(X\wedge Z\vert Y)\ \big({\rm or}\I(Y\wedge Z\vert X)\big)$, with equality iff $Y$ (or resp. $X$) and $Z$ are independent.

Property 1.52. If $X$ and $Y$ are conditionally independent given $Z$, then

(i) $I(X\wedge Z) \geq I(X\wedge Y)\ ({\rm or}\ I(Y\wedge Z)).$

(ii) $I(X\wedge Z) \geq I(X\wedge Z\vert Y).$

Property 1.53. We have

(i) $I\big((X_1,X_2)\wedge X_3\vert X_4\big)=I(X_1\wedge X_3\vert X_4)+I\big(X_2\wedge X_3\vert(X_1,X_4)\big).$

(ii) $I\big((X_1,X_2)\wedge X_3\vert X_4\big)+I(X_1\wedge X_2\vert X_4)=I\big(X_2\wedge (X_1,X_3)\vert X_4)\big)+I(X_1\wedge X_3\vert X_4).$

(iii) $I\big(X_1\wedge (X_2,X_3)\big)+I\big(X_1\wedge X_4\vert(X_2,X_3)\big)=I\big(X_1\wedge (X_2,X_3,X_4)\big).$

(iv) $I\big((X_1,X_2)\wedge (X_3,X_4,X_5)\big)=I\big((X_1,X_2)\wedge X_3\vert(X_4,X_5)\big)$

$+I\big((X_1,X_2)\wedgeX_4\vert X_5)\big)+I\big((X_1,X_2)\wedge X_5\big).$