Go to:Table of Contents

Relative Information and Inaccuracy

Kullback and Leibler's (1951) [65] measure of information associated with the probability distributions $P$ and $Q$ is given by

$$D(P\vert\vert Q)=\sum_{i=1}^n{p_i\log {p_i\over q_i}}.$$ (2.1)

The measure (2.1) has many names given by different authors such as, relative information, directed divergence, cross entropy, function of discrimination etc.. Here we shall refer it "relative information". It has found many applications in setting important theorems in information theory and statistics.

The Kerridge's (1961) [63] measure of information generally referred as inaccuracy associated with two probability distributions is given by

$$H(P\vert\vert Q)=\sum_{i=1}^n{p_i\log q_i}.$$ (2.2)

Various authors studied characterizations and properties of the measures (2.1) and (2.2) separately. Here we present their joint study.

Let us consider a measure

$${\ensuremath{\boldsymbol{\mathscr{C}}}}(P\vert\vert Q)=A\ \sum_{i=1}^n{p_i\log p_i}+ B\\sum_{i=1}^n{p_i\log q_i}.$$ (2.3)

Then for $A=1$, $B=-1$, we get (2.1) and for $A=0$, $B=-1$, we get (2.2).

For simplicity, let define 

$$\delta_n =\Big\{P=(p_1, p_2, \cdots, p_n)\vert p_i > 0, \sum_{i=1}^n p_i\leq1\Big\}.$$ The following theorem give axiomatic characterization of the measure (2.3)

Theorem 2.1. Let ${\ensuremath{\boldsymbol{\mathscr{C}}}} :\Delta_n \times\delta_n \rightarrow I\!\!R$ (reals) be a function satisfying the following axioms:
 

A1. (Symmetry). ${\ensuremath{\boldsymbol{\mathscr{C}}}}(P\vert\vert Q), \ (P,Q)\ \in\ \Delta_n\times\delta_n$ is symmetric for every permutation of elements in Q.

A2. (Branching). We have

$${\ensuremath{\boldsymbol{\mathscr{C}}}}(p_1,...,p_n\vert\vert q_1......th{\boldsymbol{\mathscr{C}}}}(p_1+p_2,p_3,...,p_n\vert\vert q_1+q_2,q_3...,q_n)$$ $$=\phi(p_1,p_2\vert\vert q_1,q_2),\ n=3,4,...$$ where $\phi$ is a continuous function defined over $\Delta_2 \times \delta_2$.

A3. (Additivity). We have

$${\ensuremath{\boldsymbol{\mathscr{C}}}}(P_1*Q_1\vert\vert P_2*Q_2......(P_1\vert\vert Q_1)+{\ensuremath{\boldsymbol{\mathscr{C}}}}(P_2\vert\vert Q_2),$$ where $Q_1=(q_1,1-q_1)$, $Q_2=(q_2,1-q_2)$,$P_1*Q_1=\big( p_{11}q_1,p_{12}(1-q_1),...,p_{1n}q_1,$$p_{1n}(1-q_1)\big)$,

etc..

Then ${\ensuremath{\boldsymbol{\mathscr{C}}}}(P\vert\vert Q)$ is given by (2.3).

By considering ${\ensuremath{\boldsymbol{\mathscr{C}}}}(p,1-p\vert\vert p,1-p)=0,\ \ 0<p<1$ and ${\ensuremath{\boldsymbol{\mathscr{C}}}}(1,0\vert\vert{1\over 2},{1\over 2})=1$ in (2.3) we get (2.1). Again taking ${\ensuremath{\boldsymbol{\mathscr{C}}}}(p_1,p_2,p_3\vert\vert q_1,q_2,q_2)= {\ensuremath{\boldsymbol{\mathscr{C}}}}(p_1,p_2+p_3\vert\vert q_1,q_2)$ and ${\ensuremath{\boldsymbol{\mathscr{C}}}}({1\over 2},{1\over2}\vert\vert{1\over 2},{1\over 2})=1$ we get (2.2).

Measure (2.3) can also be characterized by different approaches using functional equation or axiomatic aprroach. In functional's equation approach, the following two equations are frequently used: 

$$\sum_{i=1}^n{\sum_{j=1}^m{f(p_{1i},p_{2j},q_{1i},q_{2j})}}=\sum_{i=1}^n{f(p_{1i},q_{1i})}+\sum_{j=1}^m{f(p_{2j},q_{2j})},$$ and $$\psi(x_1,y_1)+(1-x_1)\psi\Big({x_2\over 1-x_1},{y_2\over1-y_1}\Big) =\psi(x_2,y_2)+(1-x_2)\psi\Big({x_1\over1-x_2},{y_1\over 1-y_2}\Big),$$ for $x_1,y_1,x_2,y_2\ \in \ [0,1),\ x_1+x_2\leq 1\$   and$\ y_1+y_2\leq 1$, where the fuctions$f$ and $\psi$ are considered under certain regularity conditions.

For more details refer to Mathai and Rathai (1975) [71], Autar (1975) [7], Taneja (1979) [99] etc..
 

Subsections

• Properties of Information Measures