Go to:Table of Contents

Divergence Measures

We see that the measure (2.1) is not symmetric in$P$ and $Q$ . Its symmetric version known J-divergence (Jeffreys, 1946 [49]; Kullback and Leibler, 1961 [65]) is given by

$$J(P\vert\vert Q)=D(P\vert\vert Q)+D(Q\vert\vert P)= \sum_{i=1}^n (p_i - q_i)\,\, log(\frac{p_i}{q_i}).$$ (2.7)

Sibson (1969) [95] for the first time introduced the idea of information radius for a measure arising due to concavity property of Shannon's entropy. This measure recently referred as Jensen difference divergence measure is given by 

 

$$=\sum_{i=1}^n{\Big[{p_i\log p_i+q_i\log q_i\over2}-\Big({p_i+q_i\over 2}\Big) \log \Big({p_i+q_i\over2}\Big)\Big]}.$$ (2.8)

By simple calculations, one can also write

$$I(P\vert\vert Q)= \frac{1}{2}\left[D\Big(P\vert\vert{P+Q\over2}\Big)+D\Big(Q\vert\vert{P+Q\over 2}\Big)\right].$$ (2.9)

Taneja (1995) [108] studied an another kind of measure based on arithmetic and geometric mean inequality calling arithemetic and geometric mean divergence measure given by

$$T(P\vert\vert Q)= \frac{1}{2}\left[D\Big({P+Q\over2}\vert\vert P\Big)+D\Big({P+Q\over 2}\vert\vert Q\Big)\right]$$

Interestingly these three measures satisfy the following inequality: 

The measures given by (2.7), (2.8) and (2.10) satisfy the following properties.

Property 2.17. We have

(i) $I(P\vert\vert Q) \geq 0$ ; $J(P\vert\vert Q) \geq 0$ and $T(P\vert\vert Q) \geq 0$

(ii) $M(P\vert\vert Q)\geq exp \big(-{1\over 2}J(P\vert\vert Q)\big)$

(iii) $M(P\vert\vert Q)\geq 1-{1\over 4}J(P\vert\vert Q)$

(iv) $J(P\vert\vert Q)\geq 4\big[1-B(P\vert\vert Q)\big]$

(v) $J(P\vert\vert Q) \leq 4 \,I(P\vert\vert Q).$

where the measures $B(P\vert\vert Q)$ and $M(P\vert\vert Q)$ are the measures given by (2.5) and (2.6) respectively.

The measure $J(P\vert\vert Q)$ used in the items (ii), (iii) and (v) is considered with natural logarithm base.