Dual total correlation

For a set of n random variables ${\displaystyle \{X_{1},\ldots ,X_{n}\}}$ , the dual total correlation ${\displaystyle D(X_{1},\ldots ,X_{n})}$  is given by

${\displaystyle D(X_{1},\ldots ,X_{n})=H\left(X_{1},\ldots ,X_{n}\right)-\sum _{i=1}^{n}H\left(X_{i}\mid X_{1},\ldots ,X_{i-1},X_{i+1},\ldots ,X_{n}\right),}$ 

where ${\displaystyle H(X_{1},\ldots ,X_{n})}$  is the joint entropy of the variable set ${\displaystyle \{X_{1},\ldots ,X_{n}\}}$  and ${\displaystyle H(X_{i}\mid \cdots )}$  is the conditional entropy of variable ${\displaystyle X_{i}}$ , given the rest.

The dual total correlation normalized between [0,1] is simply the dual total correlation divided by its maximum value ${\displaystyle H(X_{1},\ldots ,X_{n})}$ ,

${\displaystyle ND(X_{1},\ldots ,X_{n})={\frac {D(X_{1},\ldots ,X_{n})}{H(X_{1},\ldots ,X_{n})}}.}$ 

Dual total correlation is non-negative and bounded above by the joint entropy ${\displaystyle H(X_{1},\ldots ,X_{n})}$ .

${\displaystyle 0\leq D(X_{1},\ldots ,X_{n})\leq H(X_{1},\ldots ,X_{n}).}$ 

Secondly, Dual total correlation has a close relationship with total correlation, ${\displaystyle C(X_{1},\ldots ,X_{n})}$ , and can be written in terms of differences between the total correlation of the whole, and all subsets of size ${\displaystyle N-1}$ :^[6]

${\displaystyle D({\textbf {X}})=(N-1)C({\textbf {X}})-\sum _{i=1}^{N}C({\textbf {X}}^{-i})}$ 

where ${\displaystyle {\textbf {X}}=\{X_{1},\ldots ,X_{n}\}}$  and ${\displaystyle {\textbf {X}}^{-i}=\{X_{1},\ldots ,X_{i-1},X_{i+1},\ldots ,X_{n}\}}$ 

Furthermore, the total correlation and dual total correlation are related by the following bounds:

${\displaystyle {\frac {C(X_{1},\ldots ,X_{n})}{n-1}}\leq D(X_{1},\ldots ,X_{n})\leq (n-1)\;C(X_{1},\ldots ,X_{n}).}$ 

Finally, the difference between the total correlation and the dual total correlation defines a novel measure of higher-order information-sharing: the O-information:^[7]

${\displaystyle \Omega ({\textbf {X}})=C({\textbf {X}})-D({\textbf {X}})}$ .

The O-information (first introduced as the "enigmatic information" by James and Crutchfield^[8] is a signed measure that quantifies the extent to which the information in a multivariate random variable is dominated by synergistic interactions (in which case ${\displaystyle \Omega ({\textbf {X}})<0}$ ) or redundant interactions (in which case ${\displaystyle \Omega ({\textbf {X}})>0}$ .

Han (1978) originally defined the dual total correlation as,

${\displaystyle {\begin{aligned}&D(X_{1},\ldots ,X_{n})\\[10pt]\equiv {}&\left[\sum _{i=1}^{n}H(X_{1},\ldots ,X_{i-1},X_{i+1},\ldots ,X_{n})\right]-(n-1)\;H(X_{1},\ldots ,X_{n})\;.\end{aligned}}}$ 

However Abdallah and Plumbley (2010) showed its equivalence to the easier-to-understand form of the joint entropy minus the sum of conditional entropies via the following:

${\displaystyle {\begin{aligned}&D(X_{1},\ldots ,X_{n})\\[10pt]\equiv {}&\left[\sum _{i=1}^{n}H(X_{1},\ldots ,X_{i-1},X_{i+1},\ldots ,X_{n})\right]-(n-1)\;H(X_{1},\ldots ,X_{n})\\={}&\left[\sum _{i=1}^{n}H(X_{1},\ldots ,X_{i-1},X_{i+1},\ldots ,X_{n})\right]+(1-n)\;H(X_{1},\ldots ,X_{n})\\={}&H(X_{1},\ldots ,X_{n})+\left[\sum _{i=1}^{n}H(X_{1},\ldots ,X_{i-1},X_{i+1},\ldots ,X_{n})-H(X_{1},\ldots ,X_{n})\right]\\={}&H\left(X_{1},\ldots ,X_{n}\right)-\sum _{i=1}^{n}H\left(X_{i}\mid X_{1},\ldots ,X_{i-1},X_{i+1},\ldots ,X_{n}\right)\;.\end{aligned}}}$ 

• Interaction information
• Mutual information
• Total correlation

• Fujishige, Satoru (1978). "Polymatroidal dependence structure of a set of random variables". Information and Control. 39: 55–72. doi:10.1016/S0019-9958(78)91063-X.
• Varley, Thomas; Pope, Maria; Faskowitz, Joshua; Sporns, Olaf (2023). "Multivariate information theory uncovers synergistic subsystems of the human cerebral cortex". Communications Biology. 6: 451. doi:10.1038/s42003-023-04843-w. PMC 10125999. PMID 37095282.