2799:
1739:
2401:
1159:
inequalities. Zhang and Yeung reported the first non-Shannon-type inequality, often referred to as the Zhang-Yeung inequality. Matus proved that no finite set of inequalities can characterize (by linear combinations) all entropic inequalities. In other words, the region
2649:
2837:
is an open source, faster version of the same algorithm implemented in C with a graphical front end. Xitip also has a built in language parsing feature which support a broader range of random variable descriptions as input.
1611:
1459:
2113:
2828:
Several machine based proof checker algorithms are now available. Proof checker algorithms typically verify the inequalities as either true or false. More advanced proof checker algorithms can produce proof or
2199:
2013:
2854:
uses Gurobi solver for optimization and a mix of python and C++ in the backend implementation. It can also provide the canonical break down of the inequalities in terms of basic
Information measures.
1933:
590:
719:
each denote the joint distribution of some arbitrary (possibly empty) subset of our collection of random variables. Inequalities that can be derived as linear combinations of this are known as
1197:
1127:
1079:
955:
493:
421:
1801:
1219:
can be expressed as the
Kullback–Leibler divergence of the joint distribution with respect to the product of the marginals, and thus these inequalities can be seen as a special case of
97:
1875:
800:
894:
1554:
1011:
654:
2641:
838:
349:
306:
261:
1494:
1226:
On the other hand, it seems to be much more difficult to derive useful upper bounds for the
Kullback–Leibler divergence. This is because the Kullback–Leibler divergence
2447:
212:
174:
2573:
2528:
2191:
2154:
1153:
1037:
2453:, could replace the right-hand side of this inequality. This is especially significant since it implies, and is stronger than, Weyl's formulation of Heisenberg's
3030:
2593:
2548:
2503:
2483:
979:
764:
744:
717:
697:
677:
117:
2794:{\displaystyle \operatorname {E} {\big (}{\big |}\operatorname {E} (X|Y')-\operatorname {E} (X\mid Y){\big |}{\big )}\leq {\sqrt {I(X;Y\mid Y')\,2\log 2}},}
1734:{\displaystyle {\sqrt {{\frac {1}{2}}D_{KL}^{(e)}(P\parallel Q)}}\geq \sup\{|P(A)-Q(A)|:A{\text{ is an event to which probabilities are assigned.}}\}.}
3290:
Nivedita
Rethnakar, Suhas Diggavi, Raymond. W. Yeung, InformationInequalities.jl: Exploring Information-Theoretic Inequalities, Julia Package, 2021
3278:
N. R. Pai, Suhas
Diggavi, T. Gläßle, E. Perron, R.Pulikkoonattu, R. W. Yeung, Y. Yan, oXitip: An Online Information Theoretic Inequalities Prover
1081:
is known to be convex and hence it can be characterized by the (infinitely many) linear inequalities satisfied by all entropic vectors, called
263:. They satisfy the following inequalities (which together characterize the range of the marginal and joint entropies of two random variables):
3261:
3006:
2021:
1370:
2396:{\displaystyle -\int _{-\infty }^{\infty }|f(x)|^{2}\log |f(x)|^{2}\,dx-\int _{-\infty }^{\infty }|g(y)|^{2}\log |g(y)|^{2}\,dy\geq 0.}
3197:
Ho, S.W.; Ling, L.; Tan, C.W.; Yeung, R.W. (2020). "Proving and
Disproving Information Inequalities: Theory and Scalable Algorithms".
3251:
3243:
119:
2812:. (Note: the correction factor log 2 inside the radical arises because we are measuring the conditional mutual information in
3310:
1938:
1586:
1320:
1212:
3046:
2805:
597:
1902:
499:
1090:
The set of all vectors that satisfy
Shannon-type inequalities (but not necessarily other entropic inequalities) contains
3315:
2863:
1163:
1093:
1045:
899:
427:
355:
1750:
2873:
1602:
1590:
1350:
19:
1332:
43:
2868:
1568:
2898:
2809:
1580:
1497:
1216:
3284:
Siu Wai Ho, Lin Ling, Chee Wei Tan and
Raymond W. Yeung, AITIP (Information Theoretic Inequality Prover):
2888:
1816:
769:
2987:
Fuchs, Aimé; Letta, Giorgio (1970). "L'Inégalité de KULLBACK. Application à la théorie de l'estimation".
3305:
3163:"The final form of Tao's inequality relating conditional expectation and conditional mutual information"
3083:
2945:
Zhang, Z.; Yeung, R. W. (1998). "On characterization of entropy function via information inequalities".
2454:
1894:
853:
3140:
2883:
2116:
1557:
1314:
1220:
984:
607:
805:
312:
269:
217:
2450:
1467:
1357:
1040:
1520:
3214:
3130:
3100:
3063:
3024:
23:
2409:
2598:
178:
140:
3247:
3239:
3002:
2016:
126:
2159:
2122:
1132:
1016:
746:
there are further restrictions on possible values of entropy. To make this precise, a vector
3206:
3174:
3092:
3055:
2994:
2954:
2926:
2893:
1506:
1211:
A great many important inequalities in information theory are actually lower bounds for the
3238:, Chapter 16, "Inequalities in Information Theory" John Wiley & Sons, Inc. 1991 Print
3016:
3158:
3012:
2878:
2850:
uses GLPK optimizer and has a C++ backend based on Xitip with a web based user interface.
1200:
1083:
842:
595:
In fact, these can all be expressed as special cases of a single inequality involving the
122:
35:
1264:) increases without bound as an event of finite non-zero probability in the distribution
1215:. Even the Shannon-type inequalities can be considered part of this category, since the
3291:
3144:
2553:
2508:
2578:
2533:
2488:
2468:
964:
749:
729:
702:
682:
662:
102:
1243:) depends very sensitively on events that are very rare in the reference distribution
3299:
3218:
1501:
958:
130:
2988:
3118:
26:. There are a number of different contexts in which these inequalities appear.
2846:
are cloud based implementations for validating the
Shannon type inequalities.
2817:
1807:
3210:
2993:. Lecture Notes in Mathematics. Vol. 124. Strasbourg. pp. 108–131.
3179:
3162:
1337:
Another inequality concerning the
Kullback–Leibler divergence is known as
1510:
3267:
3260:
IEEE Transactions on Information Theory, Vol. 37, No. 6, November 1991.
2830:
2406:
Hirschman conjectured, and it was later proved, that a sharper bound of
1899:
In 1957, Hirschman showed that for a (reasonably well-behaved) function
3104:
3067:
2998:
2917:
Yeung, R.W. (1997). "A framework for linear information inequalities".
2958:
2930:
3135:
2108:{\displaystyle g(y)=\int _{-\infty }^{\infty }f(x)e^{-2\pi ixy}\,dx,}
1454:{\displaystyle D_{KL}(P\parallel Q)\geq \Psi _{Q}^{*}(\mu '_{1}(P)),}
3096:
3059:
2833:
is a Matlab based proof checker for all Shannon type Inequalities.
1013:, following the notation of Yeung. It is not closed nor convex for
2824:
Machine based proof checker of information-theoretic inequalities
2813:
1289:) is not even defined if an event of non-zero probability in
2008:{\displaystyle \int _{-\infty }^{\infty }|f(x)|^{2}\,dx=1,}
2976:. 2007 IEEE International Symposium on Information Theory.
3279:
3081:
Beckner, W. (1975). "Inequalities in Fourier Analysis".
2847:
2843:
3273:
2834:
1268:
becomes exceedingly rare in the reference distribution
1928:{\displaystyle f:\mathbb {R} \rightarrow \mathbb {C} }
1722: is an event to which probabilities are assigned.
2990:
Séminaire de Probabilités IV Université de Strasbourg
2652:
2601:
2581:
2556:
2536:
2511:
2491:
2471:
2412:
2202:
2162:
2125:
2024:
1941:
1905:
1819:
1753:
1614:
1523:
1470:
1373:
1166:
1135:
1096:
1048:
1019:
987:
967:
902:
856:
808:
772:
752:
732:
705:
685:
665:
610:
585:{\displaystyle H(X_{1},X_{2})\leq H(X_{1})+H(X_{2}).}
502:
430:
358:
315:
272:
220:
181:
143:
105:
46:
3285:
2851:
2839:
3121:(2006). "Szemerédi's regularity lemma revisited".
2793:
2635:
2587:
2567:
2542:
2522:
2497:
2477:
2441:
2395:
2185:
2148:
2107:
2007:
1927:
1869:
1795:
1733:
1548:
1488:
1453:
1191:
1147:
1121:
1073:
1031:
1005:
973:
949:
888:
832:
794:
758:
738:
711:
691:
671:
648:
584:
487:
415:
343:
300:
255:
206:
168:
111:
91:
1821:
1671:
1207:Lower bounds for the Kullback–Leibler divergence
133:) entropies can be computed. For example, when
3044:Hirschman, I. I. (1957). "A Note on Entropy".
846:if there is a joint, discrete distribution of
137: = 2, we may consider the entropies
2734:
2727:
2668:
2661:
1192:{\displaystyle {\overline {\Gamma _{n}^{*}}}}
1122:{\displaystyle {\overline {\Gamma _{n}^{*}}}}
1074:{\displaystyle {\overline {\Gamma _{n}^{*}}}}
8:
3256:Amir Dembo, Thomas M. Cover, Joy A. Thomas.
2804:relating the conditional expectation to the
1806:is the Kullback–Leibler divergence in
1725:
1674:
1319:This fundamental inequality states that the
827:
809:
3192:
3190:
950:{\displaystyle h_{I}=H(X_{i}\colon i\in I)}
488:{\displaystyle H(X_{2})\leq H(X_{1},X_{2})}
416:{\displaystyle H(X_{1})\leq H(X_{1},X_{2})}
3029:: CS1 maint: location missing publisher (
1796:{\displaystyle D_{KL}^{(e)}(P\parallel Q)}
3178:
3167:Advances in Mathematics of Communications
3134:
2773:
2742:
2733:
2732:
2726:
2725:
2685:
2667:
2666:
2660:
2659:
2651:
2616:
2600:
2580:
2555:
2535:
2510:
2490:
2470:
2425:
2411:
2380:
2374:
2369:
2351:
2339:
2334:
2316:
2310:
2302:
2288:
2282:
2277:
2259:
2247:
2242:
2224:
2218:
2210:
2201:
2177:
2172:
2163:
2161:
2140:
2135:
2126:
2124:
2095:
2074:
2052:
2044:
2023:
1989:
1983:
1978:
1960:
1954:
1946:
1940:
1921:
1920:
1913:
1912:
1904:
1862:
1830:
1824:
1818:
1766:
1758:
1752:
1720:
1709:
1677:
1639:
1631:
1617:
1615:
1613:
1528:
1522:
1480:
1475:
1469:
1427:
1414:
1409:
1378:
1372:
1301:be absolutely continuous with respect to
1178:
1173:
1167:
1165:
1134:
1108:
1103:
1097:
1095:
1060:
1055:
1049:
1047:
1018:
997:
992:
986:
981:. The set of entropic vectors is denoted
966:
926:
907:
901:
880:
861:
855:
807:
784:
779:
775:
774:
771:
751:
731:
704:
684:
664:
626:
609:
570:
548:
526:
513:
501:
476:
463:
441:
429:
404:
391:
369:
357:
326:
314:
283:
271:
244:
231:
219:
192:
180:
154:
142:
120:finitely (or at most countably) supported
104:
83:
64:
51:
45:
2974:Infinitely many information inequalities
2550:takes values only in the interval and
92:{\displaystyle X_{1},X_{2},\dots ,X_{n}}
3199:IEEE Transactions on Information Theory
2947:IEEE Transactions on Information Theory
2919:IEEE Transactions on Information Theory
2909:
3022:
1155:and further inequalities are known as
3268:http://user-www.ie.cuhk.edu.hk/~ITIP/
7:
1870:{\displaystyle \sup _{A}|P(A)-Q(A)|}
1364:and whose first moments exist, then
795:{\displaystyle \mathbb {R} ^{2^{n}}}
3258:Information Theoretic Inequalities.
2808:. This is a simple consequence of
2449:which is attained in the case of a
129:. There are 2 subsets, for which (
22:are very important in the study of
2704:
2673:
2653:
2311:
2306:
2219:
2214:
2053:
2048:
1955:
1950:
1472:
1406:
1170:
1100:
1052:
989:
889:{\displaystyle X_{1},\dots ,X_{n}}
14:
1880:is the total variation distance.
1587:Kullback–Leibler divergence
1129:. This containment is strict for
3234:Thomas M. Cover, Joy A. Thomas.
2465:Given discrete random variables
3047:American Journal of Mathematics
1571:is a corollary of this result.
1297:. (Hence the requirement that
1006:{\displaystyle \Gamma _{n}^{*}}
649:{\displaystyle I(A;B|C)\geq 0,}
3236:Elements of Information Theory
2806:conditional mutual information
2770:
2747:
2722:
2710:
2698:
2686:
2679:
2624:
2617:
2605:
2433:
2419:
2370:
2365:
2359:
2352:
2335:
2330:
2324:
2317:
2278:
2273:
2267:
2260:
2243:
2238:
2232:
2225:
2173:
2164:
2136:
2127:
2067:
2061:
2034:
2028:
1979:
1974:
1968:
1961:
1917:
1863:
1859:
1853:
1844:
1838:
1831:
1790:
1778:
1773:
1767:
1710:
1706:
1700:
1691:
1685:
1678:
1663:
1651:
1646:
1640:
1543:
1537:
1445:
1442:
1436:
1420:
1399:
1387:
944:
919:
833:{\displaystyle \{1,\dots ,n\}}
634:
627:
614:
598:conditional mutual information
576:
563:
554:
541:
532:
506:
482:
456:
447:
434:
410:
384:
375:
362:
344:{\displaystyle H(X_{2})\geq 0}
332:
319:
301:{\displaystyle H(X_{1})\geq 0}
289:
276:
256:{\displaystyle H(X_{1},X_{2})}
250:
224:
198:
185:
160:
147:
1:
1585:Pinsker's inequality relates
1489:{\displaystyle \Psi _{Q}^{*}}
16:Concept in information theory
1549:{\displaystyle \mu '_{1}(P)}
1184:
1114:
1066:
1321:Kullback–Leibler divergence
1213:Kullback–Leibler divergence
3332:
2864:Data processing inequality
2460:
2442:{\displaystyle \log(e/2),}
1892:
1578:
1574:
1330:
1326:
1312:
33:
2636:{\displaystyle H(Y'|Y)=0}
1603:probability distributions
1513:-generating function, of
1351:probability distributions
1308:
207:{\displaystyle H(X_{2}),}
169:{\displaystyle H(X_{1}),}
3211:10.1109/TIT.2020.2982642
2874:Entropy power inequality
1591:total variation distance
1293:has zero probability in
3311:Entropy and information
2186:{\displaystyle |g|^{2}}
2149:{\displaystyle |f|^{2}}
1217:interaction information
1148:{\displaystyle n\geq 4}
1032:{\displaystyle n\geq 3}
3180:10.3934/amc.2007.1.239
3123:Contrib. Discrete Math
2795:
2637:
2589:
2569:
2544:
2524:
2499:
2479:
2443:
2397:
2193:is non-negative, i.e.
2187:
2150:
2117:differential entropies
2109:
2009:
1929:
1871:
1797:
1735:
1550:
1490:
1455:
1353:on the real line with
1193:
1149:
1123:
1075:
1033:
1007:
975:
951:
890:
834:
802:indexed by subsets of
796:
760:
740:
713:
693:
673:
650:
586:
489:
417:
345:
302:
257:
208:
170:
113:
93:
3280:http://www.oxitip.com
3084:Annals of Mathematics
2796:
2638:
2590:
2570:
2545:
2525:
2500:
2480:
2455:uncertainty principle
2451:Gaussian distribution
2444:
2398:
2188:
2151:
2110:
2010:
1930:
1895:Hirschman uncertainty
1889:Hirschman uncertainty
1872:
1798:
1736:
1593:. It states that if
1551:
1491:
1456:
1358:absolutely continuous
1339:Kullback's inequality
1333:Kullback's inequality
1327:Kullback's inequality
1194:
1150:
1124:
1084:entropic inequalities
1076:
1034:
1008:
976:
952:
891:
835:
797:
761:
741:
714:
694:
674:
651:
587:
490:
418:
346:
303:
258:
209:
171:
114:
94:
30:Entropic inequalities
3274:http://xitip.epfl.ch
2899:Pinsker's inequality
2810:Pinsker's inequality
2650:
2599:
2579:
2554:
2534:
2509:
2489:
2469:
2410:
2200:
2160:
2123:
2022:
1939:
1903:
1817:
1751:
1612:
1581:Pinsker's inequality
1575:Pinsker's inequality
1521:
1468:
1371:
1164:
1133:
1094:
1046:
1017:
985:
965:
900:
854:
806:
770:
750:
730:
703:
683:
663:
608:
500:
428:
356:
313:
270:
218:
179:
141:
103:
44:
3145:2005math......4472T
2889:Jensen's inequality
2315:
2223:
2057:
1959:
1777:
1650:
1536:
1485:
1435:
1419:
1183:
1113:
1065:
1041:topological closure
1002:
3316:Information theory
2999:10.1007/bfb0059338
2972:Matus, F. (2007).
2791:
2633:
2585:
2568:{\displaystyle Y'}
2565:
2540:
2523:{\displaystyle Y'}
2520:
2495:
2475:
2439:
2393:
2298:
2206:
2183:
2146:
2105:
2040:
2005:
1942:
1925:
1884:Other inequalities
1867:
1829:
1793:
1754:
1731:
1627:
1546:
1524:
1486:
1471:
1451:
1423:
1405:
1189:
1169:
1145:
1119:
1099:
1071:
1051:
1029:
1003:
988:
971:
961:, for each subset
947:
886:
830:
792:
756:
736:
709:
689:
669:
646:
582:
485:
413:
341:
298:
253:
204:
166:
109:
89:
24:information theory
3286:https://aitip.org
3008:978-3-540-04913-5
2959:10.1109/18.681320
2931:10.1109/18.641556
2884:Fano's inequality
2786:
2588:{\displaystyle Y}
2575:is determined by
2543:{\displaystyle X}
2498:{\displaystyle Y}
2478:{\displaystyle X}
2017:Fourier transform
1820:
1723:
1666:
1625:
1323:is non-negative.
1315:Gibbs' inequality
1309:Gibbs' inequality
1221:Gibbs' inequality
1187:
1117:
1069:
974:{\displaystyle I}
850:random variables
759:{\displaystyle h}
739:{\displaystyle n}
712:{\displaystyle C}
692:{\displaystyle B}
672:{\displaystyle A}
127:probability space
112:{\displaystyle n}
40:Consider a tuple
3323:
3223:
3222:
3205:(9): 5525–5536.
3194:
3185:
3184:
3182:
3159:Ahlswede, Rudolf
3155:
3149:
3148:
3138:
3115:
3109:
3108:
3078:
3072:
3071:
3041:
3035:
3034:
3028:
3020:
2984:
2978:
2977:
2969:
2963:
2962:
2953:(4): 1440–1452.
2942:
2936:
2934:
2925:(6): 1924–1934.
2914:
2894:Kraft inequality
2869:Cramér–Rao bound
2829:counterexamples.
2800:
2798:
2797:
2792:
2787:
2769:
2743:
2738:
2737:
2731:
2730:
2697:
2689:
2672:
2671:
2665:
2664:
2642:
2640:
2639:
2634:
2620:
2615:
2594:
2592:
2591:
2586:
2574:
2572:
2571:
2566:
2564:
2549:
2547:
2546:
2541:
2529:
2527:
2526:
2521:
2519:
2504:
2502:
2501:
2496:
2484:
2482:
2481:
2476:
2461:Tao's inequality
2448:
2446:
2445:
2440:
2429:
2402:
2400:
2399:
2394:
2379:
2378:
2373:
2355:
2344:
2343:
2338:
2320:
2314:
2309:
2287:
2286:
2281:
2263:
2252:
2251:
2246:
2228:
2222:
2217:
2192:
2190:
2189:
2184:
2182:
2181:
2176:
2167:
2155:
2153:
2152:
2147:
2145:
2144:
2139:
2130:
2114:
2112:
2111:
2106:
2094:
2093:
2056:
2051:
2014:
2012:
2011:
2006:
1988:
1987:
1982:
1964:
1958:
1953:
1934:
1932:
1931:
1926:
1924:
1916:
1876:
1874:
1873:
1868:
1866:
1834:
1828:
1802:
1800:
1799:
1794:
1776:
1765:
1740:
1738:
1737:
1732:
1724:
1721:
1713:
1681:
1667:
1649:
1638:
1626:
1618:
1616:
1569:Cramér–Rao bound
1555:
1553:
1552:
1547:
1532:
1507:convex conjugate
1498:large deviations
1495:
1493:
1492:
1487:
1484:
1479:
1460:
1458:
1457:
1452:
1431:
1418:
1413:
1386:
1385:
1360:with respect to
1198:
1196:
1195:
1190:
1188:
1182:
1177:
1168:
1157:non-Shannon type
1154:
1152:
1151:
1146:
1128:
1126:
1125:
1120:
1118:
1112:
1107:
1098:
1080:
1078:
1077:
1072:
1070:
1064:
1059:
1050:
1038:
1036:
1035:
1030:
1012:
1010:
1009:
1004:
1001:
996:
980:
978:
977:
972:
956:
954:
953:
948:
931:
930:
912:
911:
895:
893:
892:
887:
885:
884:
866:
865:
839:
837:
836:
831:
801:
799:
798:
793:
791:
790:
789:
788:
778:
765:
763:
762:
757:
745:
743:
742:
737:
718:
716:
715:
710:
698:
696:
695:
690:
678:
676:
675:
670:
655:
653:
652:
647:
630:
591:
589:
588:
583:
575:
574:
553:
552:
531:
530:
518:
517:
494:
492:
491:
486:
481:
480:
468:
467:
446:
445:
422:
420:
419:
414:
409:
408:
396:
395:
374:
373:
350:
348:
347:
342:
331:
330:
307:
305:
304:
299:
288:
287:
262:
260:
259:
254:
249:
248:
236:
235:
213:
211:
210:
205:
197:
196:
175:
173:
172:
167:
159:
158:
123:random variables
118:
116:
115:
110:
98:
96:
95:
90:
88:
87:
69:
68:
56:
55:
3331:
3330:
3326:
3325:
3324:
3322:
3321:
3320:
3296:
3295:
3231:
3226:
3196:
3195:
3188:
3157:
3156:
3152:
3117:
3116:
3112:
3097:10.2307/1970980
3080:
3079:
3075:
3060:10.2307/2372390
3043:
3042:
3038:
3021:
3009:
2986:
2985:
2981:
2971:
2970:
2966:
2944:
2943:
2939:
2916:
2915:
2911:
2907:
2879:Entropic vector
2860:
2826:
2762:
2690:
2648:
2647:
2608:
2597:
2596:
2577:
2576:
2557:
2552:
2551:
2532:
2531:
2512:
2507:
2506:
2487:
2486:
2467:
2466:
2463:
2408:
2407:
2368:
2333:
2276:
2241:
2198:
2197:
2171:
2158:
2157:
2134:
2121:
2120:
2115:the sum of the
2070:
2020:
2019:
1977:
1937:
1936:
1901:
1900:
1897:
1891:
1886:
1815:
1814:
1749:
1748:
1610:
1609:
1583:
1577:
1519:
1518:
1466:
1465:
1374:
1369:
1368:
1335:
1329:
1317:
1311:
1280:
1255:
1234:
1209:
1162:
1161:
1131:
1130:
1092:
1091:
1044:
1043:
1015:
1014:
983:
982:
963:
962:
922:
903:
898:
897:
876:
857:
852:
851:
804:
803:
780:
773:
768:
767:
748:
747:
728:
727:
701:
700:
681:
680:
661:
660:
606:
605:
566:
544:
522:
509:
498:
497:
472:
459:
437:
426:
425:
400:
387:
365:
354:
353:
322:
311:
310:
279:
268:
267:
240:
227:
216:
215:
188:
177:
176:
150:
139:
138:
101:
100:
79:
60:
47:
42:
41:
38:
36:Entropic vector
32:
17:
12:
11:
5:
3329:
3327:
3319:
3318:
3313:
3308:
3298:
3297:
3294:
3293:
3288:
3282:
3276:
3270:
3264:
3254:
3230:
3229:External links
3227:
3225:
3224:
3186:
3173:(2): 239–242.
3150:
3110:
3091:(6): 159–182.
3073:
3054:(1): 152–156.
3036:
3007:
2979:
2964:
2937:
2908:
2906:
2903:
2902:
2901:
2896:
2891:
2886:
2881:
2876:
2871:
2866:
2859:
2856:
2825:
2822:
2802:
2801:
2790:
2785:
2782:
2779:
2776:
2772:
2768:
2765:
2761:
2758:
2755:
2752:
2749:
2746:
2741:
2736:
2729:
2724:
2721:
2718:
2715:
2712:
2709:
2706:
2703:
2700:
2696:
2693:
2688:
2684:
2681:
2678:
2675:
2670:
2663:
2658:
2655:
2632:
2629:
2626:
2623:
2619:
2614:
2611:
2607:
2604:
2584:
2563:
2560:
2539:
2518:
2515:
2494:
2474:
2462:
2459:
2438:
2435:
2432:
2428:
2424:
2421:
2418:
2415:
2404:
2403:
2392:
2389:
2386:
2383:
2377:
2372:
2367:
2364:
2361:
2358:
2354:
2350:
2347:
2342:
2337:
2332:
2329:
2326:
2323:
2319:
2313:
2308:
2305:
2301:
2297:
2294:
2291:
2285:
2280:
2275:
2272:
2269:
2266:
2262:
2258:
2255:
2250:
2245:
2240:
2237:
2234:
2231:
2227:
2221:
2216:
2213:
2209:
2205:
2180:
2175:
2170:
2166:
2143:
2138:
2133:
2129:
2104:
2101:
2098:
2092:
2089:
2086:
2083:
2080:
2077:
2073:
2069:
2066:
2063:
2060:
2055:
2050:
2047:
2043:
2039:
2036:
2033:
2030:
2027:
2004:
2001:
1998:
1995:
1992:
1986:
1981:
1976:
1973:
1970:
1967:
1963:
1957:
1952:
1949:
1945:
1923:
1919:
1915:
1911:
1908:
1893:Main article:
1890:
1887:
1885:
1882:
1878:
1877:
1865:
1861:
1858:
1855:
1852:
1849:
1846:
1843:
1840:
1837:
1833:
1827:
1823:
1804:
1803:
1792:
1789:
1786:
1783:
1780:
1775:
1772:
1769:
1764:
1761:
1757:
1742:
1741:
1730:
1727:
1719:
1716:
1712:
1708:
1705:
1702:
1699:
1696:
1693:
1690:
1687:
1684:
1680:
1676:
1673:
1670:
1665:
1662:
1659:
1656:
1653:
1648:
1645:
1642:
1637:
1634:
1630:
1624:
1621:
1579:Main article:
1576:
1573:
1545:
1542:
1539:
1535:
1531:
1527:
1483:
1478:
1474:
1462:
1461:
1450:
1447:
1444:
1441:
1438:
1434:
1430:
1426:
1422:
1417:
1412:
1408:
1404:
1401:
1398:
1395:
1392:
1389:
1384:
1381:
1377:
1331:Main article:
1328:
1325:
1313:Main article:
1310:
1307:
1276:
1272:, and in fact
1251:
1230:
1208:
1205:
1186:
1181:
1176:
1172:
1144:
1141:
1138:
1116:
1111:
1106:
1102:
1068:
1063:
1058:
1054:
1028:
1025:
1022:
1000:
995:
991:
970:
946:
943:
940:
937:
934:
929:
925:
921:
918:
915:
910:
906:
883:
879:
875:
872:
869:
864:
860:
840:is said to be
829:
826:
823:
820:
817:
814:
811:
787:
783:
777:
755:
735:
723:inequalities.
708:
688:
668:
657:
656:
645:
642:
639:
636:
633:
629:
625:
622:
619:
616:
613:
593:
592:
581:
578:
573:
569:
565:
562:
559:
556:
551:
547:
543:
540:
537:
534:
529:
525:
521:
516:
512:
508:
505:
495:
484:
479:
475:
471:
466:
462:
458:
455:
452:
449:
444:
440:
436:
433:
423:
412:
407:
403:
399:
394:
390:
386:
383:
380:
377:
372:
368:
364:
361:
351:
340:
337:
334:
329:
325:
321:
318:
308:
297:
294:
291:
286:
282:
278:
275:
252:
247:
243:
239:
234:
230:
226:
223:
203:
200:
195:
191:
187:
184:
165:
162:
157:
153:
149:
146:
108:
86:
82:
78:
75:
72:
67:
63:
59:
54:
50:
34:Main article:
31:
28:
15:
13:
10:
9:
6:
4:
3:
2:
3328:
3317:
3314:
3312:
3309:
3307:
3304:
3303:
3301:
3292:
3289:
3287:
3283:
3281:
3277:
3275:
3271:
3269:
3265:
3263:
3259:
3255:
3253:
3252:0-471-20061-1
3249:
3245:
3244:0-471-06259-6
3241:
3237:
3233:
3232:
3228:
3220:
3216:
3212:
3208:
3204:
3200:
3193:
3191:
3187:
3181:
3176:
3172:
3168:
3164:
3160:
3154:
3151:
3146:
3142:
3137:
3132:
3128:
3124:
3120:
3114:
3111:
3106:
3102:
3098:
3094:
3090:
3086:
3085:
3077:
3074:
3069:
3065:
3061:
3057:
3053:
3049:
3048:
3040:
3037:
3032:
3026:
3018:
3014:
3010:
3004:
3000:
2996:
2992:
2991:
2983:
2980:
2975:
2968:
2965:
2960:
2956:
2952:
2948:
2941:
2938:
2932:
2928:
2924:
2920:
2913:
2910:
2904:
2900:
2897:
2895:
2892:
2890:
2887:
2885:
2882:
2880:
2877:
2875:
2872:
2870:
2867:
2865:
2862:
2861:
2857:
2855:
2853:
2849:
2845:
2841:
2836:
2832:
2823:
2821:
2819:
2815:
2811:
2807:
2788:
2783:
2780:
2777:
2774:
2766:
2763:
2759:
2756:
2753:
2750:
2744:
2739:
2719:
2716:
2713:
2707:
2701:
2694:
2691:
2682:
2676:
2656:
2646:
2645:
2644:
2630:
2627:
2621:
2612:
2609:
2602:
2582:
2561:
2558:
2537:
2516:
2513:
2492:
2472:
2458:
2456:
2452:
2436:
2430:
2426:
2422:
2416:
2413:
2390:
2387:
2384:
2381:
2375:
2362:
2356:
2348:
2345:
2340:
2327:
2321:
2303:
2299:
2295:
2292:
2289:
2283:
2270:
2264:
2256:
2253:
2248:
2235:
2229:
2211:
2207:
2203:
2196:
2195:
2194:
2178:
2168:
2141:
2131:
2118:
2102:
2099:
2096:
2090:
2087:
2084:
2081:
2078:
2075:
2071:
2064:
2058:
2045:
2041:
2037:
2031:
2025:
2018:
2002:
1999:
1996:
1993:
1990:
1984:
1971:
1965:
1947:
1943:
1909:
1906:
1896:
1888:
1883:
1881:
1856:
1850:
1847:
1841:
1835:
1825:
1813:
1812:
1811:
1809:
1787:
1784:
1781:
1770:
1762:
1759:
1755:
1747:
1746:
1745:
1728:
1717:
1714:
1703:
1697:
1694:
1688:
1682:
1668:
1660:
1657:
1654:
1643:
1635:
1632:
1628:
1622:
1619:
1608:
1607:
1606:
1604:
1600:
1596:
1592:
1588:
1582:
1572:
1570:
1565:
1563:
1559:
1556:is the first
1540:
1533:
1529:
1525:
1516:
1512:
1508:
1504:
1503:
1502:rate function
1499:
1481:
1476:
1448:
1439:
1432:
1428:
1424:
1415:
1410:
1402:
1396:
1393:
1390:
1382:
1379:
1375:
1367:
1366:
1365:
1363:
1359:
1356:
1352:
1348:
1344:
1340:
1334:
1324:
1322:
1316:
1306:
1304:
1300:
1296:
1292:
1288:
1284:
1279:
1275:
1271:
1267:
1263:
1259:
1254:
1250:
1246:
1242:
1238:
1233:
1229:
1224:
1222:
1218:
1214:
1206:
1204:
1202:
1179:
1174:
1158:
1142:
1139:
1136:
1109:
1104:
1088:
1086:
1085:
1061:
1056:
1042:
1026:
1023:
1020:
998:
993:
968:
960:
959:joint entropy
941:
938:
935:
932:
927:
923:
916:
913:
908:
904:
881:
877:
873:
870:
867:
862:
858:
849:
845:
844:
824:
821:
818:
815:
812:
785:
781:
753:
733:
724:
722:
706:
686:
666:
643:
640:
637:
631:
623:
620:
617:
611:
604:
603:
602:
600:
599:
579:
571:
567:
560:
557:
549:
545:
538:
535:
527:
523:
519:
514:
510:
503:
496:
477:
473:
469:
464:
460:
453:
450:
442:
438:
431:
424:
405:
401:
397:
392:
388:
381:
378:
370:
366:
359:
352:
338:
335:
327:
323:
316:
309:
295:
292:
284:
280:
273:
266:
265:
264:
245:
241:
237:
232:
228:
221:
201:
193:
189:
182:
163:
155:
151:
144:
136:
132:
128:
124:
121:
106:
84:
80:
76:
73:
70:
65:
61:
57:
52:
48:
37:
29:
27:
25:
21:
3306:Inequalities
3257:
3235:
3202:
3198:
3170:
3166:
3153:
3136:math/0504472
3126:
3122:
3113:
3088:
3082:
3076:
3051:
3045:
3039:
2989:
2982:
2973:
2967:
2950:
2946:
2940:
2922:
2918:
2912:
2827:
2816:rather than
2803:
2530:, such that
2464:
2405:
1898:
1879:
1805:
1743:
1598:
1594:
1584:
1566:
1561:
1514:
1500:
1463:
1361:
1354:
1346:
1342:
1338:
1336:
1318:
1302:
1298:
1294:
1290:
1286:
1282:
1277:
1273:
1269:
1265:
1261:
1257:
1252:
1248:
1244:
1240:
1236:
1231:
1227:
1225:
1210:
1156:
1089:
1082:
847:
841:
725:
721:Shannon-type
720:
658:
596:
594:
134:
125:on the same
39:
20:Inequalities
18:
2643:), we have
2595:(such that
1505:, i.e. the
726:For larger
3300:Categories
2905:References
1935:such that
1039:, but its
896:such that
3219:216530139
3025:cite book
2781:
2760:∣
2740:≤
2717:∣
2708:
2702:−
2677:
2657:
2417:
2388:≥
2349:
2312:∞
2307:∞
2304:−
2300:∫
2296:−
2257:
2220:∞
2215:∞
2212:−
2208:∫
2204:−
2082:π
2076:−
2054:∞
2049:∞
2046:−
2042:∫
1956:∞
1951:∞
1948:−
1944:∫
1918:→
1848:−
1785:∥
1695:−
1669:≥
1658:∥
1526:μ
1482:∗
1473:Ψ
1425:μ
1416:∗
1407:Ψ
1403:≥
1394:∥
1199:is not a
1185:¯
1180:∗
1171:Γ
1140:≥
1115:¯
1110:∗
1101:Γ
1067:¯
1062:∗
1053:Γ
1024:≥
999:∗
990:Γ
957:is their
939:∈
933::
871:…
819:…
638:≥
601:, namely
536:≤
451:≤
379:≤
336:≥
293:≥
74:…
3161:(2007).
3129:: 8–28.
2858:See also
2767:′
2695:′
2613:′
2562:′
2517:′
2015:and its
1601:are two
1534:′
1511:cumulant
1433:′
1201:polytope
843:entropic
3272:XITIP:
3246:Online
3141:Bibcode
3119:Tao, T.
3105:1970980
3068:2372390
3017:0267669
1605:, then
1509:of the
1496:is the
3266:ITIP:
3250:
3242:
3217:
3103:
3066:
3015:
3005:
2848:oXitip
2844:oXitip
2505:, and
1744:where
1558:moment
1517:, and
1464:where
1341:. If
699:, and
659:where
3215:S2CID
3131:arXiv
3101:JSTOR
3064:JSTOR
2852:AITIP
2840:AITIP
2835:Xitip
131:joint
3248:ISBN
3240:ISBN
3031:link
3003:ISBN
2842:and
2831:ITIP
2818:nats
2814:bits
2156:and
1810:and
1808:nats
1589:and
1567:The
1349:are
1345:and
214:and
3262:pdf
3207:doi
3175:doi
3093:doi
3089:102
3056:doi
2995:doi
2955:doi
2927:doi
2820:.)
2778:log
2414:log
2346:log
2254:log
2119:of
1822:sup
1672:sup
1560:of
1305:.)
1247:.
766:in
99:of
3302::
3213:.
3203:66
3201:.
3189:^
3169:.
3165:.
3139:.
3125:.
3099:.
3087:.
3062:.
3052:79
3050:.
3027:}}
3023:{{
3013:MR
3011:.
3001:.
2951:44
2949:.
2923:43
2921:.
2485:,
2457:.
2391:0.
1597:,
1564:.
1362:Q,
1285:||
1278:KL
1260:||
1253:KL
1239:||
1232:KL
1223:.
1203:.
1087:.
679:,
3221:.
3209::
3183:.
3177::
3171:1
3147:.
3143::
3133::
3127:1
3107:.
3095::
3070:.
3058::
3033:)
3019:.
2997::
2961:.
2957::
2935:)
2933:.
2929::
2789:,
2784:2
2775:2
2771:)
2764:Y
2757:Y
2754:;
2751:X
2748:(
2745:I
2735:)
2728:|
2723:)
2720:Y
2714:X
2711:(
2705:E
2699:)
2692:Y
2687:|
2683:X
2680:(
2674:E
2669:|
2662:(
2654:E
2631:0
2628:=
2625:)
2622:Y
2618:|
2610:Y
2606:(
2603:H
2583:Y
2559:Y
2538:X
2514:Y
2493:Y
2473:X
2437:,
2434:)
2431:2
2427:/
2423:e
2420:(
2385:y
2382:d
2376:2
2371:|
2366:)
2363:y
2360:(
2357:g
2353:|
2341:2
2336:|
2331:)
2328:y
2325:(
2322:g
2318:|
2293:x
2290:d
2284:2
2279:|
2274:)
2271:x
2268:(
2265:f
2261:|
2249:2
2244:|
2239:)
2236:x
2233:(
2230:f
2226:|
2179:2
2174:|
2169:g
2165:|
2142:2
2137:|
2132:f
2128:|
2103:,
2100:x
2097:d
2091:y
2088:x
2085:i
2079:2
2072:e
2068:)
2065:x
2062:(
2059:f
2038:=
2035:)
2032:y
2029:(
2026:g
2003:,
2000:1
1997:=
1994:x
1991:d
1985:2
1980:|
1975:)
1972:x
1969:(
1966:f
1962:|
1922:C
1914:R
1910::
1907:f
1864:|
1860:)
1857:A
1854:(
1851:Q
1845:)
1842:A
1839:(
1836:P
1832:|
1826:A
1791:)
1788:Q
1782:P
1779:(
1774:)
1771:e
1768:(
1763:L
1760:K
1756:D
1729:.
1726:}
1718:A
1715::
1711:|
1707:)
1704:A
1701:(
1698:Q
1692:)
1689:A
1686:(
1683:P
1679:|
1675:{
1664:)
1661:Q
1655:P
1652:(
1647:)
1644:e
1641:(
1636:L
1633:K
1629:D
1623:2
1620:1
1599:Q
1595:P
1562:P
1544:)
1541:P
1538:(
1530:1
1515:Q
1477:Q
1449:,
1446:)
1443:)
1440:P
1437:(
1429:1
1421:(
1411:Q
1400:)
1397:Q
1391:P
1388:(
1383:L
1380:K
1376:D
1355:P
1347:Q
1343:P
1303:Q
1299:P
1295:Q
1291:P
1287:Q
1283:P
1281:(
1274:D
1270:Q
1266:P
1262:Q
1258:P
1256:(
1249:D
1245:Q
1241:Q
1237:P
1235:(
1228:D
1175:n
1143:4
1137:n
1105:n
1057:n
1027:3
1021:n
994:n
969:I
945:)
942:I
936:i
928:i
924:X
920:(
917:H
914:=
909:I
905:h
882:n
878:X
874:,
868:,
863:1
859:X
848:n
828:}
825:n
822:,
816:,
813:1
810:{
786:n
782:2
776:R
754:h
734:n
707:C
687:B
667:A
644:,
641:0
635:)
632:C
628:|
624:B
621:;
618:A
615:(
612:I
580:.
577:)
572:2
568:X
564:(
561:H
558:+
555:)
550:1
546:X
542:(
539:H
533:)
528:2
524:X
520:,
515:1
511:X
507:(
504:H
483:)
478:2
474:X
470:,
465:1
461:X
457:(
454:H
448:)
443:2
439:X
435:(
432:H
411:)
406:2
402:X
398:,
393:1
389:X
385:(
382:H
376:)
371:1
367:X
363:(
360:H
339:0
333:)
328:2
324:X
320:(
317:H
296:0
290:)
285:1
281:X
277:(
274:H
251:)
246:2
242:X
238:,
233:1
229:X
225:(
222:H
202:,
199:)
194:2
190:X
186:(
183:H
164:,
161:)
156:1
152:X
148:(
145:H
135:n
107:n
85:n
81:X
77:,
71:,
66:2
62:X
58:,
53:1
49:X
Text is available under the Creative Commons Attribution-ShareAlike License. Additional terms may apply.