In
quantum information theory, quantum relative entropy is a measure of distinguishability between two
quantum states. It is the quantum mechanical analog of
relative entropy.
Motivation
For simplicity, it will be assumed that all objects in the article are finite-dimensional.
We first discuss the classical case. Suppose the probabilities of a finite sequence of events is given by the probability distribution P = {p1...pn}, but somehow we mistakenly assumed it to be Q = {q1...qn}. For instance, we can mistake an unfair coin for a fair one. According to this erroneous assumption, our uncertainty about the j-th event, or equivalently, the amount of information provided after observing the j-th event, is
![{\displaystyle \;-\log q_{j}.}](https://wikimedia.org/api/rest_v1/media/math/render/svg/2ec3972d40a789ed6e5285630a4bf14a83c62030)
The (assumed) average uncertainty of all possible events is then
![{\displaystyle \;-\sum _{j}p_{j}\log q_{j}.}](https://wikimedia.org/api/rest_v1/media/math/render/svg/2b0c34f29d4152dc5b83dc0a8c410eca93a65c8d)
On the other hand, the
Shannon entropy of the probability distribution p, defined by
![{\displaystyle \;-\sum _{j}p_{j}\log p_{j},}](https://wikimedia.org/api/rest_v1/media/math/render/svg/68c532ef89a1266ad1601062af09878d87cf687f)
is the real amount of uncertainty before observation. Therefore the difference between these two quantities
![{\displaystyle \;-\sum _{j}p_{j}\log q_{j}-\left(-\sum _{j}p_{j}\log p_{j}\right)=\sum _{j}p_{j}\log p_{j}-\sum _{j}p_{j}\log q_{j}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/039b4f329dd44b6a173b0a0c5a188029565abbc1)
is a measure of the distinguishability of the two probability distributions p and q. This is precisely the classical relative entropy, or
Kullback–Leibler divergence:
![{\displaystyle D_{\mathrm {KL} }(P\|Q)=\sum _{j}p_{j}\log {\frac {p_{j}}{q_{j}}}\!.}](https://wikimedia.org/api/rest_v1/media/math/render/svg/59657a338156bee8cc3916dd2bd83f0676b2c74f)
Note
- In the definitions above, the convention that 0·log 0 = 0 is assumed, since
. Intuitively, one would expect that an event of
zero probability to contribute nothing towards entropy.
- The relative entropy is not a
metric. For example, it is not symmetric. The uncertainty discrepancy in mistaking a fair coin to be unfair is not the same as the opposite situation.
Definition
As with many other objects in quantum information theory, quantum relative entropy is defined by extending the classical definition from probability distributions to
density matrices. Let ρ be a density matrix. The
von Neumann entropy of ρ, which is the quantum mechanical analog of the Shannon entropy, is given by
![{\displaystyle S(\rho )=-\operatorname {Tr} \rho \log \rho .}](https://wikimedia.org/api/rest_v1/media/math/render/svg/615078d490f3cf3c7c1a5c04b92b45068c531824)
For two density matrices ρ and σ, the quantum relative entropy of ρ with respect to σ is defined by
![{\displaystyle S(\rho \|\sigma )=-\operatorname {Tr} \rho \log \sigma -S(\rho )=\operatorname {Tr} \rho \log \rho -\operatorname {Tr} \rho \log \sigma =\operatorname {Tr} \rho (\log \rho -\log \sigma ).}](https://wikimedia.org/api/rest_v1/media/math/render/svg/609076da6a64c7580b80e0ef494273f32f9c1c1f)
We see that, when the states are classically related, i.e. ρσ = σρ, the definition coincides with the classical case, in the sense that if
and
with
and
(because
and
commute, they are
simultaneously diagonalizable), then
is just the ordinary
Kullback-Leibler divergence of the probability vector
with respect to the probability vector
.
Non-finite (divergent) relative entropy
In general, the support of a matrix M is the orthogonal complement of its
kernel, i.e.
. When considering the quantum relative entropy, we assume the convention that −s · log 0 = ∞ for any s > 0. This leads to the definition that
![{\displaystyle S(\rho \|\sigma )=\infty }](https://wikimedia.org/api/rest_v1/media/math/render/svg/8594df6e716f8e6de422b4e9434c838116df0b8b)
when
![{\displaystyle {\text{supp}}(\rho )\cap {\text{ker}}(\sigma )\neq \{0\}.}](https://wikimedia.org/api/rest_v1/media/math/render/svg/24d0d52d2bd3b4f23eeb9e96f76699475175c96e)
This can be interpreted in the following way. Informally, the quantum relative entropy is a measure of our ability to distinguish two quantum states where larger values indicate states that are more different. Being orthogonal represents the most different quantum states can be. This is reflected by non-finite quantum relative entropy for orthogonal quantum states. Following the argument given in the Motivation section, if we erroneously assume the state
has support in
, this is an error impossible to recover from.
However, one should be careful not to conclude that the divergence of the quantum relative entropy
implies that the states
and
are orthogonal or even very different by other measures. Specifically,
can diverge when
and
differ by a vanishingly small amount as measured by some norm. For example, let
have the diagonal representation
with
for
and
for
where
is an orthonormal set. The kernel of
is the space spanned by the set
. Next let
for a small positive number
. As
has support (namely the state
) in the kernel of
,
is divergent even though the trace norm of the difference
is
. This means that difference between
and
as measured by the trace norm is vanishingly small as
even though
is divergent (i.e. infinite). This property of the quantum relative entropy represents a serious shortcoming if not treated with care.
Non-negativity of relative entropy
Corresponding classical statement
For the classical
Kullback–Leibler divergence, it can be shown that
![{\displaystyle D_{\mathrm {KL} }(P\|Q)=\sum _{j}p_{j}\log {\frac {p_{j}}{q_{j}}}\geq 0,}](https://wikimedia.org/api/rest_v1/media/math/render/svg/fe84a7601ddb6b2c29d5fce327b667aabda04671)
and the equality holds if and only if P = Q. Colloquially, this means that the uncertainty calculated using erroneous assumptions is always greater than the real amount of uncertainty.
To show the inequality, we rewrite
![{\displaystyle D_{\mathrm {KL} }(P\|Q)=\sum _{j}p_{j}\log {\frac {p_{j}}{q_{j}}}=\sum _{j}(-\log {\frac {q_{j}}{p_{j}}})(p_{j}).}](https://wikimedia.org/api/rest_v1/media/math/render/svg/9b81c462f01852e9c51018cb573dcfcfb50988f1)
Notice that log is a
concave function. Therefore -log is
convex. Applying
Jensen's inequality, we obtain
![{\displaystyle D_{\mathrm {KL} }(P\|Q)=\sum _{j}(-\log {\frac {q_{j}}{p_{j}}})(p_{j})\geq -\log(\sum _{j}{\frac {q_{j}}{p_{j}}}p_{j})=0.}](https://wikimedia.org/api/rest_v1/media/math/render/svg/4d96b47daace0754161e11bee51da2ca35666a31)
Jensen's inequality also states that equality holds if and only if, for all i, qi = (Σqj) pi, i.e. p = q.
The result
Klein's inequality states that the quantum relative entropy
![{\displaystyle S(\rho \|\sigma )=\operatorname {Tr} \rho (\log \rho -\log \sigma ).}](https://wikimedia.org/api/rest_v1/media/math/render/svg/d4300ac718c17a7db198e9b96cd027334b4d2642)
is non-negative in general. It is zero if and only if ρ = σ.
Proof
Let ρ and σ have spectral decompositions
![{\displaystyle \rho =\sum _{i}p_{i}v_{i}v_{i}^{*}\;,\;\sigma =\sum _{i}q_{i}w_{i}w_{i}^{*}.}](https://wikimedia.org/api/rest_v1/media/math/render/svg/643e19cb58f0a95396b350734a32abb20e0fdd88)
So
![{\displaystyle \log \rho =\sum _{i}(\log p_{i})v_{i}v_{i}^{*}\;,\;\log \sigma =\sum _{i}(\log q_{i})w_{i}w_{i}^{*}.}](https://wikimedia.org/api/rest_v1/media/math/render/svg/e7e84aebbf3c37e76b2524fe060db49996bf165e)
Direct calculation gives
![{\displaystyle S(\rho \|\sigma )=\sum _{k}p_{k}\log p_{k}-\sum _{i,j}(p_{i}\log q_{j})|v_{i}^{*}w_{j}|^{2}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/d63e775d39ed491b6cf80c4313b293dc55038307)
![{\displaystyle \qquad \quad \;=\sum _{i}p_{i}(\log p_{i}-\sum _{j}\log q_{j}|v_{i}^{*}w_{j}|^{2})}](https://wikimedia.org/api/rest_v1/media/math/render/svg/55b6984e390783af2da753e1a869bd73fdb30aaf)
![{\displaystyle \qquad \quad \;=\sum _{i}p_{i}(\log p_{i}-\sum _{j}(\log q_{j})P_{ij}),}](https://wikimedia.org/api/rest_v1/media/math/render/svg/44ecc7bf9d93e065993b7dd1299dace533d87cd5)
where Pi j = |vi*wj|2.
Since the matrix (Pi j)i j is a
doubly stochastic matrix and -log is a convex function, the above expression is
![{\displaystyle \geq \sum _{i}p_{i}(\log p_{i}-\log(\sum _{j}q_{j}P_{ij})).}](https://wikimedia.org/api/rest_v1/media/math/render/svg/8140e9627bddcb74661faa38a2fc50102994c934)
Define ri = Σjqj Pi j. Then {ri} is a probability distribution. From the non-negativity of classical relative entropy, we have
![{\displaystyle S(\rho \|\sigma )\geq \sum _{i}p_{i}\log {\frac {p_{i}}{r_{i}}}\geq 0.}](https://wikimedia.org/api/rest_v1/media/math/render/svg/6fd07c6f682d3379124b4bbdbd53e8d0e963e55b)
The second part of the claim follows from the fact that, since -log is strictly convex, equality is achieved in
![{\displaystyle \sum _{i}p_{i}(\log p_{i}-\sum _{j}(\log q_{j})P_{ij})\geq \sum _{i}p_{i}(\log p_{i}-\log(\sum _{j}q_{j}P_{ij}))}](https://wikimedia.org/api/rest_v1/media/math/render/svg/269a7c086006e88b074e22df7ca84e888008c07a)
if and only if (Pi j) is a
permutation matrix, which implies ρ = σ, after a suitable labeling of the eigenvectors {vi} and {wi}.
The relative entropy is
jointly convex. For
and states
we have
The relative entropy decreases monotonically under
completely positive
trace preserving (CPTP) operations
on density matrices,
.
This inequality is called Monotonicity of quantum relative entropy and was first proved by
Lindblad.
An entanglement measure
Let a composite quantum system have state space
![{\displaystyle H=\otimes _{k}H_{k}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/ad7c3d4b828af6873c04d50c1964c450e39e1d1e)
and ρ be a density matrix acting on H.
The relative entropy of entanglement of ρ is defined by
![{\displaystyle \;D_{\mathrm {REE} }(\rho )=\min _{\sigma }S(\rho \|\sigma )}](https://wikimedia.org/api/rest_v1/media/math/render/svg/c86318235f674e36649d491e6d2ec307cb2c006d)
where the minimum is taken over the family of
separable states. A physical interpretation of the quantity is the optimal distinguishability of the state ρ from separable states.
Clearly, when ρ is not
entangled
![{\displaystyle \;D_{\mathrm {REE} }(\rho )=0}](https://wikimedia.org/api/rest_v1/media/math/render/svg/ce2655265d461371bc4b36bf7dfefff91ce79195)
by Klein's inequality.
Relation to other quantum information quantities
One reason the quantum relative entropy is useful is that several other important quantum information quantities are special cases of it. Often, theorems are stated in terms of the quantum relative entropy, which lead to immediate corollaries concerning the other quantities. Below, we list some of these relations.
Let ρAB be the joint state of a bipartite system with subsystem A of dimension nA and B of dimension nB. Let ρA, ρB be the respective reduced states, and IA, IB the respective identities. The
maximally mixed states are IA/nA and IB/nB. Then it is possible to show with direct computation that
![{\displaystyle S(\rho _{A}||I_{A}/n_{A})=\mathrm {log} (n_{A})-S(\rho _{A}),\;}](https://wikimedia.org/api/rest_v1/media/math/render/svg/cc11ce53cdbc805015adf0b9c23bb47fde98d9db)
![{\displaystyle S(\rho _{AB}||\rho _{A}\otimes \rho _{B})=S(\rho _{A})+S(\rho _{B})-S(\rho _{AB})=I(A:B),}](https://wikimedia.org/api/rest_v1/media/math/render/svg/95657f16075fc31476067956a736ccba7fccd92c)
![{\displaystyle S(\rho _{AB}||\rho _{A}\otimes I_{B}/n_{B})=\mathrm {log} (n_{B})+S(\rho _{A})-S(\rho _{AB})=\mathrm {log} (n_{B})-S(B|A),}](https://wikimedia.org/api/rest_v1/media/math/render/svg/cc9c75cecfd88efa2684a7916a869d940fe52b25)
where I(A:B) is the
quantum mutual information and S(B|A) is the
quantum conditional entropy.
References