Learnt a new mathematical concept: Bregman divergence. Knew about KullbackLeibler divergence but this one I didn't know about. The concept of dual space is very interesting. the following is from Wiki page.
Definition[edit]
Let convex function defined on a closed convex set .
be a continuouslydifferentiable realvalued and strictly
The Bregman distance associated with F for points Taylor expansion of Faround point q evaluated at point p:
is the difference between the value of F at point p and the value of the firstorder Properties[edit]
 Nonnegativity: for all p, q. This is a consequence of the convexity of F.
 Convexity: is convex in its first argument, but not necessarily in the second argument (see ^{[1]})
 Linearity: If we think of the Bregman distance as an operator on the function F, then it is linear with respect to nonnegative coefficients. In other words, for strictly convex and differentiable, and ,
 Duality: The function F has a convex conjugate . The Bregman distance defined with respect to has an interesting relationship to
 Here, and are the dual points corresponding to p and q.
 Mean as minimizer: A key result about Bregman divergences is that, given a random vector, the mean vector minimizes the expected Bregman divergence from the random vector. This result generalizes the textbook result that the mean of a set minimizes total squared error to elements in the set. This result was proved for the vector case by (Banerjee et al. 2005), and extended to the case of functions/distributions by (Frigyik et al. 2008). This result is important because it further justifies using a mean as a representative of a random set, particularly in Bayesian estimation.
Examples[edit]
 Squared Euclidean distance is the canonical example of a Bregman distance, generated by the convex function
 The squared Mahalanobis distance, which is generated by the convex function . This can be thought of as a generalization of the above squared Euclidean distance.
 The generalized Kullback–Leibler divergence

 is generated by the convex function

 is generated by the convex function
No comments:
Post a Comment