On the duality of strong convexity and strong smoothness: Learning applications and matrix regularization Sham M. Kakade and Shai Shalev-Shwartz and Ambuj Tewari Toyota Technological Institute—Chicago, USA fsham,shai,
[email protected] Abstract strictly convex functions). Here, the notion of strong con- vexity provides one means to characterize this gap in terms We show that a function is strongly convex with of some general norm (rather than just Euclidean). respect to some norm if and only if its conjugate This work examines the notion of strong convexity from function is strongly smooth with respect to the a duality perspective. We show a function is strongly convex dual norm. This result has already been found with respect to some norm if and only if its (Fenchel) con- to be a key component in deriving and analyz- jugate function is strongly smooth with respect to its dual ing several learning algorithms. Utilizing this du- norm. Roughly speaking, this notion of smoothness (defined ality, we isolate a single inequality which seam- precisely later) provides a second order upper bound of the lessly implies both generalization bounds and on- conjugate function, which has already been found to be a key line regret bounds; and we show how to construct component in deriving and analyzing several learning algo- strongly convex functions over matrices based on rithms. Using this relationship, we are able to characterize strongly convex functions over vectors. The newly a number of matrix based penalty functions, of recent in- constructed functions (over matrices) inherit the terest, as being strongly convex functions, which allows us strong convexity properties of the underlying vec- to immediately derive online algorithms and generalization tor functions.