Exploiting the Surrogate Gap in Online Multiclass Classification

Exploiting the Surrogate Gap in Online Multiclass Classiﬁcation

Dirk van der Hoeven

Leiden University

Dirk van der Hoeven (Leiden University) Exploiting the Surrogate Gap 1 / 30 This talk

1 Brief introduction Online Multiclass Classiﬁcation 2 My contributions 3 Some intuition about my contributions 4 Future work

Dirk van der Hoeven (Leiden University) Exploiting the Surrogate Gap 2 / 30 Probably AFC Ajax, but not absolutely certain.

If we gather this information for all eredivisie games, can we predict perfectly? Probably not

Important: regardless of what we predict, we will see the true outcome

Example Application Full Information

Football prediction. ADO Den Haag versus AFC Ajax. We know that ADO plays at home There are 0 supporters for either side Players of ADO Den Haag are valued 11.35 million euros Players of AFC Ajax are valued 288.85 million euros

Who will win?

Dirk van der Hoeven (Leiden University) Exploiting the Surrogate Gap 3 / 30 Probably not

Important: regardless of what we predict, we will see the true outcome

Example Application Full Information

Who will win? Probably AFC Ajax, but not absolutely certain.

If we gather this information for all eredivisie games, can we predict perfectly?

Dirk van der Hoeven (Leiden University) Exploiting the Surrogate Gap 3 / 30 Example Application Full Information

Who will win? Probably AFC Ajax, but not absolutely certain.

If we gather this information for all eredivisie games, can we predict perfectly? Probably not

Important: regardless of what we predict, we will see the true outcome

Dirk van der Hoeven (Leiden University) Exploiting the Surrogate Gap 3 / 30 Goal: minimize the expected surrogate regret RT   " T # T X X RT = E 1[yt 6=y ˆt ] − min `(hU, xt i, yt )  U  t=1 t=1 | {z } margin

Setting: Full Information

The online multiclass classiﬁcation setting proceeds in rounds t = 1,..., T . In each round t

1 the environment picks an outcome yt ∈ {1,... K} and reveals a d feature vector xt ∈ R to the learner 2 the learner issues a (randomized) prediction yˆt

3 the environment reveals yt

4 the learner suﬀers 1[yt 6=y ˆt ]

Dirk van der Hoeven (Leiden University) Exploiting the Surrogate Gap 4 / 30 Setting: Full Information

The online multiclass classiﬁcation setting proceeds in rounds t = 1,..., T . In each round t

1 the environment picks an outcome yt ∈ {1,... K} and reveals a d feature vector xt ∈ R to the learner 2 the learner issues a (randomized) prediction yˆt

3 the environment reveals yt

4 the learner suﬀers 1[yt 6=y ˆt ]

Goal: minimize the expected surrogate regret RT   " T # T X X RT = E 1[yt 6=y ˆt ] − min `(hU, xt i, yt )  U  t=1 t=1 | {z } margin

Dirk van der Hoeven (Leiden University) Exploiting the Surrogate Gap 4 / 30 Surrogate Loss with K = 2

2.0 surrogate loss

1.5 if margin postive predict 1, otherwise predict 2 zero-one loss 1.0

0.5

0.0

-1.0 -0.5 0.0 0.5 1.0 margin

Dirk van der Hoeven (Leiden University) Exploiting the Surrogate Gap 5 / 30 Is regret a reasonable measure of performance?

Goal: minimize the expected surrogate regret RT   " T #  T  X  X  RT = E 1[yt 6=y ˆt ] −  min `(hU, xt i, yt )   U  t=1  t=1  | {z } = 0 If model predicts perfectly

Translation: I want to be close to the performance of the best oﬄine version of the model.

Dirk van der Hoeven (Leiden University) Exploiting the Surrogate Gap 6 / 30 probably not.

What is your strategy for the suggestion? mine: randomly select one movie, I don’t have any information

Important: You don’t get feedback about what you should have suggested

Example application bandit setting

You have to suggest a movie to friend from a list of movies you like (and suppose there is only 1 movie in this list your friend will like). You know: the length of your friend the weight of your friend Can you give the perfect suggestion?

Dirk van der Hoeven (Leiden University) Exploiting the Surrogate Gap 7 / 30 mine: randomly select one movie, I don’t have any information

Important: You don’t get feedback about what you should have suggested

Example application bandit setting

What is your strategy for the suggestion?

Dirk van der Hoeven (Leiden University) Exploiting the Surrogate Gap 7 / 30 Example application bandit setting

What is your strategy for the suggestion? mine: randomly select one movie, I don’t have any information

Important: You don’t get feedback about what you should have suggested

Dirk van der Hoeven (Leiden University) Exploiting the Surrogate Gap 7 / 30 Setting: Bandit

The bandit online multiclass classiﬁcation setting proceeds in rounds t = 1,..., T . In each round t

1 the environment picks an outcome yt ∈ {1,... K} and reveals a feature vector xt to the learner

2 the learner issues a (randomized) prediction yˆt

3 the environment reveals 1[yt 6=y ˆt ]

4 the learner suﬀers 1[yt 6=y ˆt ]

Goal: minimize the expected surrogate regret RT    T ! T X X RT = E  1[yt 6=y ˆt ] − min `(hU, xt i, yt )   U  t=1 t=1 | {z } margin

Dirk van der Hoeven (Leiden University) Exploiting the Surrogate Gap 8 / 30 Results 1

Results for the full information setting

Algorithm R Time (per round) T√ Standard ﬁrst-order O(kUk T ) O(dK) Standard second-order O(ekUkdK ln(T )) O((dK)2) Gaptron O(KkUk2) O(dK)

Dirk van der Hoeven (Leiden University) Exploiting the Surrogate Gap 9 / 30 Results 2

Results for the bandit setting

Algorithm E[RT ] Time (per round) Standard ﬁrst-order O((K)1/3T 2/3) O(dK) p 2 Standard second-order O(K dT√ln(T )) O((dK) ) Gaptron O(K T ) O(dK)

Dirk van der Hoeven (Leiden University) Exploiting the Surrogate Gap 10 / 30   T X 1   [yt 6=y ˆt ] − `(hWt , xt i, yt ) t=1 | {z } Very wastefull: bound by 0 T ! X + `(hWt , xt i, yt ) − `(hU, xt i, yt ) t=1 T X ≤ `(hWt , xt i, yt ) − `(hU, xt i, yt ) t=1 | {z } controlled by OGD T kUk2 η X ≤ + k∇ `(hW , x i, y )k2 2η 2 Wt t t t √ t=1 ≤kUkX T

Old Analysis: Upper Bound Zero-One Loss with Surrogate

T X 1[yt 6=y ˆt ] − `(hU, xt i, yt ) = t=1

Dirk van der Hoeven (Leiden University) Exploiting the Surrogate Gap 11 / 30 T X ≤ `(hWt , xt i, yt ) − `(hU, xt i, yt ) t=1 | {z } controlled by OGD T kUk2 η X ≤ + k∇ `(hW , x i, y )k2 2η 2 Wt t t t √ t=1 ≤kUkX T

Old Analysis: Upper Bound Zero-One Loss with Surrogate

  T T X 1 X 1  [yt 6=y ˆt ] − `(hU, xt i, yt ) =  [yt 6=y ˆt ] − `(hWt , xt i, yt ) t=1 t=1 | {z } Very wastefull: bound by 0 T ! X + `(hWt , xt i, yt ) − `(hU, xt i, yt ) t=1

Dirk van der Hoeven (Leiden University) Exploiting the Surrogate Gap 11 / 30 T kUk2 η X ≤ + k∇ `(hW , x i, y )k2 2η 2 Wt t t t √ t=1 ≤kUkX T

Old Analysis: Upper Bound Zero-One Loss with Surrogate

  T T X 1 X 1  [yt 6=y ˆt ] − `(hU, xt i, yt ) =  [yt 6=y ˆt ] − `(hWt , xt i, yt ) t=1 t=1 | {z } Very wastefull: bound by 0 T ! X + `(hWt , xt i, yt ) − `(hU, xt i, yt ) t=1 T X ≤ `(hWt , xt i, yt ) − `(hU, xt i, yt ) t=1 | {z } controlled by OGD

Dirk van der Hoeven (Leiden University) Exploiting the Surrogate Gap 11 / 30 √ ≤kUkX T

Old Analysis: Upper Bound Zero-One Loss with Surrogate

Dirk van der Hoeven (Leiden University) Exploiting the Surrogate Gap 11 / 30 Old Analysis: Upper Bound Zero-One Loss with Surrogate

Dirk van der Hoeven (Leiden University) Exploiting the Surrogate Gap 11 / 30 How to improve upon standard methods?

2.0 surrogate loss

1.5 if margin postive predict 1, otherwise predict 2 zero-one loss 1.0

0.5

0.0

-1.0 -0.5 0.0 0.5 1.0 margin

Dirk van der Hoeven (Leiden University) Exploiting the Surrogate Gap 12 / 30 Key Idea: when uncertain, randomize.

2.0 surrogate loss

1.5 Gaptron: add randomness to exploit surrogate gap zero-one 1.0 Gaptron

0.5

0.0

-1.0 -0.5 0.0 0.5 1.0 margin

Dirk van der Hoeven (Leiden University) Exploiting the Surrogate Gap 13 / 30 T ! X E [1[yt 6=y ˆt ]] − `(hWt , xt i, yt ) t=1

T X + `(hWt , xt i, yt ) − `(hU, xt i, yt ) t=1 | {z } controlled by OGD kUk2 ≤ 2η   T X 1 η 2  +  E [ [yt 6=y ˆt ]] + k∇Wt `(hWt , xt i, yt )k − `(hWt , xt i, yt )  2  t=1 | {z } Smaller than surrogate loss!

Gaptron Analysis: ﬁrst steps

T X E [1[yt 6=y ˆt ]] − `(hU, xt i, yt ) = t=1 | {z } Gaptron is random

Dirk van der Hoeven (Leiden University) Exploiting the Surrogate Gap 14 / 30 kUk2 ≤ 2η   T X 1 η 2  +  E [ [yt 6=y ˆt ]] + k∇Wt `(hWt , xt i, yt )k − `(hWt , xt i, yt )  2  t=1 | {z } Smaller than surrogate loss!

Gaptron Analysis: ﬁrst steps

T T ! X X E [1[yt 6=y ˆt ]] − `(hU, xt i, yt ) = E [1[yt 6=y ˆt ]] − `(hWt , xt i, yt ) t=1 | {z } t=1 Gaptron is random T X + `(hWt , xt i, yt ) − `(hU, xt i, yt ) t=1 | {z } controlled by OGD

Dirk van der Hoeven (Leiden University) Exploiting the Surrogate Gap 14 / 30 Gaptron Analysis: ﬁrst steps

T T ! X X E [1[yt 6=y ˆt ]] − `(hU, xt i, yt ) = E [1[yt 6=y ˆt ]] − `(hWt , xt i, yt ) t=1 | {z } t=1 Gaptron is random T X + `(hWt , xt i, yt ) − `(hU, xt i, yt ) t=1 | {z } controlled by OGD kUk2 ≤ 2η   T X 1 η 2  +  E [ [yt 6=y ˆt ]] + k∇Wt `(hWt , xt i, yt )k − `(hWt , xt i, yt )  2  t=1 | {z } Smaller than surrogate loss!

Dirk van der Hoeven (Leiden University) Exploiting the Surrogate Gap 14 / 30 Gaptron Analysis: is that really smaller than surrogate loss?

2.0 surrogate loss

1.5 Shaded area represents eta times norm of gradient

1.0 Gaptron

0.5

0.0

-1.0 -0.5 0.0 0.5 1.0 margin

Dirk van der Hoeven (Leiden University) Exploiting the Surrogate Gap 15 / 30 √ Full information before: kUkX T regret

Full information now: KX 2kUk2 regret

Gaptron Analysis: wow, that is very useful!

T X kUk2 [1[y 6=y ˆ ]] − `(hU, x i, y ) ≤ E t t t t 2η t=1   T X 1 η 2  +  E [ [yt 6=y ˆt ]] + k∇Wt `(hWt , xt i, yt )k − `(hWt , xt i, yt )  2  t=1 | {z } Smaller than surrogate loss! ≤ KX 2kUk2

Dirk van der Hoeven (Leiden University) Exploiting the Surrogate Gap 16 / 30 Gaptron Analysis: wow, that is very useful!

√ Full information before: kUkX T regret

Full information now: KX 2kUk2 regret

Dirk van der Hoeven (Leiden University) Exploiting the Surrogate Gap 16 / 30 Gaptron Analysis: wow, that is very useful!

Bandit before: O(KpdT ln(T )) regret with O((dK)2) runtime √ Bandit now: O(K T ) regret with O(dK) runtime (very cool in high-dimensional applications)

Dirk van der Hoeven (Leiden University) Exploiting the Surrogate Gap 17 / 30 A closer look at Gaptron

Input: Learning rate η > 0, exploration rate γ ∈ [0, 1], and gap map K×d d a : R × R → [0, 1] 1: Initialize W1 = 0 2: for t = 1 ... T do 3: Obtain xt ? k 4: Let yt = arg maxk hWt , xt i 0 1 5: Set p = (1 − max{a(W , x ), γ})e ? + max{a(W , x ), γ} 1 t t t yt t t K 0 6: Predict with label yˆt ∼ pt 7: Obtain 1[ˆyt 6= yt ] and set gt = ∇`t (Wt ) 1 2 8: Update Wt+1 = arg minW ∈W ηhgt , W i + 2 kW − Wt k 9: end for

Dirk van der Hoeven (Leiden University) Exploiting the Surrogate Gap 18 / 30 Choosing the right a ensures that the expected loss of Gaptron plus the norm of the gradient is smaller than the surrogate loss. If a(W , x) = 0 we recover standard algorithms such as the Perceptron. 1 If γ > 0 we sample any outcome with probability at least γ K , which is important for the bandit setting

A closer look at Gaptrons predictions

0 1 p = (1 − max{a(W , x ), γ})e ? + max{a(W , x ), γ} 1 t t t yt t t K | {z ? } | {z } I think the outcome is yt But I am not certain

Dirk van der Hoeven (Leiden University) Exploiting the Surrogate Gap 19 / 30 1 If γ > 0 we sample any outcome with probability at least γ K , which is important for the bandit setting

A closer look at Gaptrons predictions

0 1 p = (1 − max{a(W , x ), γ})e ? + max{a(W , x ), γ} 1 t t t yt t t K | {z ? } | {z } I think the outcome is yt But I am not certain Choosing the right a ensures that the expected loss of Gaptron plus the norm of the gradient is smaller than the surrogate loss. If a(W , x) = 0 we recover standard algorithms such as the Perceptron.

Dirk van der Hoeven (Leiden University) Exploiting the Surrogate Gap 19 / 30 A closer look at Gaptrons predictions

Dirk van der Hoeven (Leiden University) Exploiting the Surrogate Gap 19 / 30 Rest of the paper: ﬁnding the correct a, η, and γ to bound the surrogate gap by 0

Main Lemma of the paper

Lemma

For any U ∈ W Gaptron satisﬁes

" T T # X X E 1[ˆyt 6= yt ] − `t (U) t=1 t=1 kUk2 K − 1 ≤ + γ T 2η K T X K − 1 η + (1 − a )1[y ? 6= y ] + a − ` (W ) + kg k2 . E t t t t K t t 2 t t=1 | {z } surrogate gap

Dirk van der Hoeven (Leiden University) Exploiting the Surrogate Gap 20 / 30 Main Lemma of the paper

Lemma

For any U ∈ W Gaptron satisﬁes

" T T # X X E 1[ˆyt 6= yt ] − `t (U) t=1 t=1 kUk2 K − 1 ≤ + γ T 2η K T X K − 1 η + (1 − a )1[y ? 6= y ] + a − ` (W ) + kg k2 . E t t t t K t t 2 t t=1 | {z } surrogate gap

Rest of the paper: ﬁnding the correct a, η, and γ to bound the surrogate gap by 0

Dirk van der Hoeven (Leiden University) Exploiting the Surrogate Gap 20 / 30 A function f is H-smooth if H f (x + z) ≤ f (x) + h∇f (x), zi + kzk2. 2 2 ? Let Ut = arg minW `t (W ). For smooth surrogate losses we have

2 ? k∇`t (Wt )k2 ≤ H(`t (Wt ) − `t (Ut )) = H`t (Wt )

Choosing at and η for smooth losses in 2 dimensions

In 2 dimensions yt ∈ {−1, +1} and `t (W ) = `(hW , xt iyt ).

Dirk van der Hoeven (Leiden University) Exploiting the Surrogate Gap 21 / 30 Choosing at and η for smooth losses in 2 dimensions

In 2 dimensions yt ∈ {−1, +1} and `t (W ) = `(hW , xt iyt ). A function f is H-smooth if H f (x + z) ≤ f (x) + h∇f (x), zi + kzk2. 2 2 ? Let Ut = arg minW `t (W ). For smooth surrogate losses we have

2 ? k∇`t (Wt )k2 ≤ H(`t (Wt ) − `t (Ut )) = H`t (Wt )

Dirk van der Hoeven (Leiden University) Exploiting the Surrogate Gap 21 / 30 Choosing at and η for smooth losses

For smooth surrogate losses we have: K − 1 η (1 − a )1[y ? 6= y ] + a − ` (W ) + kg k2 t t t t K t t 2 t K − 1 ηH ≤ (1 − a )1[y ? 6= y ] + a − ` (W ) + ` (W ). t t t t K t t 2 t t

? ? 2 Picking at = `t (Wt ) = `(hWt , xt iyt ) and η = HK we have

K − 1 η (1 − a )1[y ? 6= y ] + a − ` (W ) + kg k2 t t t t K t t 2 t K − 1 K − 1 ≤ (1 − `?(W ))1[y ? 6= y ] + `?(W ) − ` (W ). t t t t t t K K t t

Dirk van der Hoeven (Leiden University) Exploiting the Surrogate Gap 22 / 30 Choosing at and η for smooth losses

? If yt = yt : K − 1 η (1 − a )1[y ? 6= y ] + a − ` (W ) + kg k2 t t t t K t t 2 t K − 1 K − 1 ≤ (1 − `?(W ))1[y ? 6= y ] + `?(W ) − ` (W ) t t t t t t K K t t K − 1 K − 1 = ` (W ) − ` (W ) t t K K t t = 0

Dirk van der Hoeven (Leiden University) Exploiting the Surrogate Gap 23 / 30 Choosing at and η for smooth losses

? If yt 6= yt : K − 1 η (1 − a )1[y ? 6= y ] + a − ` (W ) + kg k2 t t t t K t t 2 t K − 1 K − 1 ≤ (1 − `?(W ))1[y ? 6= y ] + `?(W ) − ` (W ) t t t t t t K K t t 1 K − 1 = 1 − `?(W ) − ` (W ) K t t K t t 1 K − 1 = 1 − `(hW , x iy ?) − `(hW , x iy ) K t t t K t t t 1 K − 1 ≤ 1 − `(hW , x i( y ? + y )) t t K t K t | {z } opposite sign of hWt ,xt i ≤ 0

Dirk van der Hoeven (Leiden University) Exploiting the Surrogate Gap 24 / 30 Inspiration from classiﬁcation with abstention

Setting: i 1 the learner observes the predictions yt ∈ [−1, 1] of experts i = 1,..., d 0 2 based on the experts’ predictions the learner predicts yt ∈ [−1, 1] ∪ ∗, where ∗ stands for abstaining

3 the environment reveals yt ∈ {−1, 1} 0 1 0 0 4 the learner suﬀers loss `t (yt ) = 2 (1 − yt yt ) if yt ∈ [−1, 1] and ct otherwise.

Dirk van der Hoeven (Leiden University) Exploiting the Surrogate Gap 25 / 30 Inspiration from classiﬁcation with abstention

Algorithm: Input: AdaHedge 1: for t = 1 ... T do 1 d > d 2: Obtain expert predictions yt = (yt ,..., yt ) ∈ [−1, 1] 3: Obtain expert distribution pˆt from AdaHedge 4: Set yˆt = hpˆt , yt i ? 5: Let yt = sign(ˆyt ) 6: Set bt = 1 − |yˆt | 0 ? 0 7: Predict yt = yt with probability 1 − bt and predict yt = ∗ with probability bt 8: Obtain `t and send `t to AdaHedge 9: end for

Dirk van der Hoeven (Leiden University) Exploiting the Surrogate Gap 26 / 30 Inspiration from classiﬁcation with abstention

Lemma For any expert i, the expected loss of satisﬁes:

T X ? (1 − bt )`t (yt ) + bt ct t=1   T  T  X i ln(d) X ?  ≤ `t (yt ) + inf + (1 − bt )`t (yt ) + ct bt + ηvt − `t (ˆyt ) η>0 η t=1  t=1 | {z }  Abstention gap  4 + ln(d) + 2, 3

i 2 where vt = Ei∼pˆt [(`t (ˆyt ) − `t (yt )) ].

Dirk van der Hoeven (Leiden University) Exploiting the Surrogate Gap 27 / 30 Inspiration from classiﬁcation with abstention

Abstention gap

? (1 − bt )`t (yt ) + bt ct + ηvt − `t (ˆyt )

Surrogate gap: η (1 − a )1[y ? 6= y ] + a c0 + kg k2 − ` (W ) t t t t t 2 t t t 0 ct is the cost for guessing rather than abstaining, although if abstaining 0 costs strictly less than 1 we could replace ct with abstention cost ct and obtain surrogate regret that satisﬁes

2 2 kXk2kUk2 RT (U) = O 1 − maxt ct

Dirk van der Hoeven (Leiden University) Exploiting the Surrogate Gap 28 / 30 Future work

High probability bounds Can we exploit curvature to improve regret bound? Empirical performance? Can we improve regret when the model can predict perfectly (both bandit and full information)?

I For full information, I think yes. I For bandit setting, I am not sure. Can we apply this idea in other problems such as ranking?

Dirk van der Hoeven (Leiden University) Exploiting the Surrogate Gap 29 / 30 Where to ﬁnd the paper?

my website: dirkvanderhoeven.com/research arxiv: https://arxiv.org/abs/2007.12618 NeurIPS 2020 proceedings By googling "gaptron" (the twitter and instagram accounts are not mine).

Dirk van der Hoeven (Leiden University) Exploiting the Surrogate Gap 30 / 30