Zero-Shot Learning Problem Deﬁnition Semantic Labels Visual Feature Space

Semi-supervised Yanwei Fu Vocabulary- Leonid Sigal informed Learning

Research Supervised Learning Problem Deﬁnition Supervised Learning Problem Deﬁnition Visual feature space

x2 Supervised Learning Problem Deﬁnition Semantic labels Visual feature space

airplane car

unicycle tricycle

x2 Supervised Learning Learning Semantic labels Visual feature space

airplane car

unicycle tricycle

x2 Supervised Learning Learning Semantic labels Visual feature space

x1 airplane car

unicycle tricycle

x2 Supervised Learning Learning Semantic labels Visual feature space

x1 airplane

car

unicycle tricycle

x2 Supervised Learning Learning Semantic labels Visual feature space

x1 airplane

car

unicycle tricycle

x2 Supervised Learning Inference Semantic labels Visual feature space

x1 airplane

car

unicycle tricycle

x2 Supervised Learning Inference Semantic labels Visual feature space

x1 airplane

car

unicycle tricycle

x2 Supervised Learning Inference Semantic labels Visual feature space

x1 airplane

car

unicycle tricycle

x2 Zero-shot Learning Problem Deﬁnition Semantic labels Visual feature space

x We do not have any visually labeled 1 instances of what these look like

bicycle

truck

x2 Zero-shot Learning Learning Semantic label space Visual feature space

x1 v1

airplane unicycle

bicycle tricycle v2

car

v3 truck

x2 Zero-shot Learning Learning f(x) Semantic label space Visual feature space

x1 v1 Word vector-feature category mapping: airplane v = f(x) unicycle

bicycle tricycle v2

car

v3 truck

x2 Zero-shot Learning Inference f(x) Semantic label space Visual feature space

x1 v1

airplane unicycle

bicycle tricycle v2 Word vector-feature category mapping: v = f(x) car

v3 truck

x2 Zero-shot Learning Inference f(x) Semantic label space Visual feature space

x1 v1

airplane unicycle

bicycle tricycle v2 Word vector-feature category mapping: v = f(x) car

v3 truck f( ) x2 Zero-shot Learning Inference f(x) Semantic label space Visual feature space

x1 v1

airplane unicycle

bicycle tricycle v2 Word vector-feature category mapping: v = f(x) car

v3 truck f( ) x2 Zero-shot Learning Inference f(x) Semantic label space Visual feature space

x1 v1

airplane unicycle

bicycle tricycle v2 Word vector-feature category mapping: v = f(x) car

v3 truck f( ) x2 Zero-shot Learning Inference f(x) Semantic label space Visual feature space

x1 v1

airplane unicycle

bicycle tricycle v2 Word vector-feature category mapping: v = f(x) car

v3 truck f( ) x2 Key Question: How do we deﬁne semantic space? Semantic Label Vector Spaces

Spaces Type Advantages Disadvantages

Manual annotation Good interpretability of each Semantic Supervised dimension: Attributes Limited vocabulary airplane := ﬁxed wing, propelled, has pilot

Semantic Word Good vector representation for Limited interpretability of Vectors Unsupervised millions of vocabulary each dimension (e.g. word2vec) v (Berlin) v (Germany) = v (Paris) v (France) Semantic Label Vector Spaces

Spaces Type Advantages Disadvantages

Manual annotation Good interpretability of each Semantic Supervised dimension: Attributes Limited vocabulary airplane := ﬁxed wing, propelled, has pilot

x1 v1

airplane unicycle

bicycle tricycle v2

car

v3 truck f( ) x2 Open-set Recognition Problem Deﬁnition f(x) Semantic label space Visual feature space

x1 v1

airplane unicycle

bicycle tricycle v2

car

v3 truck f( ) x2 Open-set Recognition Problem Deﬁnition f(x) Semantic label space Visual feature space

copilot ﬂight learjet x jet 1 front-landing- v1 crop_duster tailwhip dirigible gear pedaling wheelie airplane biplane pogo_stick roller-skate twin-engine- helicopter unicycle aircraft tri-motor jump_rope horse cock- velomobile segway bicyclecargo_bikes tricycle v2 penny-farthing motor-scooter mopped trike motorcycle recumbent- motorbike car bicycle suv mini-van

v3 tow-truck ﬂat- truck tyres f( ) x2 Summary:

Supervised Learning: [Fei-Fei et al. TPAMI’06], [Deng et al. ECCV’14], [Torralba et al. TPAMI’08 ], [Weston et al. IJCAI’11]

Pros: Very good quantitative performance Cons: Relatively small vocabulary (~1,000 classes) Requires manual labeling of all the data Summary:

Supervised Learning: [Fei-Fei et al. TPAMI’06], [Deng et al. ECCV’14], [Torralba et al. TPAMI’08 ], [Weston et al. IJCAI’11]

Pros: Very good quantitative performance Cons: Relatively small vocabulary (~1,000 classes) Requires manual labeling of all the data

Zero-shot Learning: [Palatucci et al. NIPS’09],[Lampert et al. CVPR’09], [Farhadi et al. CVPR’09], [Rohrbach et al. CVPR’10]

Pros: Does not require instance labeling for target classes Cons: Typically limited to recognition with target classes only Relatively small vocabulary (~50-200 classes typically) Summary:

Supervised Learning: [Fei-Fei et al. TPAMI’06], [Deng et al. ECCV’14], [Torralba et al. TPAMI’08 ], [Weston et al. IJCAI’11]

Pros: Very good quantitative performance Cons: Relatively small vocabulary (~1,000 classes) Requires manual labeling of all the data

Zero-shot Learning: [Palatucci et al. NIPS’09],[Lampert et al. CVPR’09], [Farhadi et al. CVPR’09], [Rohrbach et al. CVPR’10]

Pros: Does not require instance labeling for target classes Cons: Typically limited to recognition with target classes only Relatively small vocabulary (~50-200 classes typically)

Open-set Learning: [Scheirer et al. TPAMI’13], [Sattar et al. CVPR’15], [Bendale et al. CVPR’15] [Guadarrama et al. RSS’14] Pros: Does not require instance labeling for target classes Large vocabulary (up to 310K classes) Summary:

Supervised Learning: [Fei-Fei et al. TPAMI’06], [Deng et al. ECCV’14], [Torralba et al. TPAMI’08 ], [Weston et al. IJCAI’11]

Pros: Very good quantitative performance Cons: Relatively small vocabulary (~1,000 classes) Requires manual labeling of all the data

Zero-shot Learning: [Palatucci et al. NIPS’09],[Lampert et al. CVPR’09], [Farhadi et al. CVPR’09], [Rohrbach et al. CVPR’10]

Pros: Does not require instance labeling for target classes Cons: Typically limited to recognition with target classes only Relatively small vocabulary (~50-200 classes typically)

Key Question: Can this large vocabulary be actually useful for recognition? Vocabulary-Informed Recognition v1

unicycle

tricycle v2

v3 Vocabulary-Informed Recognition v1 f( Image)

unicycle

tricycle v2

v3 Vocabulary-Informed Recognition v1 f( Image)

unicycle

tricycle v2

v3 Vocabulary-Informed Recognition v1 f( Image)

unicycle

segway bicycle

recumbent- cargo_bikes tricycle bicycle v2

farthing penny-

v3 Vocabulary-Informed Recognition v1 f( Image)

unicycle

segway bicycle

recumbent- cargo_bikes tricycle bicycle v2

farthing penny-

v3 Formulation Semi-Supervised Vocabulary-Informed Learning (SS-Voc) Regression term Semantic label space f(x)=fW(x) Visual feature space

tricycle v2 penny-farthing motor-scooter mopped trike motorcycle recumbent- motorbike car bicycle suv mini-van

tow-truck v3 ﬂat-tyres

x2 Semi-Supervised Vocabulary-Informed Learning (SS-Voc) Regression term Semantic label space f(x)=fW(x) Visual feature space

tricycle v2 penny-farthing motor-scooter mopped trike motorcycle recumbent- motorbike car bicycle suv mini-van

tow-truck v3 ﬂat-tyres

x2 ✏(f( ,W),uairplane ) L X Semi-Supervised Vocabulary-Informed Learning (SS-Voc) Regression term Semantic label space f(x)=fW(x) Visual feature space

x1 v1

airplane unicycle

tricycle v2

car

x2 ✏(f( ,W),uairplane ) L X Semi-Supervised Vocabulary-Informed Learning (SS-Voc) Regression term Semantic label space f(x)=fW(x) Visual feature space

x1 v1

airplane unicycle

tricycle v2

car

x2 ✏(f( ,W),uairplane ) L X Semi-Supervised Vocabulary-Informed Learning (SS-Voc) Max-margin term Semantic label space f(x)=fW(x) Visual feature space

copilot ﬂight learjet jet front-landing- crop_duster x1 v1 dirigible gear tailwhip pedaling wheelie airplane pogo_stick roller-skate biplane twin-engine- helicopter unicycle aircraft tri-motor jump_rope horse cock- velomobile segway cargo_bikes

tricycle v2 penny-farthing motor-scooter mopped trike motorcycle recumbent- motorbike car bicycle suv mini-van

tow-truck v3 ﬂat-tyres x 1 1 2 [C (f( ),uairplane)+ ✏(f( ),uhelicopter)] 2L✏ 2L X Semi-Supervised Vocabulary-Informed Learning (SS-Voc) Max-margin term Semantic label space f(x)=fW(x) Visual feature space

tricycle v2 penny-farthing motor-scooter mopped trike motorcycle recumbent- motorbike car bicycle suv mini-van

v3 tow-truck ﬂat- tyres Max-margin Constant x 1 1 2 [C (f( ),uairplane)+ ✏(f( ),uhelicopter)] 2L✏ 2L X Semi-Supervised Vocabulary-Informed Learning (SS-Voc) Final full objective Semantic label space f(x)=fW(x) Visual feature space

1 1 ] ✏(f( ,W), uairplaneV )+ [C ✏(f( ),uairplaneV)+ ✏(f( ),uhelicopterV) L 2L 2L X X Semi-Supervised Vocabulary-Informed Learning (SS-Voc) Final full objective Semantic label space f(x)=fW(x) Visual feature space

1 1 ] ✏(f( ,W), uairplaneV )+ [C ✏(f( ),uairplaneV)+ ✏(f( ),uhelicopterV) L 2L 2L X Regression Term X Semi-Supervised Vocabulary-Informed Learning (SS-Voc) Final full objective Semantic label space f(x)=fW(x) Visual feature space

1 1 ] ✏(f( ,W), uairplaneV )+ [C ✏(f( ),uairplaneV)+ ✏(f( ),uhelicopterV) L 2L 2L X Regression Term X Max-margin Term Semi-Supervised Vocabulary-Informed Learning (SS-Voc) Final full objective Semantic label space f(x)=fW(x) Visual feature space

x1 orange

lemon

x2 v3

x1 orange

lemon

x2 v3 Allows ﬁne-tuning of the word-vector space 1 1 ] ✏(f( ,W), uairplaneV )+ [C ✏(f( ),uairplaneV)+ ✏(f( ),uhelicopterV) L 2L 2L X X Word-vector space global transformation

Regression Term Max-margin Term Advantages of the Approach Advantages of the Approach

• A new paradigm for learning informed by very large vocabulary Advantages of the Approach

• A new paradigm for learning informed by very large vocabulary

• A uniﬁed framework for supervised, zero-shot learning Advantages of the Approach

• A new paradigm for learning informed by very large vocabulary

• A uniﬁed framework for supervised, zero-shot learning

• Competitive quantitative performance Advantages of the Approach

• A new paradigm for learning informed by very large vocabulary

• A uniﬁed framework for supervised, zero-shot learning

• Competitive quantitative performance

• Our framework can even scale up to open set image recognition with 310,000 vocabulary entities Evaluation Datasets Datasets

Animals with Attributes(AwA) [Lampert et al. CVPR 2009]: 40 auxiliary classes (24295 images), 10 target classes (6180 images); We use 5 instances per auxiliary class for learning; Datasets

Animals with Attributes(AwA) [Lampert et al. CVPR 2009]: 40 auxiliary classes (24295 images), 10 target classes (6180 images); We use 5 instances per auxiliary class for learning;

ImageNet 2012/2010 [Deng et al. CVPR 2009]: 1000 auxiliary classes (from ImageNet 2012); 360 target classes (from ImageNet 2010). We use 3 instance per auxiliary class for learning; Recognition Tasks

Semantic Visual feature label space f(x)=fWx(x) space

No. Testing Classes No. Testing Words AwA/ImageNet Auxiliary Target Total Vocabulary Chance(%) SUPERVISED X 40/1000 40/1000 2.5/0.1 ZERO-SHOT X 10/360 10/360 10/0.28 OPEN-SET X X 50/1360 310K/310K 3.2E-04

The tasks are only separated in evaluation; We train one uniﬁed SS-Voc model for all the settings Recognition Tasks

Semantic Visual feature label space f(x)=fWx(x) space

The tasks are only separated in evaluation; We train one uniﬁed SS-Voc model for all the settings Recognition Tasks

Semantic Visual feature label space f(x)=fWx(x) space

The tasks are only separated in evaluation; We train one uniﬁed SS-Voc model for all the settings Baselines

SUPERVISED SVM: Input Features —> Semantic Labels LEARNING SVR-Map:Input Features —> Semantic Word Vectors

ZERO-SHOT DAP/IAP, DeViSE, ConSE, AMP, PST, HEX, TMV-BLP One-shotLEARNING Learning: Bart et al. CVPR 2005; Fei-Fei et al. TPAMI 2006; Mensink et al. ECCV 2012; Fu et al. TPAMI 2013;

OPEN-SET IMAGE SVR-Map RECOGNITION Baselines

SUPERVISED SVM: Input Features —> Semantic Labels LEARNING SVR-Map:Input Features —> Semantic Word Vectors

ZERO-SHOT DAP/IAP, DeViSE, ConSE, AMP, PST, HEX, TMV-BLP LEARNING

OPEN-SET DAP/IAP(LampertIMAGE et al. TPAMI 2013); DeViSE(FromeSVR-Map et al. NIPS 2013); ConSE(NorouziRECOGNITION et al. ICLR 2014); AMP(Fu et al. CVPR 2015), PST(Rohrbach et al. NIPS 2013). Baselines

SUPERVISED SVM: Input Features —> Semantic Labels LEARNING SVR-Map:Input Features —> Semantic Word Vectors

ZERO-SHOT DAP/IAP, DeViSE, ConSE, AMP, PST, HEX, TMV-BLP LEARNING

OPEN-SET IMAGE SVR-Map RECOGNITION

Schemer et al. TPAMI 2013,TPAMI 2014; Sattar et al. CVPR 2015; Bendale et al. CVPR 2015; Variants of Our Model

SS-Voc(W,V): 1 1 ] ✏(f( ,W), uairplaneV )+ [C ✏(f( ),uairplaneV)+ ✏(f( ),uhelicopterV) L 2L 2L X X Word-vector space global transformation Regression Term Max-margin Term Variants of Our Model

SS-Voc(W,V): 1 1 ] ✏(f( ,W), uairplaneV )+ [C ✏(f( ),uairplaneV)+ ✏(f( ),uhelicopterV) L 2L 2L X X Word-vector space global transformation Regression Term Max-margin Term

SS-Voc(W): 1 1 ] ✏(f( ,W), uairplaneV)+ [C ✏(f( ),uairplaneV)+ ✏(f( ),uhelicopterV) L 2L 2L X Regression Term X Max-margin Term Variants of Our Model

SS-Voc(W,V): 1 1 ] ✏(f( ,W), uairplaneV )+ [C ✏(f( ),uairplaneV)+ ✏(f( ),uhelicopterV) L 2L 2L X X Word-vector space global transformation Regression Term Max-margin Term

SS-Voc(W): 1 1 ] ✏(f( ,W), uairplaneV)+ [C ✏(f( ),uairplaneV)+ ✏(f( ),uhelicopterV) L 2L 2L X Regression Term X Max-margin Term

SVR-Map: ✏(f( W) u )V L airplane X Regression Term t-SNE Visualization of AwA 10 Testing Classes

SS-Voc: persian_cat, siamese_cat, hamster, weasel, rabbit, monkey, zebra, owl, anthropomorphized, cat SVR-Map: hamster, squirrel, rabbit, raccoon, kitten, siamese_cat, stuffed_toy, persian_cat, ladybug, puppy

150 persian cat hippopotamus leopard humpback whale 100 seal chimpanzee rat giant panda −20 pig 50 raccoon 0

20 0 40

60 −50

−100 100 −120 −100 −80 −60 −40 −20 0 20 40 60 80 −100 −80 −60 −40 −20 0 20 40 60 80 100 Semi-Supervised Vocabulary-informed Support Vector Regression (SVR) Learning (SS-Voc) Supervised Results

AwA dataset ImageNet dataset

Method Accuracy Method Accuracy