Etienne Bernard - Wolfram Research [email protected] Talk.Nb ��3 4 �� Talk.Nb

Etienne Bernard - Wolfram Research Etienneb@Wolfram.Com Talk.Nb ��3 4 �� Talk.Nb

2 �� Talk.nb

Automatic Machine Learning & Symbolic Neural Networks in the Wolfram Language Etienne Bernard - Wolfram Research [email protected] Talk.nb ��3 4 �� Talk.nb

Wolfram Language

◼ 1988: Mathematica

�� Solve[3*x+2⩵ 0, x] 2 �� x→-  3

�� DSolve[y'[x] == 10*y[x], y[x], x] 10 x �� y[x]→ⅇ 1

�� Plot3D[Sin[x]* y,{x, 1, 10},{y, 1, 2}]

��

◼ 5000+ functions ◼ http://reference.wolfram.com/language/ Talk.nb ��5

◼ Coherent ◼ High level

� BlurFacesimage_:= FoldblurSingleFace, image, FindFacesimage,"Position", "Image"

blurSingleFaceimage_, face_:= ImageComposeimage, Blurface["Image"], 50, face" Position 6 �� Talk.nb

�� BlurFaces 

��

◼ Knowledge based

��  �� Interval Year: 2012 , Day: Thu 14 Nov 2019   Talk.nb ��7

@  ��  �� Normal TimeSeries ��

 Tue 3 Jan 2012 00:00:00 GMT-8. , $24.54 ,

 Wed 4 Jan 2012 00:00:00 GMT-8. , $25.11 , Thu 5 Jan 2012 00:00:00 GMT-8. , $25.40 ,

⋯ 1975⋯ , Mon 11 Nov 2019 00:00:00 GMT-8. , $58.35 ,

��  Tue 12 Nov 2019 00:00:00 GMT-8. , $58.20 ,

 Wed 13 Nov 2019 00:00:00 GMT-8. , $57.89 

��

  ��  �� DateListPlot TimeSeries ��

�� 30

0 2012 2014 2016 2018 2020 8 �� Talk.nb

Classify

�� scrapeImages[string_]:= Thread[WebImageSearch[string, "Thumbnails", MaxItems→ 30]→ string]

�� classes={"Google logo", "Intel logo", "Apple logo", "Facebook Logo"};

�� images= scrapeImages/@ classes;

�� data= RandomSample[Join@@ images]; training= data[[ ;; 80]]; test= data[[81 ;;]];

�� training

�� logo= Classify[training]

 ��  �� ClassifierFunction ��

��

�� test[[1]]

�� → Facebook Logo

�� logo 

�� Facebook Logo

�� logo , "Probabilities"

�� Apple logo→ 0.0053377, Facebook Logo→ 0.991869, Google logo→ 0.0000165805, Intel logo→ 0.00277714

�� cm= ClassifierMeasurements[logo, test]

 ��  �� ClassifierMeasurementsObject ��

�� Talk.nb ��9

�� cm["Report"]

��

Number of test examples 40

Accuracy (88. ± 5.)%

Accuracy baseline (30. ± 7.)%

Geometric mean of probabilities 0.441 ± 0.16 Mean cross entropy 0.820 ± 0.36 Single evaluation time 95.1 ms/example

Batch evaluation speed 11.9 examples/s

Rejection rate 0%

�� pl logo Apple Logo Facebook logo Google logo Intel

Apple logo 4 0 2 0 6

Facebook Logo 0 11 0 1 12

culclass actual Google logo 0 0 10 0 10

Intel logo 0 0 2 10 12 4 11 14 11 predicted class

�� cm["Examples"→{"Apple logo", "Google logo"}]

��  → Apple logo, → Apple logo

�� cm["BestClassifiedExamples"] 10 �� Talk.nb

��  → Intel logo, → Google logo,

→ Google logo, → Google logo,

→ Facebook Logo, → Facebook Logo,

→ Facebook Logo, → Apple logo,

→ Intel logo, → Facebook Logo Talk.nb ��11

form= FormFunction[{"image"→ "Image"}, logo[#image, "TopProbabilities"]&]

image ��…

FormFunction  Submit

url= CloudDeploy[form, Permissions→ "Public"]

CloudObjecthttps://www.wolframcloud.com/obj/0246a5cd-08b7-47df-9524-d9d74055201b

URLShorten[url]

https://wolfr.am/I5j1y3V4 12 �� Talk.nb

Neural Networks framework

◼ High level ◼ No performance compromise Talk.nb ��13

A network from scratch

�� lenet= NetChain[{ ConvolutionLayer[20, 5], Ramp, PoolingLayer[2, 2], ConvolutionLayer[50, 5], Ramp, PoolingLayer[2, 2], FlattenLayer[], LinearLayer[500], Ramp, LinearLayer[10], SoftmaxLayer[] }, "Output"→ NetDecoder[{"Class", Range[0, 9]}], "Input"→ NetEncoder[{"Image",{28, 28}, "Grayscale"}] ]

 ��  �� NetChain uninitialized �� (�� × ��× ��) � �� (�� × ��× ��) � �� (�� × ��× ��) � �� (�� × ��× ��) � �� (�� ×�×�) � �� (�� ×�×�) � �� (�� ×�×�) � �� (�� ) � �� (�� ) � �� (�� ) �� (�� ) �� (�� ) �� {�� } �� {�� } �� {{�� }�{�� }} �� {�� } �� (�� × ��×�×�) �� (�� ) �� (�� × ��× ��) �� (�� ×�×�)

�� training= ResourceData["MNIST"]; test= ResourceData["MNIST", "TestData"]; 14 �� Talk.nb

�� RandomSample[training, 10]

��  → 2, → 1, → 7, → 3,

→ 2, → 7, → 6, → 3, → 3, →0

�� trainednet= NetTrain[lenet, training]

 ��  �� NetChain �� (�� × ��× ��) � �� (�� × ��× ��) � �� (�� × ��× ��) � �� (�� × ��× ��) � �� (�� ×�×�) � �� (�� ×�×�) � �� (�� ×�×�) � �� (�� ) � �� (�� ) � �� (�� ) �� (�� ) �� (�� ) �� {�� } �� {�� } �� {{�� }�{�� }} �� {�� } �� (�� × ��×�×�) �� (�� ) �� (�� × ��× ��) �� (�� ×�×�)

�� RandomSample[test, 10]

��  → 6, → 7, → 2, → 9,

→ 6, → 2, → 4, → 3, → 3, →0

�� trainednet , "Probabilities"

-9 -6 �� 0→ 5.3643× 10 , 1→ 1.40952× 10 , 2→ 0.999993, 3→ 4.97541× 10 -6, 4→ 1.29769× 10 -9, 5→ 3.08827× 10 -11, 6→ 9.05024× 10 -12, 7→ 6.28407× 10 -7, 8→ 2.40183× 10 -7, 9→ 2.87436× 10 -10 Talk.nb ��15

�� net6= NetTake[trainednet, 6]

 ��  �� NetChain �� (�� × ��× ��) � �� (�� × ��× ��) � �� (�� × ��× ��) � �� (�� × ��× ��) � �� (�� ×�×�) � �� (�� ×�×�) � �� (�� ×�×�) �� (�� ×�×�)

�� CurrentImage[]

��

�� Dynamic[Image/@ net6[CurrentImage[]]] 16 �� Talk.nb

Network repository

◼ https://resources.wolframcloud.com/NeuralNetRepository ◼ ~ 100 networks, growing

�� bert= NetModel["BERT Trained on BookCorpus and English Wikipedia Data"]

 ��  �� NetChain �� (��×�) �� (�� ) �� (�� ) �� (��× ��) �� (�� ) �� (��× ��) �� (��× ��) �� (��× ��) � �� (� ��) �� (��× ��) � �� (� ��) �� (��× ��) � �� (� ��) �� (��× ��) � �� (� ��) �� (��× ��) � �� (� ��) �� (��× ��) � �� (� ��) �� (��× ��) � �� (� ��) �� (��× ��) � �� (� ��) �� (��× ��) � �� (� ��) �� (��× ��) �� (� ��) �� (��× ��) �� (� ��) �� (��× ��) �� (� ��) �� (��× ��) �� (��× ��) �� (��× ��) � �� (� ��) �� (��× ��) � �� (� ��) �� (��× ��) �� (��× ��) ��

n ⨯ 768 n⨯ 768 n⨯ 768 + N Output 768 ⨯ add norm n

n⨯ 768n⨯ 768 D Input attention dropout �� 7

⨯

768

768 ⨯

⨯ n n n 768 6 ⨯

⨯ 64

768 ⨯ n n ⨯

768 64 768 768 ⨯ ⨯ ⨯ n 768 n n 768⨯ 5 n n ⨯ ⨯ 64 n

768 n⨯⨯ 768 n⨯ 64 n⨯ 768 n⨯ 768 n C M

Query 64 Output n768 4 ⨯ 13 14 ⨯ ⨯ n n 768

64 ⨯ n n⨯ 768 n ⨯ 768

Input n

n⨯ ⨯ 3 64 768 n 768 ⨯ n n ⨯ n ⨯ ⨯ 768 768 768 Talk.nb ��17

768 768 768 n ⨯ 768 64

n ⨯ ⨯ 2 n

n 768

n ⨯

⨯ 768

768

⨯ 12 n

⨯ 11 n

1 ��

n⨯ 768 n⨯ 64 M Query query elem n ⨯ 64

n⨯ 64 n⨯ 64 M A Output 768 64 ⨯ value⨯ attention n n

n⨯ 768 M Input key �� […] �� (��× ��) �� (��× ��) �� (��× ��) �� (��× ��) 18 �� Talk.nb

Surgery

◼ Surgery functions: ◼ NetExtract, NetRename, NetReplace, NetTake, NetJoin, NetAppend, NetDrop, NetInsert, ...

�� resnet= NetModel["Inception V3 Trained on ImageNet Competition Data"]

 ��  �� NetChain �� (�� × ��× ��) ��_�� (�� × ��× ��) ��_�� (�� × ��× ��) ��_�� (�� × ��× ��) ��_�_�� (�� × ��× ��) ��_�_�� (�� × ��× ��) ��_�_�� (�� × ��× ��) ��_�_�� (�� × ��× ��) ��_�_�� (�� × ��× ��) ��_�_�� (�� × ��× ��) �� (�� × ��× ��) ��_�_�� (�� × ��× ��) ��_�_�� (�� × ��× ��) ��_�_�� (�� × ��× ��) ��_�_�� (�� × ��× ��) ��_�_�� (�� × ��× ��) ��_�_�� (�� × ��× ��) �� (�� × ��× ��) �� (�� ) �� (�� × ��× ��) �� (�� ) �� (�� × ��× ��) �� (�� ) �� (�� × ��× ��) �� (�� ) �� (�� × ��× ��) �� (�� ) �� (�� × ��× ��) �� (�� ) �� (�� × ��× ��) �� (�� ) �� (�� × ��× ��) �� (�� ) �� (�� × ��× ��) �� (�� ) �� (�� ×�×�) �� (�� ) �� (�� ×�×�) �� (�� ) �� (�� ×�×�) ��_�� (�� ×�×�) �� (�� ) �� (�� ) �� (�� ) �� (�� ) �� Talk.nb ��19

�� extractor= NetTake[resnet, 30]

�� features= extractor[training[[All, 1]]];

�� head= NetChain[ <| "lin"→ LinearLayer[], "softmax"→ SoftmaxLayer[] |>, "Output"→ NetDecoder[{"Class", classes}] ]

�� uninitialized �� NetChain ��  ��

�� head= NetTrain[head, features→ training[[All, 2]]]

�� (�� ) �� NetChain ��  �� 20 �� Talk.nb

�� newnet= NetJoin[extractor, head]

test[[ ;; 3]]

 → Apple logo,

→ Apple logo, → Google logo

newnet 

Apple logo

newnet 

Google logo 22 �� Talk.nb

Some Applications...

ImageContents / ImageCases

�� ImageContents 

Image Concept BoundingBox Probability

domestic cat Rectangle[{38.2942, 0},{195.678, 197.941}] 0.880597

��

domestic dog Rectangle[{171.454, 2.39182},{459.438, 300.018}] 0.884554

FindTextualAnswer

�� FindTextualAnswer[ "The dodo(Raphus cucullatus) is an extinct flightless bird that was endemic to the island of Mauritius, east of Madagascar in the Indian Ocean. The dodo's closest genetic relative was the also extinct Rodrigues solitaire, the two forming the subfamily Raphinae of the family of pigeons and doves. The closest living relative of the dodo is the Nicobar pigeon. A white dodo was once thought to have existed on the nearby island of Réunion, but this is now thought to have been confusion based on the Réunion ibis and paintings of white dodos.\n Subfossil remains show the dodo was about 1 metre(3 ft 3 in) tall and may have weighed 10.6–17.5 kg(23–39 lb) in the wild. The dodo's appearance in life is evidenced only by drawings, paintings, and written accounts from the 17th century.", "How big was the dodo?"]

�� 1 metre

AudioIdentify Talk.nb ��23

TextCases

◼ General text recognizer ◼ 300 + entity types

�� planet= StringTake[WikipediaData["Planet"], 10 000];

�� planet

Entity

�� TextCases[planet, "Planet"]

�� {Neptunian, Earth, Earth, Earth, Mercury, Venus, Earth, Mars, Jupiter, Saturn, Uranus, Neptune, Jupiter, Earth, Earth, Earth, Earth, Earth, Earth, Earth, Earth, Venus, Venus, Venus, Mercury, Mars, Jupiter, Saturn, Earth, Venus, Earth, Earth, Earth, Earth, Mercury, Venus, Mars, Jupiter, Saturn, earth, Saturn, earth, Jupiter, Saturn, Mars, Mercury, earth, Venus}

�� TextCasesplanet, �� → "HighlightedSnippet"

�� {}

�� TextCasesplanet, �� → "HighlightedSnippet"

�� four terrestrials, Mercury, Venus, Earth, and

Mars , then the four giant planets, Jupiter, Saturn,,

planets. Venus, Mercury, and the outer planets Mars

, Jupiter, and Saturn were all identified by,

names): the Moon, Mercury, Venus, the Sun, Mars

, Jupiter, and Saturn.Cicero, in his De Natura,

Πυρ�εις(Pyroeis), which is called the planet of Mars

, and traverses the same round as the two planets

�� TextCases["Ratatouille is so flavorful. Piano is a bigger instrument that guitar.", "FoodAndDrinkTopic"]

�� {Ratatouille is so flavorful.} 24 �� Talk.nb

Topics, Languages, ...

�� TextCases["Ratatouille is so flavorful. Piano is a bigger instrument that guitar.", "FoodAndDrinkTopic"]

�� {Ratatouille is so flavorful.}

�� TextCases["What is this Language? Quelle est cette langue? It was english. C'était du français.", "French"]

�� {Quelle est cette langue?, C'était du français.}

Identification element

�� TextContents[ "Stephen Wolfram email: [email protected] Phones: 123-456-7890 100 Trade Center Dr, Champaign, IL 61820 https://resources.wolframcloud.com/NeuralNetRepository local host 127.0.0.1", {"Person", "EmailAddress", "IPAddress", "PhoneNumber", "URL", "ZIPCode"}, All]

String Type

Stephen Wolfram Person

[email protected] EmailAddress

123-456-7890 PhoneNumber �� 61820 ZIPCode

https://resources.wolframcloud.com/NeuralNetRepository URL

127.0.0.1 IPAddress Talk.nb ��25

Quantities & currency amounts

�� textVarietyStore= "The variety stores in the world: In Malaysia: RM 2 stores(2 ringgit) In Spain there are Todo a 100 shops(\"everything for 100 pesetas\" (€0.60)), although due to the introduction of the euro and inflation, most products cost a multiple of€0.60 or€1. Most of these shops maintain their name in pesetas, and most of them have been renamed as Casi todo a 100(\"almost everything for 100 [pesetas]\"), Todo a 100, 300, 500 y más(\"everything for 100, 300, 500 or more\") or Todo a un euro. Colloquially, the expression todo a 100 implies that something is either cheap, kitsch or low quality.In Portugal, there were Trezentos shops(300 escudos,€1.50) In Greece: 300(300 drachmas,€0.90) In Norway: Tier´n, a colloquialism for ten kroner, US$1.75. In Norway: Tier´n, a colloquialism for ten kroner, US$1.75. In Hungary, there are 100 forintos bolt stores In Portugal: Eupoupo- Tudo a€0,99 ou€1,49 In Sweden: Bubbeltian, a colloquialism for ten kronor, US$1.20. Another chain that has been spreading in Sweden during the last seven years is Dollarstore, where everything costs either 10, 20, 30, 40, 50 and steps of 50 up to 500 kr. In Russia: Fixprice(50 rubles) In Argentina, variety stores are called todo por dos pesos(2 pesos). In Brazil, these stores are called um e noventa e nove(one and ninety-nine, meaning BRL1.99, about US$1.20) usually written as 1,99.";

�� TextCasestextVarietyStore, "CurrencyAmount"→#String→ CurrencyConvert[#Interpretation, "USDollars"]&

�� RM 2→ $0.48 , 2 ringgit→ $0.48 ,€0.60→ $0.66 ,€0.60→ $0.66 ,€1→ $1.10 , 300 escudos→ $3.01 ,€1.50→ $1.65 ,€0.90→ $0.99 , ten kroner→ $1.09 , US$1.75→ $1.75 , ten kroner→ $1.09 , US$1.75→ $1.75 , 100 forintos→ $0.33 , €0,99→ $1.09 ,€1,49→ $1.64 , ten kronor→ $1.03 , US$1.20→ $1.20 , 500 kr→ $4.07 , 50 rubles→ $0.78 , 2 pesos→ $0.10 , BRL1.99→ $0.48 , US$1.20→ $1.20  26 �� Talk.nb

Grammatical units

�� TextCases[ "Los Angeles, officially the City of Los Angeles and often known by its initials L.A., is the most populous city in California, the second most populous city in the United States, after New York City, and the third most populous city in North America.",{"ProperNoun", "Adjective"}]

�� ProperNoun→{Los Angeles, Los Angeles, L.A., California, United States, New York City, North America}, Adjective→{populous, second, populous, third, populous}

Locations

�� locations= TextCases[planet, "Location"→{"String", "Interpretation"}, VerifyInterpretation→ True]

�� {{Greece, GeoPosition[{39., 22.}]}, {China, GeoPosition[{35., 105.}]},{Babylon, GeoPosition[{33.5959, 43.424}]}, {Babylonians, GeoPosition[{33.5959, 43.424}]}, {Babylonian, GeoPosition[{33.5959, 43.424}]}, {Babylonian, GeoPosition[{33.5959, 43.424}]}, {Assyrian, GeoPosition[{34.6444, 41.9578}]}, {Babylonian, GeoPosition[{33.5959, 43.424}]},{Greco, GeoPosition[{39., 22.}]}, {Roman, GeoPosition[{43.6966, 6.01513}]},{Greeks, GeoPosition[{39., 22.}]}, {Babylonians, GeoPosition[{33.5959, 43.424}]}, {Babylonians, GeoPosition[{33.5959, 43.424}]}, {Greeks, GeoPosition[{39., 22.}]},{Babylonians, GeoPosition[{33.5959, 43.424}]}, {Babylonians, GeoPosition[{33.5959, 43.424}]}, {Greeks, GeoPosition[{39., 22.}]},{Romans, GeoPosition[{44.1016, 8.43809}]}, {Greeks, GeoPosition[{39., 22.}]},{Greek, GeoPosition[{39., 22.}]}, {Greeks, GeoPosition[{39., 22.}]},{India, GeoPosition[{20., 77.}]}} Talk.nb ��27

�� GeoBubbleChart[Counts[Last/@ locations]]

�� 28 �� Talk.nb

Implementation

���� NetModel["ELMo Contextual Word Representations Trained on 1B Word Benchmark"]

���� NetGraph

E C M # M # M

M M SR SM

��_��

�� (�� × ��× ��) � �� (�� × ��× ��) � �� (�� × ��) �� (�� × ��) �� {�� } �� {�� } �� {{�� }�{�� }} �� {�� } �� (�� × ��×�×�) �� (�� ) �� (��× ��× ��) �� (��× ��× ��) Talk.nb ��29

���� NetModel["Wolfram Entity Recognition Net for TextCases WL 12.0 V2"]

���� NetGraph

E T M A M # M #

D M D M

�� (��× ��× ��) �� (��× ��)

Etienne Bernard - Wolfram Research [email protected] Talk.Nb ���3 4 ��� Talk.Nb

Etienne Bernard - Wolfram Research [email protected] Talk.Nb ��3 4 �� Talk.Nb