How Good Are Humans at Solving Captchas? a Large Scale Evaluation

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

Elie Bursztein, Steven Bethard, Celine Fabry, John Lab Security StanfordComputer Mitchell, Dan Jurafsky, E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation http://ly.tl/p11 ?

E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation http://ly.tl/p11 ? users

E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation http://ly.tl/p11 ? bots users

CAPTCHA Completely Automated Public Turing test to tell Computers and Humans Apart

E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation http://ly.tl/p11 E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation http://ly.tl/p11 E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation http://ly.tl/p11 E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation http://ly.tl/p11 E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation http://ly.tl/p11 E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation http://ly.tl/p11 E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation http://ly.tl/p11 E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation http://ly.tl/p11 E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation http://ly.tl/p11 E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation http://ly.tl/p11 E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation http://ly.tl/p11 E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation http://ly.tl/p11 E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation http://ly.tl/p11 E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation http://ly.tl/p11 E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation http://ly.tl/p11 E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation http://ly.tl/p11 E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation http://ly.tl/p11 E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation http://ly.tl/p11 E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation http://ly.tl/p11 E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation http://ly.tl/p11 E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation http://ly.tl/p11 E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation http://ly.tl/p11 E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation http://ly.tl/p11 93%

E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation http://ly.tl/p11 93%

86%

E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation http://ly.tl/p11 93%

86%

70%

E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation http://ly.tl/p11 8.2 sec 7.3 sec

9.3 sec

E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation http://ly.tl/p11 8.2 sec 10.6 sec 7.3 sec

9.3 sec

E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation http://ly.tl/p11 Outline

• Study methodology • Population demography • Captcha measures

E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation http://ly.tl/p11 Accuracy

Solving time

E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation http://ly.tl/p11 The supply chain

Websites Paper

E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation http://ly.tl/p11 The supply chain

Websites Paper

Scraping

E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation http://ly.tl/p11 The supply chain

Websites Paper

Scraping Solving

E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation http://ly.tl/p11 The supply chain

Websites Paper

Scraping Solving Data Mining

E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation http://ly.tl/p11 Scraping

• Alexa top 50 • 23 scheme • 10 000 captcha samples • Custom scraper • Cookies • Javascript events • Ip rate limiting • User agent

E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation http://ly.tl/p11 Scraping

• Alexa top 50 • 23 scheme • 10 000 captcha samples • Custom scraper • Cookies • Javascript events • Ip rate limiting • User agent

E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation http://ly.tl/p11 Authorize

Baidu

E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation http://ly.tl/p11 Authorize

Baidu

captcha.net

E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation http://ly.tl/p11 Authorize

Baidu

captcha.net

eBay

E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation http://ly.tl/p11 Authorize

Baidu

captcha.net

eBay

Digg

E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation http://ly.tl/p11 Authorize

Baidu

captcha.net

eBay

Digg

Google

E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation http://ly.tl/p11 Authorize

Baidu

captcha.net

eBay

Digg

Google

Blizzard

E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation http://ly.tl/p11 Authorize Yahoo

Baidu

captcha.net

eBay

Digg

Google

Blizzard

E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation http://ly.tl/p11 Authorize Yahoo

Baidu Microsoft

captcha.net

eBay

Digg

Google

Blizzard

E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation http://ly.tl/p11 Authorize Yahoo

Baidu Microsoft

captcha.net recaptcha

eBay

Digg

Google

Blizzard

E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation http://ly.tl/p11 Authorize Yahoo

Baidu Microsoft

captcha.net recaptcha

eBay Skyrock

Digg

Google

Blizzard

E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation http://ly.tl/p11 Authorize Yahoo

Baidu Microsoft

captcha.net recaptcha

eBay Skyrock

Digg Slashdot Google

Blizzard

E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation http://ly.tl/p11 Authorize Yahoo

Baidu Microsoft

captcha.net recaptcha

eBay Skyrock

Digg Slashdot Google mail.ru Blizzard

E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation http://ly.tl/p11 Authorize

Digg

E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation http://ly.tl/p11 Authorize

Digg

eBay

E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation http://ly.tl/p11 Authorize

Digg

eBay

Google

E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation http://ly.tl/p11 Authorize Microsoft

Digg

eBay

Google

E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation http://ly.tl/p11 Authorize Microsoft

Digg recaptcha

eBay

Google

E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation http://ly.tl/p11 Authorize Microsoft

Digg recaptcha

eBay Slashdot

Google

E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation http://ly.tl/p11 Authorize Microsoft

Digg recaptcha

eBay Slashdot

Google Yahoo

E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation http://ly.tl/p11 Precision is costly

1000

0.1% Precision accuracy

E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation http://ly.tl/p11 Precision is costly

1000 x 3

0.1% Precision Knowing the accuracy probable answer

E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation http://ly.tl/p11 Precision is costly

1000 x 3 3000

0.1% Precision Knowing the by scheme accuracy probable answer

E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation http://ly.tl/p11 Precision is costly

63000 captcha

E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation http://ly.tl/p11 Underground API

E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation http://ly.tl/p11 MTurk by Amazon

Worker(s) Requester (us)

E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation http://ly.tl/p11 Requester interface

E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation http://ly.tl/p11 Worker interface

E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation http://ly.tl/p11 Largest captcha experiment ever

• 8 audio schemes

E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation http://ly.tl/p11 Largest captcha experiment ever

• 8 audio schemes • 13 images schemes

E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation http://ly.tl/p11 Largest captcha experiment ever

• 8 audio schemes • 13 images schemes • 1000 x 3 image captchas / scheme (bypass-captcha)

E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation http://ly.tl/p11 Largest captcha experiment ever

• 8 audio schemes • 13 images schemes • 1000 x 3 image captchas / scheme (bypass-captcha) • 3500 x 3 audio captchas / scheme (MTurk)

E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation http://ly.tl/p11 Largest captcha experiment ever

• 8 audio schemes • 13 images schemes • 1000 x 3 image captchas / scheme (bypass-captcha) • 3500 x 3 audio captchas / scheme (MTurk) • 5000 x 3 image captchas / scheme (MTurk)

E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation http://ly.tl/p11 Largest captcha experiment ever

• 8 audio schemes • 13 images schemes • 1000 x 3 image captchas / scheme (bypass-captcha) • 3500 x 3 audio captchas / scheme (MTurk) • 5000 x 3 image captchas / scheme (MTurk) • 318 000 captchas annotated overall

E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation http://ly.tl/p11 Language repartition

Other 149 Russian 12 Balochi 13 Portuguese 15 Hebrew 15 Punjabi 15 Vietnamese 15 Bikol 16 Cebuano 17 Arabic 19 Macedonian 21 Dutch 21 French 23 German 28 Gujarati 30 Slovene 33 Marathi 39 Mandarin 51 Bengali 52 Kannada 64 Spanish 71 Romanian 95 Telugu 331 Hindi/Urdu 578 Malayalam 625 English 2791 Tamil 3502 0 500 1000 1500 2000 2500 3000 3500 4000 E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation http://ly.tl/p11 Repartition by education !"#$%&'%#'&()'**+,#-.#&/$0)*+,# &"#&"#

&'"#

!!"# $%"#

()*+,-./0# 123+#4*+..-# 5)06,/# 7.#8./9)-#,:;*)<.=# >+?@#

E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation http://ly.tl/p11 Age repartition

800 727 708

700 686 658 647

600

500 429

400 388 364 361 351 318

Numberusersof 300 268 239 230 205

200 182 177 137 133 132 113 106 104 103

100 91 76 70 63 58 56 46 42 38 37 36 35 26 26 26 21 18 16 15 11 10 6 2 2 2 2 1 0 1 18192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646667687172

E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation http://ly.tl/p11 Nb of user distinct answer for image scheme (bp)

1 answer 2 answer 3 answer 3.8934 Yahoo 25.615 70.492 3.8462 Blizzard 24.519 71.635 19.306 Slashdot 33.839 46.855 1.4778 Skyrock 18.966 79.557 21.729 Recaptcha 40.576 37.694 5.6893 Microsoft 25.821 68.49 37.44 Mail.ru 42.512 20.048 5.3333 Google 24 70.667 2.7311 eBay 16.597 80.672 5.2036 Digg 30.543 64.253 14.139 Captchas.net 42.008 43.852 4.3678 Baidu 21.839 73.793 0.69124 Authorize 13.825 85.484 0% 20% 40% 60% 80%

E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation http://ly.tl/p11 Nb of user distinct answer for image scheme (mk)

1 answer 5.2632 2 answer Yahoo 26.032 68.705 3 answer 0.87449 Blizzard 13.632 85.494 5.7506 Slashdot 26.132 68.117 0.89514 Skyrock 11.995 87.11 18.689 Recaptcha 38.565 42.747 13.119 Microsoft 33.633 53.248 24.388 Mail.ru 41.004 34.608 7.8974 Google 25.385 66.718 1.9013 eBay 15.827 82.271 2.4745 Digg 19.031 78.495 7.9385 captchas.net 32.113 59.949 2.7027 Baidu 16.319 80.978 0.23095 Authorize 6.5178 93.251

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation http://ly.tl/p11 Nb of user distinct answer for audio scheme

1 answer 29.197 2 answer Yahoo 36.439 3 answer 34.363

28.04 Slashdot 39.474 32.486

66.62 Recaptcha 25.512 7.8678

87.65 Microsoft 10.823 1.5264

95.403 Google 4.1875 0.40965

36.417 eBay 38.797 24.787

86.372 Digg 12.615 1.0129

41.655 Authorize 39.232 19.113

200 authorize 0 baidu 200 captchas.net 0 digg

100 ebay 0 google 200 mailru 0 mslive 200 recaptcha 0 skyrock 100 slashdot 0 blizzard 50 yahoo 0 100

0 100

0 200 0

100 0 100 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation http://ly.tl/p11 Audio solving time

50 Authorize Digg 0 eBay 50 Google Microsoft 0 Recaptcha

50 Slashdot Yahoo 0

0 50

0 3 4 5 6 7 8 9 10 11 1213141516171819202122232425262728293031323334353637383940

E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation http://ly.tl/p11 Accuracy by education

Image captcha Audio captcha

0.8

0.7

0.6

0.88 0.88 0.5 0.87 0.87 0.85

0.4 0.54 0.54 0.51 0.52 0.51 0.3

0.2

Not formal High School Bachelor Master Phd

E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation http://ly.tl/p11 Solving time by education

60 Solving time for image Solving time for audio 50

30 seconds

23.67 23.25 10 19.75 19.44 21.33

9.6 8.49 9.36 9.16 7.64 0 Not formal High School Bachelor Master Phd

E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation http://ly.tl/p11 Image captcha accuracy time by language

Native speakers Non-Native speakers 0.881 Yahoo 0.874 0.946 Blizzard 0.955 0.867 Slashdot 0.890 0.954 Skyrock 0.956 0.738 Recaptcha 0.772 0.804 Microsoft 0.794 0.700 Mail.ru 0.704 0.861 Google 0.873 0.935 eBay 0.935 0.919 Digg 0.925 0.837 captchas.net 0.848 0.927 Baidu 0.928 0.976 Authorize 0.979 0.70 0.75 0.80 0.85 0.90 0.95 1.00

E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation http://ly.tl/p11 Image captcha solving time by language

Native speakers Non-native speakers 11.1 Yahoo 9.4 9.7 Blizzard 8.3 8.3 Slashdot 6.2 8.3 Skyrock 7.0 12.8 Recaptcha 9.4 13.7 Microsoft 11.5 13.5 Mail.ru 11.1 10.4 Google 8.0 7.5 eBay 6.7 8.6 Digg 7.2 8.7 captchas.net 7.0 7.4 Baidu 6.3 7.1 Authorize 5.9

0 s 2 s 4 s 6 s 8 s 10 s 12 s 14 s

E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation http://ly.tl/p11 Audio captcha accuracy time by language

Native speakers Non-native speakers

0.67 Yahoo 0.71

0.65 Slashdot 0.73

0.45 Recaptcha 0.50

0.37 Microsoft 0.39

0.35 Google 0.35

0.62 eBay 0.65

0.37 Digg 0.39

0.57 Authorize 0.63

0.2 0.3 0.4 0.5 0.6 0.7

E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation http://ly.tl/p11 Audio captcha solving time by language

Native speakers Non-native speakers

27.27 Yahoo 25.00

18.39 Slashdot 11.70

30.63 Recaptcha 30.12

19.48 Microsoft 16.63

33.21 Google 35.20

14.49 eBay 11.84

18.89 Digg 14.83

15.41 Authorize 11.94

0 s 5 s 10 s 15 s 20 s 25 s 30 s 35 s E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation http://ly.tl/p11 Conclusion / ongoing work

• Captcha security rely on many different techniques • Not clear how to design them well • Ongoing work • What makes a captcha hard/slow for human ? • What makes a captcha secure ?

E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation http://ly.tl/p11 Features interactions

E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation http://ly.tl/p11