Santa Clara Law Santa Clara Law Digital Commons
Rosetta Stone v. Google (Joint Appendix) Research Projects and Empirical Data
3-18-2010 Vol. IX, Tab 47 - Ex. 34 - Deposition of Susan Wojcicki (Google Vice President - Product Management) Susan Wojcicki Google
Follow this and additional works at: http://digitalcommons.law.scu.edu/appendix Part of the Computer Law Commons, Intellectual Property Commons, and the Internet Law Commons
Automated Citation Wojcicki, Susan, "Vol. IX, Tab 47 - Ex. 34 - Deposition of Susan Wojcicki (Google Vice President - Product Management)" (2010). Rosetta Stone v. Google (Joint Appendix). Paper 105. http://digitalcommons.law.scu.edu/appendix/105
This Deposition is brought to you for free and open access by the Research Projects and Empirical Data at Santa Clara Law Digital Commons. It has been accepted for inclusion in Rosetta Stone v. Google (Joint Appendix) by an authorized administrator of Santa Clara Law Digital Commons. For more information, please contact sculawlibrarian@gmail.com. r
Cerlified Copy
iN THE UNI'IED,STAlES DISTRICT COl.iRT FOR nm'EAsTERNDispUCT OFVTRGIN!A ALEXANDl'JADMSION
RoSETTA STONE, LTD"
" j P~zjntijf. Cue No. 1:091'Y-CO 73 ~(GBIirCB) , GOOGLE, JNC.
: CONFIDENTIAL DEPOSITION'OF i SUSA.", WOJClCla
Mnrr.h J8, 2010 i 9:22 8.m. r.. . ':'" ( ")' S2S Universily Ayenue) Suib 900 1 ". . Palo Al~. Calif0!11ia iI I, .1.
--
Telephone: <-15.5912333
.~ Fa~c.: "l.S5iL.3335
(,.d Nontgornery St/tet ESQUIRE S!ri31.l00 •..... ,.'-w.."-- Sin Frands:D, CA 94104
i·
, ' ,
6343 .' .r.,• J. 1
2 , . TJiURSDAY: "t~~CH 18 r 2 010; 9:22.1LM. , , WOJCICKI I • SUSJu.~ 5 1 having been ' fir~c du~y sworn, testifies as £~~lows: < , 7
e ilL" MR. SEEK:
• Q Good morning . As I m~nt iO~ed ' e~rlier, my Lame 10 is Be.;!:;'nard Shek. I rl!pre3a:1t Rosetta Stolle.in this, U accion . . >2 Rav.e you 6 'uer he~n deposed betore' ~ 1.3 A Yes. 14 ,j . Q EOI" many .times? 15 :P. LeliS' than ten !:Iut more cl::!a... five . l1aybe seveb" 16 eight. ! don ' t remember exactly. 17 ,. Q You're aware t:lJ.at 'Coogle has hee.."l involve,d. in other litigation matters relatL~s to its advertiSin~ l' tradcma.rl: - ?Ol~cies. correct.? 20 A Other tban Ro~~tta Stene? '21 Q Right. 22 A I am aware of p;-evl.ous cases. but I I m not aware. 23 of other ongoing $~its . '24 Q ' ·Cases t::hat have been resolve-a io !=-he. past, ·25 right'?
' Ton Free; S()O.770.JJ63 . .. fecslmlO~! 415.591.3335 Suite 1100 ESQY1R~ .. ~ I>'.ont oomety Street 53t; frtr:dscrt. CA 9<1104 -,CS't~CluUons,co."J1
. ~ ~ '. " .:
!"-. -- . - , • . :, : " 7 ~. 6344 Sus~n "l-Toj cicft.i March 1.8, 2'010
;. 6
1 A Yes.
2 Q Did you ever te.s.tify in any ot those mat.ters·? :. '!> No, . I did not:.
Q i-ib.a.n...... ~s your last deposition?
·5 >. Las~ ..... eek.
6 Q So given .how recent your .last cieposit:ioD "'·2.S, ...:. i , you l re familiar t.he rules of a deposit.ion? j . 7 take it w:it~ i, . f, Ye.s . ' , 1 i Q Do you need ms to %0 o~er them at all today? i I 10 ,A No . ! • , i ' n Q Okay. I ; , :l2 ~. I mean unl.ess there's anythln.g ·specie.l about ~3 this case. I'assume itla like other ~epositions .
, ') Q Just likE it's en ~dinary deposition.
.h ¥ G·S.
Q . Can you give me your e~ccational .backgrO'l.Old 1.7 s·tarting from college ..
l:B A I have an undergradu ate degree f 'rom H"arVard
19. Un.i.'"C'ersity t hat! graduated from . in 1999· · - 1990. ,excuse
20 ro •• I 'have a. master 1 9 ill eccnomics from· UC Sant"a Cruz 21 and . I g raduated in 1993. and I nave an NEA from OCL.lt a!1d '
22 I graduatad in ~9~B .
23 o . Wha~ls YQur curreh~ position at, Googie? A' My current posi.tion · is -.rice president of
25 product managemeli~ reeponS~~le for our advert.isi:lg
.TotS fl1!e; "1100.770.3353 Fa~lTIne: 415".591.)335
S~lte Iloa 44 Mont;Ctner")' stnlet ESQUIRE San FJ"fIncisOJ, CA 9·UO'l •... A~"" .. O.P. I ...... , l\"\'(Y..~quir c:;o iutioru..i:om (
-:- _.. .. ~ : ' .. -...... 6345 '; .' - Susan Woj cicki l~arch .Hlr 2010 CO!\o""F!DENTIAL
:: -:J. 7 J. produpts.
2 Q Wben you 5a~ ady~r~isin9 products, do you mean
3 a2~ ~f Google ' s advertising products~
.4 5 Doub1eClick, znd our Arialytics produots.
Q ~~en dip you 'join Google? 1999.
B
9 A N~Y·· 10 Q Do -you know· when Google ' \oP-S founded'?
·11 A Google waa iounde~ in the ttl)· o~ 1998.
Q Do you know \-tbat emploYG.e number :YO".l ar.e?
.A. 16 .
~ .. Q What was 'your position when you. j a i ned . G?ogl.e i 1S 5...; Hay ~Q9 51?
16 ~ r wae tbe :mal:;keti:o.g m.enager·, so 1 was ' I 17 =esponsLble fQr all of Coogle ' s ~~rketirr9. So I was the ! J.S. fi:rst pe=so~ at Go?sle .that did ma.rket.ing, so i. "'orked l 19 ioitial~y on things' like our logo, our brand ide~tity, ; i· 20 and tryin~ to mar~et oUr . com~any with 'DQ budget.
21 Q Which is usually the cas~ with star~-ups. i2 And b.o\o.~ . long we.r·e you iIIarketing maoager?
23 A wel~, my title dfficially di~~'t . c?ange for a
24 coup1.e. of years .bece.u~e S300gle \"asn't that focused c.n
25 titles. but.I resily focused on ma=k~~ing ' probably for
TDII Fret!; BOO.770.J363 FdcshlJle: 415.591.3335
sUite 1100 .;4 tr.orlt9om'!ry Street San F~cl5tl;1, CA ~104 ESQ1IJ.Bb• ....ww:esquLt:esol~o~ . co.m
-.-"" .. .-
6346 Susan 'Woj ci d:i l<1arch 18, 2010
69
1 that comes after it. "\':e don't t:::y to f igu=e Oll t who
2 deserves the most blame . >..1so if something needs to be
3 doiLe, do it., don't say it'S" not dly job. "
4 I thL~~ ~he overall goal :0£ this statement was ' 5 ;:'0 say t.hat -- similar to anotbe= key principle we used
6 to have which is act ~ike . an dwner. Don't ,say: I Oh" this
7 isn't my job, ,you ,YJlOW, the~efore . you know, I'm -- you B know, I'm not going to pay attent:i.op': , '. The idea is that if' teu see something. you
10 think it should be fixed, you shoQ-ld go f·ix it. I f you
11 see the -- I mean, ~here was one example , this is a 12 small example, but I remember once we used to get. these 13 1?ott.le:s 'of water, and there wa.s like a -- some new .J 14 ~mpl?ye .e said, "lell. whose, jop is it to 'c!"..ange the ,.. ater 15 bottle? ~~d ~elre like, Look, we don't have' a person in
the' company whose title is water bottle c~ar.g~r. If you
17 think -- if you're thirsty, yo.~ go and you put the
18 bottle on the \-fater and you cha~ge it.
19 And so I t.hiIlk it:' 5 . that general' idea ~,hich is
20 that you should take responsibility for the things'
21 around you and not be expect other people to do ~t or
22 ~o bl~me oth~r peQple if something else isn't done the 23 "ray that you want. it:
24 Tum back to page 785_, 25 {Nitness compliE-s.)
Toli ('reI!: 800.710.3353 , Facsl;;JlI~ .t;15,SS1,33JS
Suite liDO ' 'i'; I",;;;rti:gomery Strcet ESQUIRE 'S:!IiI ~n~co. 0. 9 .. 104 , •' . ~ "" 'o&o1c;"u .to~ vIYV;\'.CHl:l!I~S:'Jlu tions,com
6347 , . , , ,~ ': -, ' ,S\!sa.'"\ 'Noj cicki March 18. 2010
- . : 70 1 Q NO"), the' first parag.raph. the last. sentence:
2 reads I "Our ·uS"e.rs ." Do you see that up at the top?
3 _!>... Yes.
4 Q It says •• Our' users trus~ Goo3lefs object~~ity
5 2nd no short-term gQin could ever justify breaching that
6 trust. If "1 A Yes .
a .. 9 Do y eu' agree with th~t st~tei:leo.t? 3 . .. Yes . I be-I ieve tbat t:.he way the compa...T'l.y is run 10 is by focusing on users becaus~ we are -- we need users
1..1 t~ come to our site to generat.e the revenue, and that
12 we -- ~sers come to Google because they crust Google,
13 and t.hat Google ope'rates 'a· 'business fer the long, term.
1 '!"4 'lie operate the b~61n1~ss -- ..... e look at the ecosy5b:m ./ 15 between users, advertisers. and p'.!blishers', 9.na W~ look
16 at hO~l ce..n we - - because sometimes use=s, advertis~rs,
17 an? pub1ishers h?ve -- or particul~rly ~ublishe~s and
18 =.dvertisers will have different deil'lC.nos, .and they may be
19 ~cntradictory . So we always think about ",'hat is the
20 long-term polution' and what's the right .way to manage
2l the company long term. 22 ]ind, you know, for eX2rople, no shott-term gain
23 could ever justify breaching that tr~st. You know, we
2. v.ould -7 I think what th~ylre tryi~9 to say herE is that 25 we're g.oing to rnaIlag: the business for the long term and
TeU rn~ 600.no.3363 .=aafrtl/le: ~H.59!"33JS
., Suitt l"cro ~.; !-'.:lntgom!/"Y Stneet S~n Fnnds!:c, CA 94JOC Vl'ww.!:SQ1Jlruolutions.om
6348 SUs an t'1oj cicki r1arch ~8, 2010 CONEIDENTL'IL
:.--:-, 1
~ that we want our users to trust and always believe that
2 the answers behind why ~'e did things· are reasonable.
Q You said that USErs come .to Google because they
4 trust Google. Users come to Google bec~use they· trust 5 that the result.s that Google gives thew. a.re going to b.e 6 relev.ant to whet it is thit theY've ·searched for, .right?
7 A They come to Google ~ - well, ! thirJK ~ne rea~on
.. ; a why they come to Google is because they believe that the
9 resul ts they get are going to b~ good. bu.t th.ey J.O believe -- I mean. t.here's also the \'lord "objE:cti·.... e"
i~ here, and "'.fhat that I S referring to, that ·! thi.n.k is i ,. import.ant to point out, is they believ~ the results ,·fill , I 12 13 be objective. They' believe that Google wili u.Ge· its ,i. !' ': .. ~'I best to its e.bility t: o ret:urn the ri·gbt" result at tbe J , .:J ( IS top. So, for e~:am!?le, because we have an ad partnership
1.6 vii ell somebody, we' re o.~t going' to chaDge the ranking.
17 Because we serVe search results fQr AOL, we're not:. going 18 to put AOL at the top.
19 We are -- .we go to e~treme lengths co be 20 objective in our search resulcs . . We don't mix 21 advertising in -- \-te don ' t do paid inclusion. wpicn I
22 think is iropor~ant to point out that ·that was a . practice
23 that was done early by many search e!!gin~s ...,her .e the· ads 24 were presented as search resu.lts ,. so GooSle bas' never 2S don-a paid inclusion; whereas, I would say probably most
Toll ~ree:: 800.7?O.3353 Facsl!"/'lil~: 4lS.591.:'I3JS
Suite HOI) <14 /o\ontgorruuy StP.::et ESQUIRE San francisco, CA 9410'1 •, ...... , ...... -> WW'f.' .e;;quires.,luti!J C'IS:.colTI
. .. :. .. : . . - 1.· ', 6349 SUSaIl 1floj cicki i1arch 18,. 201U
1.42 -
1 diff.'erent countries. And..we hav.e a whole team 'in 2 :f:inance th.at. tries' l;.;> gen!!!;ra'ce revenue report.5, but 'that
3. . te.am i ,s :Q.o;:: foc'used on . this .-pecific feature or I:.h.is
4 pol icy change.
5 BY MR. SHEK: , 6 , Q . ' Do you belie~,;e that the revenue report that yoa ! 7 received in Decernb~r ' or Jen~ary was prepaxed b~ Someone i, e other than a memhar -of this finance team that genEra ily ! , ; . 9 generates the revenu ~ rp.po~ts regarding ~dvert~sing I 10 r~Venue for Google? III i A It could have be~. sqmeone in sales. I just
12 don't remelTJJer. It wasn'./:. a person 't,hat.: I interact with I 13 regu-la.rly. I~ didn't stand out . This I!'.epor:t did not " .'\ stand O",::X. in a,ny way ~Jl~ th§!t r s why I don' t rem.ember'. J 15 Q . Did ·the DperGi.t:.:i~g Commi~t:ee . S,i.ve it.s app=ov~l I I Ior the trade~k po1icy change at the meeting where the . I ~ po1icy · change proposa l ~as preseDted? 'U P. I believe so. ! ,. . have. detailed. memories sf this ~e€tiDg, but usually !! ! 20 deci-slons. ' ~re ntade in a meeting _ .It' a very rare tha.t. we 21 decide to decide late·r.
22 o . To your )r...nowl·edge·, did Google cortduct any user 23 scud1.es ox e,:pe.rirnents in cOlllleccion with, the 2009 2i trademark policy change?
25 I am n~t aware of any_
, ~ Toll Fr~e: BOO.770.3lE:3 ~ FRSlmiie: "I1S.sgUDS Suilt uoo .;.; MOl1l;om~ Str-et ESQ1[lRE Sao mntG(.o. CJ; S''ilC'l "'ww.as:'iuj~!utlOIlS.Cb.TI
~. " ~ . .. " .... - , ' . . _' ... ' ...... , -,
6350 Harch 1-8, 2010
., ") 1<3
~ Q Ii' you" l:ook :Sack ~_t Exhibi't Karen 16. which is
2 . t.h~ "'arch 20'0.' e-mail · frompr.shimt l'uloria_ . So on
3 page :.. - on- the -second page, page 245, We-Ire Iookil1.g at .j < the e>;perimen.t H'o. 3 results . &. P'tiloria' l-!J:p~e that 5 npreliminary n~~erical results from user experiment
'5 ][0. 3· ''in~ic ate t.~t. confUsion r~rnains ):;. ~gh whe!l
7 trade~arR9 ' a=e' aI~owed in the ad body bue not in the ad 8 title. For 'a user; it . seems to Ita.'ce litt.1e d,ifference • ""oe'ClJer she/he s ees a tradema.rk in the ad ti tIe c:;:- ad . 10 hody - ~he likelihood of contusion =em~~ htgb." And
11 that was in Harch of 2004 in conn:,!'ct:i.on wi th the 2004.
>2 - trademark ' policy change.
13 'i.1hen the 2009 trademark policy challge was ( ;:.) 1.4 pr.esented to you, did someone te11 .,you that I:;he-:-e Wi:!..S ' no 15 lOnger -a 'bis-h likelihood' of confusion from the u;:;e of
16 trademark in the ad ~itle or ad text?
17 MS . CA1!.USO: . To th~ extent .this requires you, to 1S .disc lose commUllications with counsel , I instl:)Jct you not I. to answer the q".J.estion. If you. can .a.~wer the que·st.ion 20 'without disc~oairig such cpmmunications, 90 ~head . .
21 T!:.""E W:ITNESS; My underst:::anding from the team
22 ~Qs that we .had more inforr.~tion about ho~ eavertisers
23 were using t rademark ~ and ~hat ths , proposal that we had
2' was ill :re:~inement. a'nd I think it is m~re nuanced ·th;m 2S this experiment shaws '. and we had experience ",i tr.
ToO free.: &10,770.3363 flCS:m.ile: 415.591.3335"
" SultillOO -44 Mtl ntgom~r)" S'Jee! . Sal'l Frc: llcirrn, CA 9,,)04 ESQ.mM• vmw.~qlllres~Itr.:~S".:;om
, ,: . -"
6351 .. ~ :" , . .... ~U5an y7ojcic k± Miu:t:h 1.8,. 2010
144
1 en~ling.advertisera ·or no~ ~abling advertisers to use
2 tra.d.~ma.rk il.x;.d seeing w1;>.:lt the results ..... ae for the·
3 a6crs, ~ the=c were mult~pla ~mples as· to w~er6 ~ha
ad quali.ty was low· beca'.lse of riot enabling advert~sers
5 ~o name the specific p~oduccs ~hat th~y bad. s 1. maan,.· it' every br:rd to imagine how 7 advertisers in a ne"fspaper or a mrigazir.e wo:.116 list the 8 produces tbat: tbey: ca.rry if they can' e actually name
9 them . J...nd so· we fe~t similarly en the Web i and ther2
10 ",:e.re lots of e~tnpies shown o~ where adve~tisers were
11 trYiog t~ de9crib~ a product or using v~ry generic ~erms
12 li~~ leading. a 2eading b~aud. oz trying some other 13 sillY 6.crcllynl to describe it ·when they ju·st ....· ented to
'.";:).. --".j. comm~Dicate to the ~Ger that they're carry~ng this 15 product.
16 ~d I beLie~e that this was a nuance that waD
17 not considered in 2~04 because we were just qross~g tbe
l8 first b~idge of using t~ade~rks as k~yw~rd& for 4-9 arlvertise=s to serve on.
20 BY NR . SHB.K:
Q 22 .pre.seZlted to you, did anyone at C--ooglf! tell you that 23 there was no lons.e.r a high likelihood of co.p.£usion from
. 24 th~ use of· tredemarks :in the: ti!:le ·cr t exc of an ad?
25 MS . · CARUSO: Obje~t1on. · ~9ked a.nd answered a:-td
rc~ Pre~ aOO .77tl.335~ Faalrriile: 41S.S'31:D:3S -e StfiallOO ·H r'~on."'yDmU)· Street - ESQQIE~ Sill F~"ds::o. CA !? 6352 Susan 1'7ojcicki March If{, 2010 COl~IDENTrAL ,. 145 ,1 ,mi'Gcbaracterizes t.he ,record. 2 So I w«s not,awa~~' or thIs ,3 specific study" as I ,?rev ieusly testii,ied, a...' -4 presentaJ;.ion' r had was s\l££icieJ:.t to convince roe that: 5 , there was importan.t. \U;:er an?- advertiser va,luf:! by 6 eoa.b~ing this' trad~!'QS.rk , chcinge.· 'But' I 00 not ~emern1::ier ., ? specific referencE t. ~ this study, and , -- but that' ; ! '8 doesn't mean the ~am didnit do more work there. I' am 9 . just no t aware, arid that was not the focus of the or 10 mY ,disct!ssions .dth th"2m, My focus ~~ s on the problems tbat we vare 12 baving that:' ~dvert;:isers ,eculdn' t ~ist ~ nzmes of the 13 . J?=0dUc;:tB that they we,re carrying. , . : ~ :.) 1~ ~y MR, S~K.: 15· Q SO ' in co~nection with the proposed tr~aemark 16 policy cbange in :iDOS, is i 't t.rue that you did oat 17 di.ecuzti ....·itb. an~ne invohrec.. ~ , n the pol?-.cy change the 18 subject of use= confusion from the use of trademarks in 19 th,e, title or text Of aD ad? 20 !liS. c..UOSQ: Obj e.ct.l,on. 'Mi6cbaraccerizes the '21 re'cord. }.nd to t:.he ex.tent this calls 'f~r you to 22 disclose attorney-cl i ent, 9ommunications, I, instruct you 23 not to diEclose those _ You can snsw-e-:- again leaving ' 2< those: Bsi.de. 25 'r:.:rs WITl'~RS'S: ' I do not retnembei" ai.scussing I To~ :=~: 300.770;3353" Fac:;;h!.i!e: 'HS,591.3J3~ Suit! '1100 44 ( '~; '" ," 6353 Susan voIo] cicki . Narch 18, 2010 CONFIDE!iTIJi.L -: .... 154 :. .-~~) 1 there. was no action ~aken but the study was zun. 2 Q In the '2003 tir.l.e frame. were you a ine1t1ber of 3 the e-mail [email protected]? ~ A Probably_ I don't re-me.mber s.pecifically, but I 5 assume I w~s ~ Ne \-!ere a small team. 6. (EyJllbit 11 marked) 7 BY MR . SHZK, 8 Q You 've been handed a do"cument marke.d as Exhibit 9 i'iojcicki 11. A document that was produced by GoogJ.e, 10 Bates numb~red GOOG -P$-Q0043SS to 386. 11 Do you recognize this document? 12 A Yes . This is the one I "Jas referring to D earlier in tbe d?y. - I :) 1.ll Q If you turn to the second page, the chain 15 starts off ,·rith an e-mail from Orkut, I won't e:v-en 16 sttempt to pronounce his last name, and then ·there's an 17 e - m.ail from I"1arissa t·1ayer back to Orkut: is that 18 correct? 19 A Yes. 20 Q And th~se two e - mails are referring ~O an 21 experiment .:hat t ... as .conducted by· Orkut, correct? 22 F. Yes . 23 Q This is the . experiment that you ,.,ere testifying 24 abo:.lt earlier in the day? 25 · A .Yes. Toll Free: 80'1L 770.3353 F;!oc$imile; 4l.S.SH3335 Suite 1100 ( 44 r-lontgcmery 5:re..::t 5z:n Franci~co. CA 94104 w'Hw.esqlJiresc!utioil!.co:n 6354 Sllsan t'1ojcicki l-1arch lB , 2010 CONFIDENTIAL · " ~' 155 .. ...1 1 Q Prior to this experiment that was conducted in 2 2Q03, are you a::/(are of 'an.y user study. or experiment tha,t 3 Googl.e 'conducted regarding the use of the cert!l 4 n sponsored I,inks II ? 5 50 I am not aware. I didn't remember this 6 until I searched on the ter~ in my e-mail box. 7 Q In l-1s. l'1ayer' s ' e-mail she vlrites in the first S t~'10 sentences, nWe now have resul:ts on the paid 9 advertisements versus spon~ored link~ experiment. Paid 10 advertisements used as a label causes our a:d pe rformance 11 to decline,'" 12 Do you' see t:hat? l3 YBG. ( Q O;:her than what's contained in this e-mail, do 15 you have a recollection of '·the results of the user 16 experiment that sbe 1 s talking about? 17 So I do not have a recol'lection other than 18 what's in this a-mail. I was ac,tually surprised ,·;hen I 1.9 saw this j because I don't ,rememl?er. 20 One . thing I just want to point out for the 21 reccrd vlhich I found confusing about this e - mail '\t,~as 22 says,• . "While the decline is small , rarely :Larger than . 23 10 percent," wben you look at what Orkut v1rote he says .. 2'4 "The click-through from 4' .62 percent to 4:.5 percent.. " t 2S almost ,under if he. means a tenth of • percent rather To!! Fre-!; : 800.770.,3353 Fillcsimiie: 415.591.3335 Suit!:! llO~ 44 MontgOmery S t r~t San Fror.;:isco. CA 9410..; YI~IW.esquireso l utions.com 6355 . ; ' .. " '"",~.: .~. Susan t'loj cicki 11arch 18, 2010 CONFIDEN'rI1-L 155 1 than 10 percen~. 2 So I think if theylr~ going to do more -- If 3 you're going to look at this e-mail more carefully, I 'm ... just pointing out. that I'm tLTlclear from this e-mail \.... hat 5 he actually meant, because I don I t" see 10 percent. I 6 · see a tenth of a percent. And they don't -- I also note 7 .they don 't k:n.o;..,r if it. I S sta~i.5t.ically s.ignificant. She e says it's likely statistically significan~. but at this 9 point when' this e-mail issent.sh~doesnot~...no\<.· . 1 0 Q tolhat does stati.stica~ly significant mean to II you? 22 A Statistically significant means that, and I'm J.3 no~ a statistician, but it means that there is enough , ) J.4 data and the data -- t.here "s enough data that if you '15 ...Jere to replicate this again you would likely get the :l6 same r esult. And there are different confidence l evels 27 of sta~i~tically sig~ificant. 95 percent, 99 percent, 28 '!leaning if you ","ere to do t .his again, 95 percent or the 29 '. time you ..... ould get the same -- or you were 95 percent 20 sure that that answer \\·as correct. 21 So if something is not statistically 22 significant you c~n·t take the data to be ' meaningful, is 23 I think the highlight here. In order for a result to be 24 meaningful you need tha d~ta to be statistically 25 significant. Toll Fr~~: BOO.i10.3363 Facsimile: 415.591.3335 e Suite !lOD 44 Iolontgomery S,reet SM francisco, CA 94104 ESQQIB.~ w\vw.esquiresoiutions.tom . ~. 6356 Susan i·raj cicki March 18, 2010 COls7IDENTLiUJ l57 .. . j Q So. if - - if' .someone considered the data not tc 2 be statisticallY'insignificant L then the appropriate 3 course of action would be to conduct another experiment, 4 right? 5 t1S; CARUSO: Obj ection. T:.he ques.tion is very 6 confusing and it" s a hypothetical. 7 THE WITNESS: So rIve never heard ,of not 8 stat~stica~ly i~significant. 9 BY 11R. SHEK, ~o Q Oh, I' rn sorry . Consider tIle data to be 11 statistically insignificant. 12 J .. I mean, usually if data is not' statistically 13 significant, \Olhich is how it's referred to, then more ,i data is collected. 15 Q Can you turn to the first page of the document. 16 A Sure. 17 Q. In response to Ms. lJlayer' 5 e-mail there' s ~." 16 . e-mail from Mr . Rosenberg, right? 19 A Yes. 20 Q Mr. Ro~eIl.berg writes, nWhat I'd really. like to 2~ know is the impact on both the eTR and conversion rates . 22 If it turns out, as it s~Quld, that honestly labeling 23 ads causes fe\',er people tc click. on them, but that a 2~ cqmmensur~tely higher percentage of those people who did 25 do 'cli ck ~onve rt, everyone is better off i·f we label Toll Free;; 600.770-3363 ., Facsi:nlle: 41S.S31.3335 Suite· 1100 44 Montgomery Street ESQV.l.B.~ San Francisco, CA 94104 wW\'Y.esqui r eso!utiQns.~m . 6357 Susan Woj cicki I~arch 18, 2"010 CONFIDENrIll.L - , 158 ... ~ them a 5 such." 2 Did you, a2so receiv-e !-1r . Rosenbe!.'g f s e-mail ? 3 -'l. Yes. I had all the e-ma~ls in one thread . 4 Q Was it your understanding when. you receive'd 5 this that when Mr. Ro~e~~e~g 6 A Actually -- sorry. I don ' t remerr~er receiving 7 this in 2003. But go on. scrry, I didn't me~~ to 8 interrupt you. 9 Q "Readirig this e -.mail now, an. e-mail that y~u 10 received i~ 2003, Ln looking at Mr. Rosenberg's ~l reference to honestly labeling ads, is it your 12 understanding that !"'ir. Rosenberg is referring to 13 labeling ads as paid advertisements? 14 A I be.lieve that he is referring to sponsored -- 1.S this experiment ,,;here we 1 reo looking at sponsored links 16 versus paid adver tisement. But ae also says sponsored 17 links is, ~ote, honest. Q Nell , is, quote, pret ty hone.st. ~9 A Pretty honest, yes . ~The 1~ percent ~eduction 20 number. tells us t:hat," q:uote, IIsponsored links is pretty 21. honest, " in quotes, ~but the real question is do any of 22 our" -- "any or e -.... en a modest nuiilber of 10 percent of 23 use~s who were," quote, ".fcoled by our current label ' 7.4 convert?" 25 So I think Jonathan is asking a provocat.ive Toll Free: 600.770.3363 Facsimile: 415.!:Sl.3335 . SUIte 110e 44 HOf!tgamery Street San frandsco, CA. 94104 WI"I.w .eSauireso!utions .com 6358 Susan t'1oj ci~:ki March :18, 2010 159 i question, ~~d the reas~n he"is -- and he's us~g strong 2 terms. p~d the reason he is i~ because, as I mentioned 3 before,' we want to have dis~ussions at Google about \'That 4 we think the right thing to do is, and I think hers ,5 asking this in a provocative way. Marissa !:esponds , back 6 and she responds back saying that she believes tp~t t he 7 users understood and that they expressed preference fo~ -· 0 the term "sponsored links. II 9 JL.nd I do want to point out that at this· time 20 Harissa did not have renponsibility for any kind of II revenue. Marissa is a person 'at Google \o!ho is 22 -responsible 'for ' the users. Marissa is the pe.r~on ~vho 13 has responsibi:Li·ty for Google. com · for search and would .rI 14 see herself as the per60n at Google who is responsible 15 for users. ~ 6 So I don't remember seeing a response from 27 Jonathan. I th~nk, and you have the wbole thread too, 18 but I thin}.. that t.he threa.d ended at this point, and 19 that io/as wby there was no change that was made. '- 20 Q In Ms.· Mayer 1 s response to Mr. Rosenberg, I 21 think you reference this, she ~Ti~es, ~h~en asked 22 directly and indirectly iTI user studies what label they 23 preferred, the nearly universal feedback has been a 2~ preference for sponsored .links over anything that says 25 advertiseme~t.u ToJI FreE' : BOO.770.3363 Fa cs!m!l~: "15.591.3335 Swite 1100 4-<; l"lo:ltgomery Street ESOUIEE- San :"rcndsco, CA 9410~ ..... _" """"""'''''*''''~ _i wwr··esQulresolutions,com ,: .', . '.'.:. :" .. ',:', 6359 Susan Hojcicki Harch 18. 2010 CONFIDENTIAL 160 1 Do you see that? 2 A Yes. 3 Q Do you Knot., ' wh'at user studies she's referring 4. to? 5 A ' I do not. 6 Q And you don ' t know whether Google conducted any 7 'user studies relating to the use of the term "sponsored B li~sll prior to OrKut's experiment, correct.?, 9 I ·am not aware. I do knOtoJ' interr..allY. if you 10 look at old e-mails, that we talked about the category 11 that. ,,"e ,played in as sponsored links, and that I s why 12 AdSense's ea=lier name was called the Googl~ Spo~sored ~3 Link Program. \ So when we spoke t:o each o\:her. vIa didn't say, 15 How, many ads does a partner ,,/ant to have? We ....· Quld say. 16 How many sponsored links does the partDer want to have? 17 So I think there "jas some acceptance that this was the 18 'l:lay tha t these types of advertisements ~!ere labeled and ~~ called. 20 Q ...·lould: it ha'\"e concerned you if ~he user 21 studies:- if you knew at the time you received this 22 e-mail that the user studies that Ms. Mayer is talking 23 about refers to a single user stuay that was conducted 24 involving· lj individuals? 2S 119. CP.RUSO: Objection. Vague. Ton Free~ 800.770.3353 Fa~~"Ilile: ~lS.521.33]5 e SuIte 1100 44 r1ontgome:"'t' Street ESQ12IR.~ 5=;1 F~nds:;-o. CA 9UM \'IW\\':esquir~,ohst'lons,com 6360