( 12 ) United States Patent (10 ) Patent No.: US 10,719,969 B2 Scapel Et Al
Total Page:16
File Type:pdf, Size:1020Kb
USO10719969B2 ( 12 ) United States Patent (10 ) Patent No.: US 10,719,969 B2 Scapel et al. (45 ) Date of Patent : Jul. 21, 2020 (54 ) OPTIMIZED AVATAR ZONES (56 ) References Cited ( 71 ) Applicant: Apple Inc., Cupertino , CA (US ) U.S. PATENT DOCUMENTS (72 ) Inventors: Nicolas V. Scapel, Sunnyvale , CA (US ) ; 8,614,714 B1 * 12/2013 Koperwas G06T 13/20 Alexandre R. Moha, Los Altos , CA 345/473 (US ) ; Guillaume P. Barlier , San 9,786,084 B1 * 10/2017 Bhat G06T 7/246 Mateo , CA (US ) ; Thomas Goossens , 2017/0069124 A1 * 3/2017 Tong G06T 13/40 Paris (FR ) 2019/0122411 A1 * 4/2019 Sachs G06T 17/20 ( 73 ) Assignee : Apple Inc., Cupertino , CA (US ) FOREIGN PATENT DOCUMENTS WO ( * ) Notice: Subject to any disclaimer , the term of this 2016161553 10/2016 patent is extended or adjusted under 35 U.S.C. 154 ( b ) by 0 days . OTHER PUBLICATIONS International Patent Application No. PCT/ US2019 /033180 , “ Invi (21 ) Appl. No.: 16 /283,614 tation to Pay Additional Fees” , dated Jul . 24 , 2019 , 16 pages . International Patent Application No. PCT /US2019 /033180 , “ Inter ( 22 ) Filed : Feb. 22 , 2019 national Search Report and Written Opinion ” , dated Nov. 8 , 2019 , Prior Publication Data 22 pages. (65 ) (Continued ) US 2019/0371033 A1 Dec. 5 , 2019 Related U.S. Application Data Primary Examiner Yu Chen (60 ) Provisional application No.62 / 679,875 , filed on Jun . ( 74 ) Attorney , Agent, or Firm — Kilpatrick Townsend & 3 , 2018 Stockton LLP (51 ) Int. Ci. ( 57 ) ABSTRACT G06T 13/40 ( 2011.01) G06T 9/00 ( 2006.01) Various customization options are provided for customizing G06F 3/0484 (2013.01 ) a 3D avatar of a head . Features of the head and assets (52 ) U.S. Ci. corresponding to the features can be customized using blend CPC G06T 13/40 (2013.01 ) ; G06F 3/04845 shapes. An amount of storage for the plurality of blend ( 2013.01) ; GO6F 3/04847 ( 2013.01 ); G06T shapes is minimized by determining overlapping blend 9/001 ( 2013.01 ) shapes that can be reused for a plurality of different assets . (58 ) Field of Classification Search Further, techniques are provided for dynamic changes to an CPC G06T 13/40 ; G06T 9/001 ; GO6F 3/04845 ; avatar in accordance with selected features and assets . G06F 3/04847 See application file for complete search history. 17 Claims, 20 Drawing Sheets 180 10:09 2001 CANCEL DONE 174 EYES MOUTH ACCESSORIES EARRINGS ????? 171 HATS 172 GLASSES ( ) US 10,719,969 B2 Page 2 ( 56 ) References Cited OTHER PUBLICATIONS Ward et al. , “ A Survey on Hair Modeling : Styling , Simulation , and Rendering” , IEEE Transactions on Visualization and Computer Graphics , vol . 13 , No. 2 , 2007 , pp . 213-234 . U.S. Appl . No. 16 / 283,610 , “ Non -Final Office Action ” , dated Mar. 11 , 2020 , 16 pages. Orvalho, “ A System to Reuse Facial Rigs and Animations” , ECAG 2008 Workshop Facial and Bodily Expressions for Control and Adaptation of Games , 2008 , 3 pages. * cited by examiner U.S. Patent Jul. 21 , 2020 Sheet 1 of 20 US 10,719,969 B2 DONE EYES OC HAIR HEAD O10 00 co FIG.IB 10:09 CANCEL FACESHAPE 00 120 DONE (0) FIG.IA X 60:01 AVATARS 110 U.S. Patent Jul. 21 , 2020 Sheet 2 of 20 US 10,719,969 B2 DONE o OL MOUTHEYESHAIRHEAD FIG.ID 10:09 CANCEL TEXTURE STYLE 140 wahe ? DONE forma ? ( HEADHAIREYESMOUTH FIGIC. 10:09 CANCEL TEXTURE STYLE 2 130 U.S. Patent Jul. 21 , 2020 Sheet 3 of 20 US 10,719,969 B2 all DONE 22 () FIG.IF ACCESSORIESEYESMOUTH 10:09 CANCEL EARRINGS HATS GLASSES 160 ????? DONE 00 ACCESORIESMOUTHHAIRHEAD OD FIG.IE 10:09 CANCEL LIPS 00 150 151 152 U.S. Patent Jul. 21 , 2020 Sheet 4 of 20 US 10,719,969 B2 174 171 172 odl DONE EYESMOUTHACCESSORIES () IHFIG. 10:09 CANCEL EARRINGS HATS GLASSES 180 171 172 DONE 173 ]P () FIG.IG ACCESSORIESMOUTHEYES P 10:09 CANCEL EARRINGS HATS GLASSES 170 U.S. Patent Jul. 21, 2020 Sheet 5 of 20 US 10,719,969 B2 201 Mih 2B.FIG 210 201 FIG.2A 200 U.S. Patent Jul. 21 , 2020 Sheet 6 of 20 US 10,719,969 B2 261 FIG.2D 260 251 FIG.2C 250 U.S. Patent Jul. 21 , 2020 Sheet 7 of 20 US 10,719,969 B2 300 Feature_Eyes Asset Eyes BlinkShape 310 Asset Eyes Blink BlendShape A Asset Eyes Blink BlendShape B 311 Asset Eyes WinkShape 320 Asset_Eyes Wink BlendShape A Asset_Eyes Wink_BlendShape D 1 321 FIG . 3 U.S. Patent Jul. 21, 2020 Sheet 8 of 20 US 10,719,969 B2 400 410 IDENTIFY ASSET FOR RIG 420 OBTAIN 3D GEOMETRY FOR THE ASSET 430 COMPRESS THE 3D GEOMETRY 440 IDENTIFY BLEND SHAPES FOR THE ASSET STORE THE OFFSETS FOR THE BLEND SHAPES OF 450 THE ASSET FIG . 4 U.S. Patent Jul. 21 , 2020 Sheet 9 of 20 US 10,719,969 B2 500 510 STORE ASSETS AND BLEND SHAPES FOR FEATURES 520 RECEIVE A SELECTION OF A FIRST FEATURE IDENTIFY ASSETS CORRESPONDING 530 TO THE FIRST FEATURE 540 RECEIVE A SELECTION OF A FIRST ASSET IDENTIFY BLEND SHAPES CORRESPONDING TO THE 550 FIRST ASSET 560 RECEIVE A SELECTION OF A SECOND ASSET IDENTIFY BLEND SHAPES CORRESPONDING TO THE 570 SECOND ASSET FIG . 5 U.S. Patent Jul. 21 , 2020 Sheet 10 of 20 US 10,719,969 B2 600 Feature_Hair - Asset_Hairstyle Asset Hairstyle_Bob 610 Asset_Hairy Style_Bob_3DGeometry 7611 Asset_HairyStyle_Bob zone1 Asset_HairyStyle Bob zone2 Asset_HairyStyle Bob zone3 Asset HairyStyle Bob zone4 Asset HairyStyle Bob zone5 Asset_HairyStyle Bob zone 6 612 Asset_HairyStyle Bob zonen Asset Hairstyle_Mohawk 620 Asset_HairyStyle_Mohawk_3D Geometry -621 Asset HairyStyle_Mohawk_zone1 _Asset_HairyStyle_Mohawk_zone2 622 Asset_HairyStyle_Mohawk_zonen FIG . 6 U.S. Patent Jul. 21, 2020 Sheet 11 of 20 US 10,719,969 B2 Mobile Device 700 Power 742 System 703-8 744 Camera Computer- Readable 703-8 Medium 702 Sensors 746 722 Operating System 703-7 724 Communication Module GPS Unit 748 726 Location /Motion Module 728 Applications 703-6 730 Haptic Feedback Module External 736 732 Blend Shape InputModule Port 703-4 703-5 704 716 Controller Wireless 708 720 Circuitry 703-3 Peripherals 703-2 Interface 710 SPEAKER 703-25 712 Audio poh 718 Processor Circuitry a 714 703-1 MICROPHONE 706 VO Subsystem FIG . 7 U.S. Patent Jul. 21 , 2020 Sheet 12 of 20 US 10,719,969 B2 800 Blend Shape #8 SubsetforAsset#2 FIG.8 Asset #2 #9 Blend Shape #7 Blend Shape subsetOverlapping #6 Blend Shape Blend Shape #5 Blend Shape #4 2nddimensionin configurationspace Blend Shape #3 Blend Shape 1# Asset #1 SubsetforAsset#1 Blend Shape #2 space configuration in dimension gst U.S. Patent Jul. 21 , 2020 Sheet 13 of 20 US 10,719,969 B2 Blend Shape #3 Blend Shape #6 Blend Shape #9 Blend Shape 2# Blend Shape #5 Blend Shape #8 901 900 memory Blend Shape #1 Blend Shape #4 Blend Shape #7 Persistent FIG.9 1-6)shapes subset(blend 7,8,9)shapes 970 corresponding subset(blend 930 corresponding Retrieve corresponding subset 920 Retrieve corresponding subset 960 Working 903memory Working 903memory Blendshapes module902 Blendshapes 902module shapes)4-9 selection shapes1-6) selection subset(blend 980 correspondingsubset(blend 940 corresponding SelectAsset#1 Asset#2Select 910 Simulation module904 950 Simulation module904 U.S. Patent Jul. 21, 2020 Sheet 14 of 20 US 10,719,969 B2 1000 1010 RECEIVE SELECTION OF FIRST HAIRSTYLE DETERMINE FIRST SET OF ZONES ON WHICH FIRST 1020 HAIRSTYLE IS POSITIONED I 1030 IDENTIFY FIRST SET OF BLEND SHAPES FOR THE FIRST HAIRSTYLE 1040 RECEIVE FIRST MOVEMENT INFORMATION 1050 SIMULATE FIRST MOVEMENT OF HAIRSTYLE 1060 RECEIVE A SELECTION OF A SECOND HAIRSTYLE DETERMINE SECOND SET OF ZONES ON WHICH 1070 SECOND HAIRSTYLE IS POSITIONED IDENTIFY SECOND SET OF BLEND SHAPES FOR THE 1080 SECOND HAIRSTYLE 1090 RECEIVE SECOND MOVEMENT INFORMATION 1091 SIMULATE SECOND MOVEMENT OF HAIRSTYLE FIG . 10 U.S. Patent Jul. 21 , 2020 Sheet 15 of 20 US 10,719,969 B2 14 1140 Additiveresult FIG.11D 1130 variantLips FIG.11C 1120 Smile FIG.11B 1110 Defaultmouth FIG.11A U.S. Patent Jul. 21 , 2020 Sheet 16 of 20 US 10,719,969 B2 1250 PH deformationcorrected= 12EFIG. 1240 correctiveshape FIG.12D 1230 brokenresult FIG.12C 1220 Squinteye 12BFIG. 1210 Blinkeye FIG.12A U.S. Patent Jul. 21, 2020 Sheet 17 of 20 US 10,719,969 B2 1310 FIG.13B 1300 FIG.13A U.S. Patent Jul. 21 , 2020 Sheet 18 of 20 US 10,719,969 B2 1450 FIG.14E 1440 FIG.14D Samedynamicshapedirection,differentregion 1430 FIG.14C 1420 FIG.148 1410 Neutralshape FIG.14A U.S. Patent Jul. 21 , 2020 Sheet 19 of 20 US 10,719,969 B2 1500 1510 IDENTIFY ASSETS 1520 COMPOSITE TEXTURE FOR IDENTIFIED ASSETS 1530 STORE COMPOSITE IN BUFFER 1540 APPLY COMPOSITED TEXTURE TO MODEL FIG , 15 U.S. Patent Jul. 21 , 2020 Sheet 20 of 20 US 10,719,969 B2 1600 1610 STORE LOW RESOLUTION GEOMETRY 1620 PRECOMPUTE AND CACHE SUBDIVISION OPERATIONS 1630 GENERATE AND CACHE SHADERS 1640 RECEIVE HAIRSTYLE SELECTION 1650 GENERATE HIGH RESOLUTION GEOMETRY FIG . 16 US 10,719,969 B2 2 OPTIMIZED AVATAR ZONES identify a first set of the plurality of blend shapes that correspond to the first asset that is selected among the first CROSS REFERENCES TO RELATED set of the plurality of assets corresponding to the first APPLICATIONS feature . 5 The mobile device can also receive a selection of a second The present application claims the benefit of and priority asset among the first set of the plurality of assets corre to U.S. Provisional Application No.62 /679,875 , filed Jun . 3 , sponding to the first feature and can identify a second set of 2018 , the entire contents of which is incorporated herein by the plurality of blend shapes corresponding to the second reference for all purposes .