(12) United States Patent (10) Patent No.: US 9,164,506 B1 Zang (45) Date of Patent: Oct
Total Page:16
File Type:pdf, Size:1020Kb
US009164,506B1 (12) United States Patent (10) Patent No.: US 9,164,506 B1 Zang (45) Date of Patent: Oct. 20, 2015 (54) SYSTEMS AND METHODS FORTARGET 2009/0187299 A1* 7/2009 Fregene et al. ................. TO1/23 TRACKING 2010, 0004802 A1 1/2010 Bodin et al. 2010/025.0022 A1* 9, 2010 Hines et al. ....................... 7O1/2 2011/O141287 A1* 6, 2011 Dunkel et al. ... 348,169 (71) Applicant: SZ, DJITECHNOLOGY Co., Ltd, 2011/0304737 A1* 12/2011 Evans et al. .... ... 348,169 Shenzhen (CN) 2012/0143808 A1* 6, 2012 Karins et al. .................... TO6/46 2012/0154579 A1* 6/2012 Hampapur et al. ........... 348/143 (72) Inventor: Bo Zang, Shenzhen (CN) 2012/0200703 A1 8, 2012 Nadir et al. 2012/0287274 A1* 11/2012 Bevirt ........................... 348,144 (73) Assignee: SZ, DJITECHNOLOGY CO.,LTD, 2012/0307042 A1* 12/2012 Lee et al. ...................... 348,114 Shenzhen (CN) (Continued) (*) Notice: Subject to any disclaimer, the term of this FOREIGN PATENT DOCUMENTS patent is extended or adjusted under 35 CN 102809969. A 12/2012 U.S.C. 154(b) by 0 days. CN 1031499.39 A 6, 2013 (21) Appl. No.: 14/471.954 WO WO 2010/089738 A2 8, 2010 OTHER PUBLICATIONS (22) Filed: Aug. 28, 2014 Rafi, et al. Autonomous target following by unmanned aerial Related U.S. Application Data vehicles. In Proceedings of the SPIE, May 2006. (63) Continuation of application No. (Continued) PCT/CN2014/0833 15, filed on Jul 30, 2014. Primary Examiner — Thomas G Black (51) Int. Cl. Assistant Examiner — Peter D Nolan G05D L/12 (2006.01) (74) Attorney, Agent, or Firm — Wilson Sonsini Goodrich & G05D I/00 (2006.01) Rosati (52) U.S. Cl. CPC ............ G05D I/0038 (2013.01); G05D I/0094 (57) ABSTRACT (2013.01); G05D 1/12 (2013.01) The present invention provides systems, methods, and (58) Field of Classification Search devices related to target tracking by UAVs. The UAV may be CPC ............................... G05D 1/0094; G05D 1/12 configured to receive target information from a control ter USPC ........ 701/2, 11: 348/113, 114, 144, 169, 170, minal related to a target to be tracked by an imaging device 348/171, 172, 211.99 coupled to the UAV. The target information may be used by See application file for complete search history. the UAV to automatically track the target so as to maintain predetermined position and/or size of the target within one or (56) References Cited more images captured by the imaging device. The control terminal may be configured to display images from the imag U.S. PATENT DOCUMENTS ing device as well as allowing user input related to the target 7.970,507 B2 * 6/2011 Fregene et al. ................. TO1/23 information. 2008/0054158 A1* 3/2008 Ariyur et al. ... ... 250,203.1 2009/0157233 A1* 6/2009 Kokkeby et al. .................. TO1/3 30 Claims, 18 Drawing Sheets -100 US 9,164,506 B1 Page 2 (56) References Cited OTHER PUBLICATIONS U.S. PATENT DOCUMENTS International search report and written opinion dated May 6, 2015 for PCT/CN2014/0833.15. 2013/0085643 A1* 4/2013 Mathews ........................ TO1/49 2013/017.6423 A1* 7, 2013 Rischmuller et al. ......... 348/114 2014/0049643 A1* 2/2014 Segerstrom et al. .......... 348,144 * cited by examiner U.S. Patent Oct. 20, 2015 Sheet 1 of 18 US 9,164,506 B1 1. 100 FIG. 1 U.S. Patent Oct. 20, 2015 Sheet 2 of 18 US 9,164,506 B1 1. 200 .204 HD O8 HD 210 FIG. 2 U.S. Patent Oct. 20, 2015 Sheet 3 of 18 US 9,164,506 B1 1. 300 FIG. 3 U.S. Patent Oct. 20, 2015 Sheet 4 of 18 US 9,164,506 B1 400 Obtain target information Identify target based on target information Detect deviation of target from predetermined position and/or size Generate Commands for UAV, carrier and/or imaging device to Substantially Correct the deviation 4.08 FIG. 4 U.S. Patent Oct. 20, 2015 Sheet 5 of 18 US 9,164,506 B1 FIG. 5 U.S. Patent Oct. 20, 2015 Sheet 6 of 18 US 9,164,506 B1 Y 606 Po (uo, Vo)6O4 U.S. Patent Oct. 20, 2015 Sheet 7 of 18 US 9,164,506 B1 1. 700 712 X708 FIG. 7 U.S. Patent Oct. 20, 2015 Sheet 8 of 18 US 9,164,506 B1 1. 800 Receive user navigation Commands and target information Control movable object according to navigation Commands Adjust movable object, carrier and/or imaging device to track target according to target information FIG. 8 U.S. Patent Oct. 20, 2015 Sheet 9 of 18 US 9,164,506 B1 1. 900 Receive tracking input Generate navigation commands based On navigation input Generate target information based on tracking input Provide navigation commands and target information 91O FIG. 9 U.S. Patent Oct. 20, 2015 Sheet 10 of 18 US 9,164,506 B1 ? 1000 Display images captured by movable object Receive user selection of a target Generate target information based on user Selection Provide target information to the movable object FIG. 10 U.S. Patent Oct. 20, 2015 Sheet 11 of 18 US 9,164,506 B1 - 1100 Receive images captured by UAV Receive tracking data Display images with tracking data FIG. 11 U.S. Patent Oct. 20, 2015 Sheet 12 of 18 US 9,164,506 B1 1. 1200 12O2 1210 12O6 1208 1205 1204 FIG. 12 U.S. Patent Oct. 20, 2015 Sheet 13 of 18 US 9,164,506 B1 1304 Oct. 20, 2015 Sheet 14 of 18 US 9,164,506 B1 U.S. Patent Oct. 20, 2015 Sheet 15 of 18 US 9,164,506 B1 1. 1500 1506 1506 15121 1 2 FIG. 15 U.S. Patent Oct. 20, 2015 Sheet 16 of 18 US 9,164,506 B1 1. 1600 FG 16 U.S. Patent Oct. 20, 2015 Sheet 17 of 18 US 9,164,506 B1 1. 1700 FIG. 17 U.S. Patent Oct. 20, 2015 Sheet 18 of 18 US 9,164,506 B1 -10 FIG. 18 US 9,164,506 B1 1. 2 SYSTEMIS AND METHODS FORTARGET along a flight path, and (2) target information of a target to be TRACKING tracked by an imaging device on the UAV; and one or more processors, individually or collectively, configured to track CROSS-REFERENCE the target according to the target information by automati cally adjusting at least one of the UAV or the imaging device This application is a continuation application of Interna while the UAV moves along the flight path according to the tional Application No. PCT/CN2014/083315, filed on Jul.30, one or more navigation commands from the remote user. 2014, the content of which is hereby incorporated by refer According to another aspect of the present invention, S ence in its entirety. system for controlling an unmanned aerial vehicle (UAV) is 10 provided. The system comprises: one or more receivers, indi BACKGROUND OF THE INVENTION vidually or collectively, configured to receive from a remote user (1) one or more navigation commands to move the UAV Aerial vehicles such as unmanned aerial vehicles (UAVs) along a flight path, and (2) target information of a target to be can be used for performing Surveillance, reconnaissance, and tracked by an imaging device on the UAV; and one or more exploration tasks for military and civilian applications. Such 15 processors, individually or collectively, configured to track aerial vehicles may carry a payload configured to perform a the target according to the target information by automati specific function Such as capturing images of Surrounding cally adjusting at least one of the UAV or the imaging device environment. while the UAV moves along the flight path according to the In some instances, it may be desirable for aerial vehicles to one or more navigation commands from the remote user. track a specific target. For Small-sized aerial vehicles. Such In some embodiments, the imaging device includes a cam tracking is traditionally achieved via control commands from era or a camcorder. a user-operated remote control terminal or device. Such In some embodiments, the one or more navigation com manual tracking control may become difficult in certain cir mands are adapted to control a speed, position or attitude or cumstances, such as when the movable object or target is the UAV. moving quickly or when the movable object is at least par 25 In some embodiments, the target is Substantially stationary tially blocked from view of the user. Furthermore, the atten relative to a reference object. tion necessary for Such manual tracking typically requires a In some embodiments, the target is moving relative to a dedicated user that controls a camera that onboard the aerial reference object. vehicle separate from a user that controls the navigation of the In some embodiments, the target information includes ini aerial vehicle, thereby increasing the cost for aerial photog 30 tial target information. raphy and other applications of the aerial vehicles. In some embodiments, the initial target information includes an initial position or an initial size of the target SUMMARY OF THE INVENTION within an image captured by the imaging device. In some embodiments, the target information includes tar In some instances, it may be desirable for aerial vehicles to 35 get type information. track a specific target. Thus, a need exists for improved UAV In some embodiments, tracking the target according to the tracking methods and systems that provides automatic or target information further includes identifying, based on the semi-automatic tracking of a target, thereby relieving opera target type information, the target to track from within one or tors of the aerial vehicles of manually tracking the targets.