(12) Patent Application Publication (10) Pub. No.: US 2014/0378222 A1 Balakrishnan Et Al
Total Page:16
File Type:pdf, Size:1020Kb
US 20140378222A1 (19) United States (12) Patent Application Publication (10) Pub. No.: US 2014/0378222 A1 Balakrishnan et al. (43) Pub. Date: Dec. 25, 2014 (54) MOBILE VIRTUAL CINEMATOGRAPHY A63F 3/29 (2006.01) SYSTEM H04N5/9 (2006.01) (71) Applicant: Drexel University, Philadelphia, PA (52) U.S. Cl. (US) CPC ............... H04N5/2224 (2013.01); H04N 5/91 (2013.01); H04N5/77 (2013.01); A63F 13/04 (72) Inventors: Girish Balakrishnan, Santa Monica, CA (2013.01); A63F2300/1093 (2013.01) (US); Paul Diefenbach, Collingswood, USPC ............................................. 463/31: 386/224 NJ (US) (73)73) AssigneeAssi : PADREXEL (US) UNIVERSITY, rhiladelphia,Philadelphi s. ABSTRACT (21)21) Appl. NoNo.: 14/309,8339 A virtual cinematography system (SmartWCS) is disclosed, (22) Filed: Jun. 19, 2014 including a mobile tablet device, wherein the mobile tablet device includes a touch-sensor Screen, a first hand control, a Related U.S. Application Data second hand control, and a motion sensor. The SmartWCS (60) Provisional application No. 61/836,829, filed on Jun including a motion tracking camera configured to track the 19, 2013 sy-1-1 s movements of the motion sensor and generate motion track s ing information and a computer operatively coupled to the Publication Classification motion tracking camera configured to receive the motion tracking information and to transmit the motion tracking (51) Int. Cl. information to the first mobile tablet device, wherein the first H04N 5/222 (2006.01) mobile device is configured to overlay the motion tracking HO)4N 5/77 (2006.01) information onto a virtual landscape. OC Patent Application Publication Dec. 25, 2014 Sheet 1 of 14 US 2014/0378222 A1 Of 150 F.G. B Patent Application Publication Dec. 25, 2014 Sheet 2 of 14 US 2014/0378222 A1 2OO FG 2 Patent Application Publication Dec. 25, 2014 Sheet 3 of 14 US 2014/0378222 A1 3 O F.G. 3 Patent Application Publication Dec. 25, 2014 Sheet 4 of 14 US 2014/0378222 A1 oo: I 2.02 Tracking rack oying Patent Application Publication Dec. 25, 2014 Sheet 5 of 14 US 2014/0378222 A1 M-SO3 F.G. 5B Patent Application Publication Dec. 25, 2014 Sheet 6 of 14 US 2014/0378222 A1 Oa. 5p1 SO4. FG 5D Patent Application Publication Dec. 25, 2014 Sheet 7 of 14 US 2014/0378222 A1 -- SO3 Patent Application Publication Dec. 25, 2014 Sheet 8 of 14 US 2014/0378222 A1 N REA-OR) WRA-OR. (O w w w w w Y O Decease : Seisitivity o - ich foot increase sensitivity SSSSSS FG. 8 Patent Application Publication Dec. 25, 2014 Sheet 9 of 14 US 2014/0378222 A1 8 3S Of 35 50 Orr OO 2OOff 3OO Wedi in Wide Angie Norra elephoto elephoto saaaaaaaaaaaaaaaaaaaaaass saanaaaaaaaaaaaaaaaaaaaaas, Shorter Focal length longer Foca length T" 3COff FG 9 Patent Application Publication Dec. 25, 2014 Sheet 10 of 14 US 2014/0378222 A1 4:3 Standard Patent Application Publication Dec. 25, 2014 Sheet 11 of 14 US 2014/0378222 A1 Camera Saw & ( F.G. Patent Application Publication Dec. 25, 2014 Sheet 12 of 14 US 2014/0378222 A1 M Patent Application Publication Dec. 25, 2014 Sheet 13 of 14 US 2014/0378222 A1 K wear rer . San (...) s S. 5 . rer C 8. (f Patent Application Publication Dec. 25, 2014 Sheet 14 of 14 US 2014/0378222 A1 FG. 4. Reai-ite Carnerg-Position Wisdaizer PS Move co SCC itect F.G. 5 US 2014/0378222 A1 Dec. 25, 2014 MOBILE VIRTUAL CINEMATOGRAPHY 0007. The SmartVCS uses affordable hardware, an intui SYSTEM tive user interface, real-time benefits of game engines, and an intelligent camera system, to provides professional directors CROSS-REFERENCE TO RELATED as well as a new market of amateur filmmakers the ability to APPLICATION previsualize their films or game cinematics with familiar and 0001. This application claims the benefit of U.S. Provi accessible technology. The Support of both free-space move sional Application No. 61/836,829 filed Jun. 19, 2013, which ment and controller-based navigation with adjustable scene is incorporated by reference as if fully set forth herein. scales permits the user to navigate the virtual space and record camera motion in a variety of means. The user can addition FIELD OF INVENTION ally mark key frames for virtual dollies or booms, and control camera parameters such as focal length or aspect ratio. The 0002. The embodiments described herein are related to a SmartWCS can include cinematic principals for intelligent mobile virtual cinematography system and a method of pro generation of shots. For example, dolly tracks and timings are ducing films and game cinematics using commercially avail generated based on physically-guided principals in order to able equipment. produce realistic camera paths. SmartWCS also Supports rudi mentary scene editing, and has the ability to integrate into a BACKGROUND professional production pipeline Such as exporting camera 0003. With development of powerful computer processing paths or scene descriptions with industry Software packages equipment and graphic Support, virtual cinematography has such as the one sold under the trademark of Autodesk Maya. become a staple of the movie industry. Virtual content is 0008. In addition, the system has precision virtual camera assembled into a scene within a 3-D engine, (e.g. by filming controls mapped to analog joysticks, buttons, and Scroll live action elements or through computer generation). After wheel input. With the systems flexible design, operators may wards the scene can be composed, re-photographed, and modify the form factor of the device through shoulder, tripod, edited from other angles by a virtual camera as if the action dolly, and hand-held mounts was happening for the first time. The rendered image can 0009. A virtual cinematography system (VCS) is dis appear realistic. closed herein. The virtual cinematography system may com 0004 Virtual cinematography with a new virtual camera prise: a first mobile tablet device including a touch-sensor system has a wide field of applications and opportunities to be screen, a first hand control, a second hand control, and a useful, ranging at least from film industry, architecture/inte motion sensor, a motion tracking camera configured to track rior/landscape design studios, city planning offices, video and record movements of the motion sensor and generate game industries and medical field, for example. However, in motion tracking information to navigate a virtual camera; and the current production environment, the ability to previsual a computer operatively coupled to the motion tracking cam ize shots utilizing a virtual camera system requires expensive era that is configured to receive from the motion tracking hardware and large motion capture spaces only available to camera data of the motion tracking information and transmit large studios. There exists a need to develop a new virtual the data to the first mobile tablet device. In this system, the camera system that is more easily accessible in terms of the first mobile tablet device is configured to generate and over space, costs, and availability. lay virtual camera images based on the motion tracking infor mation data onto a virtual landscape and produce a virtual SUMMARY motion capture. 0005. A Smart Virtual Cinematography System (Smart 0010. In another aspect, a method of producing a virtual VCS) is designed for both amateur and professional users, motion capture using a SmartWCS is provided. The method who wish to embrace the notion of virtual production for may comprise the steps of pre-loading and launching a stan films, game cinematics, designs and medical images without dalone SmartWCS application into a mobile tablet device of a big studio budget. A user will be able to compose and record the SmartWCS device; tracking a movement of the motion camera motions in freespace or in fixed, steady cinema shoot sensor of the SmartWCS device via a motion tracking camera ing motions, and manipulate scene elements such as charac connected to a game console to produce motion information ters and environments through a touch interface or hand con data; transmitting the motion information data from the game trols. These interfaces permit previsualizing scenes and shot console to the mobile tablet device; operating a virtual camera compositions without the need for expensive hardware or within the SmartWCS application based upon the motion large motion capture Volumes. SmartWCS has potential appli information data So as to navigate the virtual camera in a cations to other areas including game level design, real-time virtual 3-D landscape according to the motion sensor, and compositing and post-production, and architectural visual recording a virtual motion capture including the virtual cam ization. era motion. 0006. The SmartWCS, a consumer based previsualization platform as described herein, provides a solution that was BRIEF DESCRIPTION OF THE DRAWINGS developed to make production of virtual cinematics more efficient. Being an open platform device, developers may 0011. A more detailed understanding may be had from the begin to build on top of the existing virtual production expe following description, given by way of example in conjunc rience, tailoring the Software to their production needs and tion with the accompanying drawings wherein: providing a system and method available for both profes 0012 FIG. 1 shows a traditional virtual camera capture in sional and amateur content makers including students, archi a motion capture studio (1A) in comparison with a SmartWCS tects, designers, city planners, and medical professionals to motion capture (1B); Small-production teams that cannot afford the cost of studio 0013 FIG. 2 shows an example SmartWCS system archi previsualization tools. tecture; US 2014/0378222 A1 Dec. 25, 2014 0014 FIG.3 shows an example hand control, motion sen includes strong graphical processing power to display com Sor, and camera; plex models, animations, lighting, etc.