US5495576.Pdf
Total Page:16
File Type:pdf, Size:1020Kb
||||||||| US00549.5576A United States Patent (19) 11 Patent Number: 5,495,576 Ritchey (45) Date of Patent: * Feb. 27, 1996 54) PANORAMIC MAGE BASED WIRTUAL Dennis Crawford REALITYFTELEPRESENCE AUDIO-VISUAL 57) ABSTRACT SYSTEMAND METHOD An improved panoramic image based virtual reality/telep 76) Inventor: Kurtis J. Ritchey, 26374 Tonganoxie resence audio-visual system and method includes panoramic Rd., Leavenworth, Kans. 66048 three-dimensional input devices, a computer processor, and a panoramic audio-visual output device. In one embodiment * Notice: The portion of the term of this patent of the system the input devices comprise a sensor assembly subsequent to Jul. 14, 2009, has been including a plurality of positionable radar, camera, and disclaimed. acoustical sensors for recording signatures of all sides of three-dimensional subjects simultaneously. The computer 21 Appl. No.: 2,582 processor integrates the sensor signals, processes signals as a virtual model, updates the model based on participant 22) Filed Jan. 11, 1993 interaction, and selects and distributes portions of the pro 51 Int. Cl. ...r. G06T 15/10 cessed virtual model for presentation on display units and 52) U.S. C. ......................... 395/125; 395/154 audio speakers. The processor includes participant interac 58) Field of Search ..................................... 395/119, 125, tive input devices for instantaneous interaction with the 395/130, 154, 50,902, 2.55, 2.6, 2.79, virtual model. The panoramic audio-visual output device includes a head-mounted display or a closed structure hav 2.85; 34.5/139, 158, 184 ing contiguous individual display units mounted in all References Cited viewable directions surrounding the participant. Conven 56) tional, stereoscopic, autostereoscopic, and holographic dis U.S. PATENT DOCUMENTS play systems are provided to view the panoramic three 4,868,682 9/1989 Shimizu et al. ........................ 358/335 dimensional image based model. Computer graphics, 4,951,040 8/1990 McNeil et al. .......................... 340,729 artificial intelligence, telecommunications, and vehicle con trol embodiments of the system are provided. Primary Examiner-Mark K. Zimmerman Attorney, Agent, or Firm-Litman, McMahon, & Brown; 26 Claims, 13 Drawing Sheets PANORAMC PANORAMC l AUDIO SIGNAL VISUAL SIGNAL ar PROCESSING PROCESSENG 2 3 - 1 - - - - - 5:wn PAN.NPUT 8 SOURCE SC FUSON OF SENSOR ST SIGNALS TO FORM Eis MODEL SEGMENTS 15 POSITION NG 8 FUSON OF MOOEL as ÉEc SiGNALS6So 228 SEGMENTS MEANS O SUB PAN. JECTS IN PAN. AUDO THE - SIG. PROC, MODEL GRAPH PROC. MEANS CS MEANS VISUAL MASS SIMULATION - STOR. OF PAN. (VIS) ION OF PAN. COMPUT. GENERAT EDMODE IMAGE PROC. SFORDS. 8 DISPLAY PARTICIPANT NTER. CONTROL OF SSEM NTERACTIVE 22 O INPUT DEVICE SCM APAN.AN SEA gi CONVAUDVIS, PANORAMC ASSEM. STER. EastEENSMODEL w SEAUTO t; 2 (AUDIO 8 VISUAL) 3, Holol U.S. Patent Feb. 27, 1996 Sheet 1 of 13 5,495,576 PANORAMC PANORAMIC 1. AUDIO SIGNAL VISUAL SIGNAL re PROCESSING PROCESSING 2 V N 3-DSSE AUDIO 7&SEM&E3-D SHAPE6 3-D IMAGE INPUTPAN. 31 INPUT INPUT INPUT SOURCE - 5 NTHSat NTH s NTH FUSION OF SENSOR ST SIGNALS TO FORM PROC. MODEL SEGMENTS MEANS /5 POSITION 5 : ING 8 2Gh 2ND 6SSo of 53. 266 E.S SSSIs 228, 2O, SEGMENTS /6 DEV. s 2. AUDIOEis MODELTHE A.K O 5 --SESGRAPH MEANSPROC 5 5 VISUAL 3, SIMULATION 3RD () SIMULAF OF PAN. PROC, TIONS MEANS COM5. E. /7 GENERAT ED MODEL 5 4TH s' FRSMEANS DISPLAY 18 CONTROL OF INTERACTIVE INPUT DEVICE sc span. BS37 CONV. ASSES SEEastEREENS MODE AYgle (AUDIO 8 M : VISUAL) 3.es U.S. Patent Feb. 27, 1996 Sheet 2 of 13 5,495,576 OVERLAPPNG FIELD-OF-REGARD COVERAGE N SMC? 24 - RADAR SAMERez N 7 N 1. HONY So- 85 86-8 NiaSsay V VAN NS VY ANN NW2/Z, Les AAYAAAs see AYAs a UDIO f AY39P PROC. 561 ROC. Fig.5. U.S. Patent Feb. 27, 1996 Sheet 3 of 13 5,495,576 54, 5b, 5C U.S. Patent Feb. 27, 1996 Sheet 4 of 13 5,495,576 6. .. U.S. Patent Feb. 27, 1996 Sheet S of 13 5,495,576 U.S. Patent Feb. 27, 1996 Sheet 6 of 13 5,495,576 IMAGE - M 4 3-D COMPUTER GENERATED MODEL |-a-b 1 ty / WIREFRAME AUDIO SIG. RECORDING U.S. Patent Feb. 27, 1996 Sheet 9 of 13 5,495,576 U.S. Patent Feb. 27, 1996 Sheet 10 of 13 5,495,576 3-D 3-D l,71 INPUT SOURCE SIGNAL S-D 12 MEANS PROCESSING MEANS DISPLAY MEANS e - - 7WTH23NTH - 3 - - - 4OW7h 14a,25d. ATA coMERHOST PR DiSEAY as 5ANTH PAN, NTH 3-D IMAGE MASSDATA BASESTOR, e fa,234)A7–SksPKNTH splay |b,25b coisHOST TER is D'A'.DISPLA 3-5DioEAIAEASE T "E" KSPK. A 4-35a MASS STOR, 82 q.7/SP FitTMING opT as255NS9. 3. |CONB9(CLOCK) ER 5s - PROCESSINGUNIT File:F----\a ?o,78 88 Fig.16.O 6 4c - 4c4 32,33\ fi PSEN223901e/-SY A-8 2 ea2al ASSEMBLY"le 12212 21-4 4-2% Oa U.S. Patent Feb. 27, 1996 Sheet 11 of 13 5,495,576 INPUT SOURCE i49. 20. la 7 O 3-D AUD.-VIS. INTER. l,20se PANORAMCDIGITIZER ASSEMBLMEANS Y NPUT OPERATORVIEWER/\ ^ SYSTEM 11, 12 SYSTEM -2 6 92 43 To PANöRAMC3-D ESSE ENSSPEE/Encrypt. LOCATION gER systEMS COMPRESS. 9 )78 3-D PANORAMC DECODER/ AUDO EXPANDER FROM SYSTEM LOCATION EECOMM.h.2Od UNIT A LOCATION B LOCATION A U.S. Patent Feb. 27, 1996 Sheet 12 of 13 5,495,576 fl. 2 DISPLAY WIEWING MEANS SPACE t tent 3 LADAR5. O 2 PROC rod DELAYSE SIDE CAM. f. A 150 A SPEAKER MIC, or AXE 23d 35. A Sar Tib 70 SIDE N &Nyke VSS.S U.S. Patent Feb. 27, 1996 Sheet 13 of 13 5,495,576 l2. DISPLAYSPHEBIACQYSRAGE ASSEMBLY MEANS O1a TRANS- G CEIVERSRS 255RASOPERATOR SIGNAL PROCESS.4 46 UNIT REMOTELY PLOTED VEHICLE 3-D PAN. IMAGING, SHAPE 8 AUDIOSYS. 5,495,576 1. 2 PANORAMIC MAGE BASED WIRTUAL In this manner action of a 3-D subject is recorded from REALITYITELEPRESENCE AUDO-VISUAL substantially all aspects at a single moment in time. The SYSTEMAND METHOD participant operates interactive input devices associated with the computer to manipulate the virtual object. In one example, the participant observes the model on a head BACKGROUND OF THE INVENTION mounted display system. In another example, the participant is surrounded by contiguous audio-visual display units. In 1. Field of the Invention the latter example, each display unit displays a segment of This invention relates generally to panoramic display the model. methods and more particularly to the sensor fusion of data O It is therefore the objective of this invention to provide a from the panoramic arrangement of three-dimensional imag more versatile image based panoramic virtual reality and ing sensors and surface contour sensors to form virtual telepresence system and method. Still another objective is to objects and scenes, the processing of the virtual objects and produce systems and methods for recording, formatting, scenes based on a viewer operating interactive computer processing, displaying, and interacting with data represent input devices to affect the manipulation of the virtual objects 15 ing 3-D beings, objects, and scenes. More specifically, an and scenes defined in the computer, and the display of the objective of this invention is to provide a positionable affected virtual objects and scenes on a panoramic display multi-lens camera system for recording contiguous image unit to the extent that the viewer perceives that the virtual segments of an object, being, adjacent surrounding scene, or objects and scenes completely surround the viewer. any combination of these types of subjects; a signal pro 2. Description of the Related Art 20 cessing means comprising first computerized fusion pro My previous U.S. Pat. No. 5,130,794 describes a pan cessing system for integrating the positional camera system oramic image based virtual reality system that incorporates with corresponding digitized shape and contour data, a a multi-lens camera system with spherical field-of-view second computerized fusion processing system for integrat (FOV) coverage. As shown in FIG. 2, objective lenses of the ing first fused data with other fused data representing '794 camera system face outward with adjacent or overlap 25 adjacent portions of a being, object, or scene comprising a ping FOV coverage. The imagery from the camera is surface panoramic computer generated model; where various 3-D mapped onto the interior of a three-dimensional(3-D) shape digitizer systems may be incorporated for entering 3-D defined in a special effects processor of a computer. Alter shape and contour data into a image processing computer; a natively, the input source is at least one computer graphics third processing means to manipulate the geometry of sub system that generates three-dimensional graphics of spheri 30 jects comprising the virtual model; a forth processing means cal FOV coverage. The viewer operates interactive input for sampling out given fields of regard of the virtual model devices associated with the computer to manipulate the for presentation and distribution to display units and audio texture mapped virtual images. The virtual environment is speakers; where signal processing means includes an expert instantaneously affected before the viewer and displayed on system for determining the actions of subjects of the com either a head-mounted display assembly or on contiguous 35 puter generated model; where the signal processing means display units positioned beneath, to the sides, and above the includes image segment circuit means for distributing, pro viewer. cessing, and display of the model; where the system includes a 3-D graphics computer system for the generation, alter Limitations of the panoramic video camera system in 794 ation, and display images; and a system and method for are that the panoramic camera does not record a non 40 image based recording of 3-D data which may be processed spherical field of view(FOV) and does not incorporate a for display on various 3-D display systems to include head non-contact shape sensor.