A Cost-effective Multiple Camera Vision System using FireWire Cameras and Softwarehelp in recreating Synchronization 3D models and applications such as Virtual Reality, which Piyush Kumar Rai Kamal Tiwari Prithwijit Guha Amitabha Mukerjee need mixing of humans with artificial data. Undergraduate Student Undergraduate Student PhD. Student Professor Multiple camera systems such as AVIARY Computer Science and Computer Science & andElectrical Easy-Living Engg., (by ComMicrosoftputer ScienceResearch) and Engg., Engg., wereIIT Kanpur, among the first of Engg.,this kind of systems. IT-BHU, Varanasi-5, IIT Kanpur, However,Kanpur, UP, problem India.s aroseIIT with Kanpur, these systems UP, India. Kanpur, UP, India. [email protected] to reasons such Kanpur,as high-cost UP, India. (costly rai_piy ush@y ahoo.co.in kama [email protected] cameras and [email protected] cards), lack of mobility and complicated hardware Abstract: synchronization mechanisms (which hinder In this paper, we describe a cost-effective the mobility of such systems). Multiple-Camera Vision system using low cost Hence, in our approach, we are using simple simple FireWire web cameras. The FireWire and low-cost FireWire web cameras for cameras, like other FireWire devices operate on image-acquisition, which operate on the high speed FireWire bus. Current supported bandwidth is 400 Mbps. Right from its FireWire (IEEE 1394). The IEEE 1394 introduction, the FireWire (synonymously known specification offers a bandwidth of 400 as IEEE 1394) bus interface specification has Mbps, which is suitable for such systems proved its capabilities and has been supported dealing with large amounts of data transfer. by both developers and users. Due to its low cost These cameras do not have the capability of and ease in connecting, FireWire has been hardware synchronization as in case of high- recommended as the technology to be used in cost “pan and tilt” cameras, so we propose a Machine-Vision systems and Image-processing software based synchronization mechanism applications. We have developed a Multiple- for such a system using a server-client camera synchronized Vision system using model. FireWire cameras. The synchronization has been achieved using “Software Triggering” which has been discussed in the paper. Possible II. Features of the system: applications of such a system have also been discussed in the paper. Our system does away The system offers following advantages with the need of costly cameras and frame- over the existing systems: grabber cards which are generally used in conventional multiple-camera systems. The Cost effectiveness: The FireWire web direct pixel-to-data correspondence without the cameras used in the setup are much cheaper need of a frame grabber or classic synchronization systems (requiring Hardware than the traditional high-cost pan and tilt synchronization mechanisms) justifies the cameras and frame-grabber cards. novelty of such a system. Calibration of the multiple camera system has also been discussed. Scalability: Any number of cameras can be added to such a system (of course, limited I .Introduction: by the capability of communication channel, Vision is seen as the primary input through which the individual cameras for robot applications that need to be communicate to the server PC). Also, any performed in short time and efficiently. number of cameras can be removed from the Keeping this in mind, vision researchers system without affecting the system. This around the world have been working on makes the system entirely reconfigurable. developing efficient algorithms and vision systems that can accomplish this task. As for Software Synchronization: Provides a vision systems are concerned, several synchronized capture capability, which is panoramic and stereo vision systems have very crucial for such multiple camera been developed in the past. Many multiple- systems. camera systems have been developed which Setup and Hardware: The server triggers the clients The setup has a 4 meter cubical covered simultaneously but whether they get with green curtains from three sides to triggered at the same time or not, depends minimize the noise due to light intensity upon several factors. We shall discuss them fluctuations. Fourth side of the cubical is a one by one: calibrated white display screen controlled by Network Latency: The server can send a a projector which is placed outside the multicast message to all the client. However, cubical. An integral part of this setup is a set the difference of time in getting this of three firewire cameras mounted on message by first and last client will depend strategic position across this cube so as to upon the current load of the network. In an cover the entire workspace within. The unloaded network, this latency may be very camera calibration is through the technique small (~ 1ms), but for heavily loaded of SELF CALIBRATION which will network, these delays may be indeterminist accomplish the task of calibration in bits of and significant. seconds. This setup will help an accurate background subtraction and silhouette Scheduling in Linux: The Linux kernel is estimation. The 3 FireWire cameras non-preemptive. So, it is the Linux kernel mounted on 3 different machines running on which dictates how fast the clients can Linux act as client PCs and send the respond to the trigger signal by the server. captured images to a centralized server, Present Linux kernels have a time-slice of which controls and coordinates the client 10 ms which can cause different clients to be PCs. marked-off by the multiples of 10 ms. Running the clients with a real-time priority The library libraw1394 provides direct would be a way to get a deterministic access to the IEEE 1394 bus through the behavior from the system but the standard Linux 1394 subsystem's raw1394 user space Linux kernel doesn’t support this. However, interface. Another library libdc1394 is the RTLinux patch can be used for running intended to provide a high level our processes in Real-Time. programming interface for application developers who wish to control IEEE 1394 Camera specific features: Delays may also based cameras that conform to the 1394- be caused by the camera drivers and the based Digital Camera Specification. hardware. One can not know precisely how However, we wanted to make a new API on long does it take for the camera to start top of the existing API so that it is easier to grabbing the frame after getting the trigger use the cameras for image acquisition and signal from the client. We assume this delay processing in our application programs. So, to be same for all the clients. instead of writing the lengthy routines, the users can call the simple functions in the Synchronization of clients: Our setup uses new API and at the same time remain cheap FireWire cameras, which do not have transparent to the lower-level functions. support for Hardware synchronization like the costly SONY DFW-V5000 digital Initially, the server waits for connections camera. Some other FireWire based cameras from different clients. When the connections do provide the support for external hardware are established, it triggers all the clients based triggering, but these are much costly simultaneously for grabbing the images. The than the simple ADS PYRO camera we are clients grab the images and send the using. Hardware triggering would require images/processed results from images to the additional hardware and cabling which server where the server displays them after would hinder the mobility of the system. combining the results from clients. Therefore, we propose a software based Issues in Multiple camera setup synchronization. The scheme is given for all the child processes, it means that the below. synchronous data from all the clients is available and server may read it from the Software Synchronization: In our setup shared memory part described by the same there are currently one server and three value of “SHMGET”. Once the data is read clients. Each client has a FireWire camera. by the parent process, it sets the value of The camera captures a 320X240 frame in flags to “0” to ask more data from the child YUV422 format, converts it to 320X240 processes. RGB and sends to the server for further processing. For processing we need data Synchronized Image Acquisition: The coming from different clients to be software architecture for this setup is shown synchronized, that is data sent from different in the next figure. There is one server, clients should correspond to same time which has three processes running. One is coordinate. For achieving this goal we need the main process, which gives the command to synchronize the clocks of all the clients to clients for image grabbing and controls with the clock of the server. One way to do and coordinates the clients for synchronous this is to simply send the time of the server image capture. One process uses the frames clock to all the clients and setting the time of stored in the shared memory, does the the client to the time value send by the required image processing and stores the server. But the problem in this approach is processed images again in the shared that the time taken to send time related data memory. Another process also uses from server to client requires time, which theshared memory and displays the images adds a substantial discrepancy in sent by all the clients or the processed image synchronization of clients. This problem is results. solved by calculating the time for sending data from server to client over the network The server synchronizes the clients by sending some test data and calculates the according to its clock at the start of the network “lag”. Now “lag” is added to server program. time and sent to all the clients. Clients accept this value as their time value thereby Synchronization Error measurements: synchronizing their clocks with the server Because of the reasons responsible for time.