Feature Articles: Communication Science that Connects Information and Humans Information Processing of Sensor Networks Takayuki Suyama and Yutaka Yanagisawa Abstract In this article, we introduce technologies for collecting information from many sensor nodes deployed in the environment and/or attached to people, interpreting the sensor data as meaningful information, and presenting it to people appropriately. 1. Introduction In our daily lives, we are surrounded by many sen- sors. For example, a video game machine uses an accelerometer as one of its input devices, and a mobile phone uses GPS (global positioning system) to determine the user’s location. In the field of weath- er forecasting, pressure, temperature, and other infor- mation about various places is collected in a network and utilized in creating a weather forecast. Although individual sensors are used in various situations, sen- sor networks are becoming easier to use as a result of sensor evolution and the spread of sensor network technology driven by progress in semiconductor technology. Sensor networks and their information Fig. 1. Sensor nodes. processing technology are expected to be used for various purposes, such as collecting information to enable the provision of appropriate services that suit sides of about 1 mm to 2 mm, and observes environ- a particular situation and for recognizing the real mental information collected from the sensor nodes. world. However, the need for miniaturization, which was initially advocated and dictated the sensor node’s 2. Research on sensor networks size, is no longer considered to be such a high prior- ity. Research on sensor networks is progressing with a The flow of information on a sensor network is focus on a sensor node equipped with various sen- shown in Fig. 2. We assume that sensor nodes are sors, a calculation function (central processing unit deployed in the environment and/or attached to peo- (CPU), memory, etc.), wireless-communications ple. First of all, it is necessary to collect information functions, a battery, and networking capability. The efficiently from a lot of sensor nodes. Because the sensor nodes that we have developed are shown in collected sensor data is the waveform of a time series, Fig. 1. Such research was triggered by the Smartdust it is difficult for humans to understand the waveform’s project announced in 1999. In the Smartdust concept, meaning. So the next step is to interpret such time the user deploys many sensor nodes, each of which series information. When suitably interpreted mean- consists of sensors, a battery, a digital signal proces- ing is presented to the user, the collected sensor data sor, and a transmitter assembled into a cuboid with is finally being used effectively. 1 NTT Technical Review Feature Articles Collection Interpretation Sensor data 50 40 30 Information 20 10 0 10 20 30 40 50 60 70 80 90 Sensor network Presentation Fig. 2. Flow of sensor data. Same behavior Sensor node 1 Sensor node 2 Program Program Program 1 Program 2 VM VM CPU 1 CPU 2 CPU 1 CPU 2 Fig. 3. Virtual machine for wireless sensor node. 3. Virtual machine for wireless sensor node 4. Sensor data collection from many sensor nodes When many sensor nodes are used, it is important to reduce the cost of using them. In particular, in a In order to collect information from many sensor network where two or more kinds of sensor (e.g., nodes, analyze the situation in the real world from the ones from different makers or with different CPUs) sensor data, or create a sensor network that provides nodes are intermingled, the usage cost is high. To services according to a situation, it is necessary to enable easy use in such situations as well, we have collect data efficiently. Data compression technology developed a virtual machine (VM) for a wireless sen- based on the similarity of sensor data is shown in sor node. Usually, when users use various different Fig. 4. Sensor data measuring the situation in the real sensor nodes, as shown on the left in Fig. 3, even world has the characteristics that there are periodic when the same operation is carried out, another pro- changes in the same sequence or changes in similar gram needs to be prepared. But with the VM, even if values obtained from neighboring sensors. Figure the CPUs differ, the same program can be run on 4(a) shows the compression when sensor data from a them. single sensor node shows a similar value repeatedly. Our VM is based on CIL (Common Intermediate First, the sequence of sensor data repeated periodi- Language). Since CIL originated in Windows .NET, cally is cut out. Then, the sensor data is decomposed executable code can be created using the develop- into typical data sequences and weight variables by ment environment of MS-Windows. Moreover, a sen- using singular value decomposition (SVD). sor network’s manner of operation can be changed, Weight variables express the importance for every even after sensor nodes have been deployed in the typical data sequence, and the original sensor data environment, because this VM has a function for can be restored by using these typical data sequences wirelessly distributing created programs. and weight variables. When information is transmit- ted from a sensor node, it is possible to compress the Vol. 10 No. 11 Nov. 2012 2 Feature Articles (a) Compression of repeated similar sensor data Typical data sequences Decomposition to typical data sequences and weight variables by SVD (b) Compression of neighboring sensors’ data Sensor 1 Sensor 2 Sensor 3 Sensor 4 Fig. 4. Compression of sensor data. Base station Sensor G H Sensor E Sensor F Sensor B Sensor A DC Fig. 5. Collection of sensor data in a hierarchical network. information by not transmitting all of the typical data reducing the data amount in the wireless communica- sequences but transmitting only typical data sequenc- tions, we can reduce the battery consumption, es that make a high contribution. However, because enabling the sensors to measure efficiently for a long not all of the information is sent, a certain amount of time and collect more sensor information [1]. error occurs. An example of transmitting the data of neighboring sensors is shown in Fig. 4(b). Tempera- 5. Interpretation of collected data ture may be data that shows similar values among neighboring sensors. This sensor data can be com- If the information collected from the sensors can be pressed by using the abovementioned method. The interpreted and human activity can be inferred, this sensor data is transmitted to the base station while the kind of technique lets us make applications such as sensor nodes are used as a hierarchical network ones for automatic lifelog creation, elderly care sup- (Fig. 5). After the data of sensor A and sensor B has port, and home automation. We assume that almost been compressed and sent to sensor E, it is further all human activities are performed using the hands, so sent to sensor G. Finally, the data is sent to a base sta- we developed technology to infer the kind of action tion. Thus, it becomes possible to reduce the amount being performed by using only a sensor device of information even further by applying the compres- attached to a person’s wrist. Moreover, whereas pre- sion technique shown in Fig. 4 hierarchically. By vious methods can infer only simple human activities 3 NTT Technical Review Feature Articles Microphone Camera Accelerometer Light sensor Digital compass Prototype sensor device Wireless sensor device Cocoa Images Acceleration data Use cocoa Use fridge Fig. 6. Wrist-worn sensor devices and example of sensor data. such as running and walking by using an accelerom- color information for recognition. A histogram of the eter, our method can infer more complex activities color contained in a picture is used as the feature. with a comparatively high degree of abstraction, such Other sensor data is also converted into correspond- as vacuuming and making cocoa. The wrist-worn ing features. Next, the extracted features are input to sensor device consists of two or more kinds of sen- a classifier. sors, for example, a camera, accelerometer, and First, we prepare several binary classifiers—one for microphone. The cameras acquire visual information each activity that we want to identify—and the prob- about an object being manipulated, the accelerometer abilities for all actions are output. For example, the captures hand movements, and the microphones cap- data input from one binary classifier to the operation tures ambient sounds [2]. Prototype sensor devices “making cocoa” judges the degree of agreement with and the information obtained from them are shown in the cocoa-making operation. Since 15 kinds of Fig. 6. actions were identified in the experiment, 15 classi- Human activity is inferred from sensor data by fiers were used. Features were input in parallel with applying the machine learning technique. A system these classifiers. Next, they were input to a hidden that can infer actions from sensor data is shown in Markov model (HMM) classifier, the time direction Fig. 7. First, features are extracted from the sensor was inferred, and the output result of the binary clas- data. For example, in the case of a camera image, the sifier produced the final classification result. In the raw image might be heavily blurred and privacy experiment, when all the sensors were used, activities issues might arise depending on what other details are were recognized with probabilities ranging from 80% captured, including ones in reflections. In our system, to 90%. these problems are avoided by using the picture’s Vol. 10 No. 11 Nov. 2012 4 Feature Articles Static classifiers HMM classifiers Binary classifier 1 HMM 1 Sensor Feature Binary classifier 2 HMM 2 HMM likelihoods data extraction Binary classifier n HMM n Fig.