Challenge D: A world of services for passengers

Development of a Vision based Railway Platform Safeguard

Sehchan Oh1, Youngki Yoon2, Yongkyu Kim3 Korea Railroad Research Institute, Uiwang, Korea123

Abstract The CCTV is the most common safeguard, widely used in many railway applications. However, It has a functional restrictions providing only video information about preset monitoring area, such as boarding platform, and so on. In the system, operators should observe multiple screens continuously during whole day at the station office. It is very hard to detect and keep up with the accident immediately. To solve that problem, we propose a vision based railway platform safeguard using stereo image processing technology. It detects passenger’s danger and informs operators with video and alarms. Moreover, the system automatically deals with the accidents, such as automatic stop, and broadcasting for passenger’s safety on the platform about the accident. In the paper, we present the system configuration, detection process and experimental result.

Introduction In general, railway platform in the metropolitan area are increasingly crowded with passengers, especially during rush hours. Therefore, to reduce accidents on the platform, such as passenger caught in between train doors and fallen on the line and so on, is primary concern of railway operators. The most common safeguard is CCTV, which is widely used in many railway applications. However, It has limited capability providing only video streaming of the boarding platform because it is a passive and lack of intelligence. In the system, operators should watch many CCTV monitors continuously during whole day. Therefore, it is very hard to detect and keep up with the accident immediately. Recently, railway operators in South Korea, have widely installed platform screen door(PSD) to prevent safety accidents at platform area. However, it is difficult to be applied in all the stations such as the station with almost no passengers due to enormous cost of installation and maintenance. The development concept of vision based railway platform system is the replacement of human’s very repeated and continuous monitoring work with intelligent and automatic monitoring system by using highly image processing technology.

CCTV

Station Office Boarding Platform

Fig. 1 Platform Monitoring with CCTV

The Korea Railroad Research Institute has developed such an intelligent automated monitoring system with image processing technology as a government project since 2005. The proposed new type of platform safeguard automatically detects possible accidents on the platform and informs operators about the accidents with video information and alarms. Moreover, the system automatically deals with the accidents, such as automatic , and broadcasting for passenger’s safety on the platform about the accident. Figure 2 shows the concept of vision based platform monitoring system. The application of image processing in the railway environment has requirements such as real-time processing, low cost, the cameras viewpoint in the limited space, etc. Moreover, the illumination conditions are characterized by various light effects, such as sudden change of luminous intensity due to frequent train arrival and departure, shades and reflected lights in the scene, and so on. Due to the environmental restrictions, we adopted stereo vision technology using three-dimensional position information with stereo cameras for minimizing the various illuminant effects.

Challenge D: A world of services for passengers

Multicasting Detection Result

Stereo Camera

Camera Lens

CCR Office Passenger Announcement

Train Emergency Stop

STOP

Fallen Passenger Extracting Features Fig. 2 Development Concept of Vision based Platform Monitoring System

Vision based Railway Platform Safeguard The vision based Railway platform safeguard is mainly composed of camera unit acquiring images in real-time, information processing unit extracting features from the input images, and information providing unit making signal for train stop, passenger announcement and multicasting detection result. To monitor the entire length of the platform, we use multiple stereo and thermal cameras which are installed above the roof of platform edges as show in the Figure 2. The acquired stereoscopic and thermal information from the cameras is used in the information processing unit in order to remove luminance effects and to extracts meaningful information. The number of camera to be installed can be differed by structural feature of the each station platform. For example, it is more cameras needed in case of long curved platform because of fixed camera monitoring area. The information processing unit extracts features to determine the boundary between normal and abnormal operation conditions. With stereo cameras, it extracts 3-D position information of object, and it picks out thermal information of the object with thermal camera, in the preset monitoring area. Final decision making is done with both position and thermal information, such as fallen passenger in the track line, fire in the platform, passengers over the safety-line, and so on. The information providing unit is responsible for data conversion and signal conversion. Data converter produces audio/video information according to the detection result from the information processing unit. For example, an event which a passenger goes over the safe-line occurs, data converter produces an audio message such like, “please step back out of the safety line” and video about the event. Information is physically converted by signal converter for various receivers, such as monitors and speakers in train driver’s cab and station office, track circuit for train emergency stop, and so on.

Information processing unit Information Station Office Providing Unit Stereo Camera #1 A/D Processor #1

Stereo Camera #2 A/D Processor #2 Data Central Control Room Stereo Camera #3 A/D Processor #3 Converter (Audio/Video)

Stereo Camera #N A/D Processor #N Decision Train Driver’s Cab Thermal Camera #1 A/D Processor #1 Maker

Thermal Camera #2 A/D Processor #2 Signal Train Emergency Stop Converter Thermal Camera #3 A/D Processor #3 (D/A, Wire/Wireless)

Thermal Camera #K A/D Processor #K Passenger Announcement

Fig. 3 System Configuration

Challenge D: A world of services for passengers

The functional requirements of the system are described in Table 1. In track monitoring, the smallest size of the object to be detected is 40Cm(W) x 40Cm(D) x 100Cm(H), regarding as a size of an element school student. The system outputs train stop signal and passenger announcement as well as informs operators of the detection result. In platform monitoring, various passenger’s dangers situations are detected. All the system output should be done within one seconds` because processing time is critical factor as considering quite long braking distance of train.

Table 1 System Functional Requirements Processing Division Events to be detected System output time - Multicasting detection result Track Size of fallen object more than - Train stop signal Monitoring 40Cm(W) x 40Cm(D) x 100Cm(H) - Passenger announcement - Multicasting detection result Passengers who exceed safety line - Passenger announcement Within 1 - Multicasting detection result Passengers caught in between train second Platform - Train stop signal doors Monitoring - Passenger announcement - Multicasting detection result Fires in the platform area - Train stop signal - Passenger announcement

Detection Process Each processor in information processing unit follows the detection process described in Figure 4. Train detection means finding out the train state in the monitoring area, which is defined as four different states as follows: OFF-state means there is no train in the area, IN-state means train is coming in the area, ON-state means train is stopped in the area, OUT-state means train is pulling out from the area. The exact determination of the train state in the monitoring area is important because it is top reference point to decide what normal operation and dangerous situation. Track Monitoring is done in OFF-state and detection of passenger’s caught in between train doors is carried out OUT-state. In the track monitoring and platform monitoring, the system uses thermal information to determine whether the object is human or not, as well as the 3-D coordinate information of object.

Start

Train Detection

NO Is OFF state?

YES Is OUT state? Track Monitoring YES

NO Platform Monitoring Is fallen passenger? Platform Monitoring

YES NO YES Fire in platform? Is passenger caught in between train doors? NO NO

YES Does passenger exceeds safety-line?

YES

System Output

Fig. 4 System Detection Flow

Challenge D: A world of services for passengers

To monitor separately both track and platform area, the system predefines two monitoring area as shown in Figure 5(a). The blue lined area is for detection of passengers who exceeds safety-line, and the red one is for detection of fallen passenger from the boarding platform. The way of train detection is detection of track bottom, especially the pattern of railroad crosstie. In the Figure 5(b), red boxes with yellow numbers show six checking points with the number of detections in the monitoring area. If train is coming in the area, all of the checking points are disappeared and system recognizes that the train state is not OFF. Object detection process with stereo image processing algorithm is described in Figure 5(c). It exploits deviation in the number of pixels from the left image with respect to the right image as a reference. The deviation is caused by disparity from the stereo pair. With the deviation, the system calculates distance between object and camera. In general, finding out the deviation is called stereo matching, which searches for the best-matched block from the reference image. In the figure, green boxes show the result of stereo matching, one is object block in the left image and the other is reference block in the right image. The final processing of object detection is presented in Figure 5(d) with 3-D coordinate information of the object in track area. The green typed values of x, y, z mean 3-D coordinates of the object from the camera. In the figure, the center position of the fallen person is at the distance of 2 meters(X-axis), 3.4 meters(Y-axis), 7.7 meters(Z-axis) from the monitoring camera. The yellow typed number means appearance number of the object.

(a) (b)

z x y

(c) (d) Fig. 5 Detection Process in the Information Processing Unit, (a) Monitoring area, (b) Patterns for Train Detection, (c) Stereo Matching for Object, (d) Object’s 3-D Information

Experimental Result The system was installed in Dae-gu station of DMTC(Dae-gu Metropolitan Transit Corporation), and it has been operated in business time since December 2009. Dae-gu station is underground station with 120 meters length of boarding platform. Four stereo cameras was setup with approximately 30 meters intervals, and two thermal cameras are equipped within the two cameras facing each other in the middle to monitor the entire platform area as shown in Figure 6. The monitoring area of one stereo

Challenge D: A world of services for passengers camera is from 5 meters to 30 meters, assuming the immediately below the installation position is 0 meter.

Stereo Camera Stereo with Thermal camera 5m 30m

Safety line 1 2 3 4

2

Fig. 6 Camera Installation Position and Monitoring Area

The vision based railway platform monitoring system is installed as shown in Figure 7. Figure 7(a) shows the installation of cameras in the platform of Dae-gu station. As shown the figure, two-lens camera is stereo camera, three-lens camera is what stereo with thermal camera. Event they are configured in one case, but independently operated by information processing unit. The system servers for information processing unit and information providing unit are presented in Figure 7(b) and 7(c), respectively.

(a)

(b) (c) Fig. 7 System Installation in Dae-gu Station, (a) Stereo and Thermal Cameras, (b) System Server for Information Processing, (c) System Server for Information Providing

Challenge D: A world of services for passengers

As shown in Figure 8(a), the management system for the station operator displays video information of each camera in the middle, and event types in the left side with different colored camera icons. Information display is automatically and instantly changed when an accident occurs to inform operator as shown in Figure 8(c). The camera video image is scaled up and detected object is marked with red object box to easily indentify it.

(b)

(b) Fig. 7 Operations for the Operator Management System, (a) Information Display in Normal Condition, (b) Display Change in Accident Occurrence

Challenge D: A world of services for passengers

With help of the DMTC, the operators of DMTC daily tested intensively the system about the events for fallen passenger and exceeding safety line after business time from 21nd April to 21st May 2010. The detection result is presented in Table 2. The system detected all the events of passenger exceeding safety line. However, it missed some of the events of fallen passenger because, in some case, the pixels of extracted object which is more than 25 meters away from the camera are too few to indentify by image processing. It is caused by resolution of camera and low luminous intensity.

Table 2 Detection Rate of Events for Fallen Passenger and Exceeding Safety Line Detection of Passenger Exceeding Fallen Passenger Detection Day Safety Line Number of Trails Number of Detections Number of Trails Number of Detections 4/22 1 1 2 2 4/23 1 1 3 3 4/24 - - - - 4/25 1 1 3 3 4/26 3 3 8 8 4/27 - - - - 4/28 3 1 11 11 4/29 1 1 2 2 4/30 3 3 4 4 5/1 - - - - 5/2 1 2 9 9 5/3 - - - - 5/4 2 2 6 6 5/5 - - - - 5/6 4 4 5 5 5/7 - - - - 5/8 - - - - 5/9 - - - - 5/10 2 2 4 4 5/11 - - - - 5/12 1 1 2 2 5/13 1 1 2 2 5/14 1 1 1 1 5/15 2 2 5 5 5/16 1 1 2 2 5/17 2 2 10 10 5/18 1 1 2 2 5/19 3 3 8 8 5/20 1 1 3 3 5/21 2 2 12 12 Total 38 35 104 104 Detection Rate(%) 92.1 100

The system server of information processing unit saves every detected event in its data based. The operators can retrieve all the saved events with their operator management system. In Figure 8, some of the saved events about the passengers exceeding safety line in business time are presented. The operators of DMTC strongly requested that the system has to detect person who exceeds safety line even though the current train state is OFF. For all the events, the system outputs passenger announcement with platform speaker as well as alarms to operators.

Challenge D: A world of services for passengers

Fig. 8 Saved Events about Passengers Exceeding Safety Line

In Figure 9, detection results for fallen passenger are presented. In the figure, all the passengers intentionally jumped down from the platform in order to go directly up or down line, or to pick up their dropped belongings. For all the events, the system informed operators and output both train stop

Challenge D: A world of services for passengers signal and passenger announcement.

Fig. 9 Saved Events of Fallen Passengers

Conclusion

In the paper, we proposed a vision based railway platform safeguard, which detects possible accidents on the platform and informs operators about the accidents with video information and alarms. Moreover, the system automatically deals with the accidents, such as automatic train stop, and broadcasting for passenger’s safety on the platform to give warning about the accident. The detection algorithm of the proposed system exploits stereo vision algorithm to minimize ambient illumination changes due to frequent train arrival and departure in the platform. We verify the system performance with experimental result in real railway platform condition. The experimental result shows that detection of train state and object is conducted robustly by using proposed stereo-vision based object detection algorithm. From the results, we need to adjust monitoring distance or consider applying another sensor to compensate missed events of fallen passenger. We expect the proposed monitoring system will play key role in establishing highly intelligent monitoring system for passenger’s safety in future railway environment.

Challenge D: A world of services for passengers

Reference

[1] Sehchan Oh, Hanmin Lee, “Performance Analysis of Vision based Monitoring System for Passenger’s Safety on Railway Platform”, ICCAS, Korea, Oct. 2010. [2] Sehchan Oh, Sunghyuk Park, Changmu Lee “Railway Platform Monitoring System Using Stereo Vision Algorithm for Passenger’s Safety,” Intelec, 2009. [3] Sehchan Oh, Gildong Kim, Wootae Jeong and Youngtae Park, “Vision based Object Detection for Passenger’s Safety in Railway Platform”, ICCAS, Seoul, Korea, Oct. 2008. [4] Y.Sasaki, N.Hiura. “Development of Image Processing Type Fallen Passenger Detecting System,” JR-EAST Technical Review Special Edition Paper, No. 2, pp.66-72, 2003. [5] I.Yoda, K.Sakaue. “Ubiquitous Stereo Vision for Controlling Safety on Platforms in Railroad Station,” IEEJ Tr. on Electronics, Information and Systems, Vol. 124, No. 3, Mar., pp.805-811, 2004. [6] I.Yoda, "Image processing technology for advanced safety to people in railroad transportation - For railroad crossing and station platform," IPSJ Magazine Vol.48, No.1, pp.10-16, Jan. 2007. [7] F.Kruse, S.Milch, H.Rohling. "Multi Sensor System for Obstacle Detection in Train Applications," Proc. of IEEE Tr., June, pp.42-46, 2003. [8] J. Vhzquez, M. Mao, "Detection of moving objects in railway using vision," IEEE Intelligent Vehicles Symposium University of Parma, Parma, Italy Jun. 1447, 2004. [9] N. Paragios and V. Ramesh. An MRF-based approach for real-time monitoring. In IEEE Conference on Computer Vision and Pattern Recognition, 2001. [10] Shigeki Sugimoto, Hayato Tateda, Hidekazu Takahashi, Masatoshi Okutomi, "Obstacle Detection Using Millimeter-Wave Radar and Its Visualization on Image Sequence," icpr, pp. 342-345, 17th International Conference on Pattern Recognition (ICPR'04) - Volume 3, 2004