Coded Rolling Shutter Photography: Flexible Space-Time Sampling Jinwei Gu∗ Yasunobu Hitomi Tomoo Mitsunaga Shree Nayar Columbia University Sony Corporation Sony Corporation Columbia University Abstract We propose a novel readout architecture called coded rolling shutter for complementary metal-oxide semiconduc- tor (CMOS) image sensors. Rolling shutter has tradition- ally been considered as a disadvantage to image quality since it often introduces skew artifact. In this paper, we show that by controlling the readout timing and the expo- sure length for each row, the row-wise exposure discrep- ancy in rolling shutter can be exploited to flexibly sample (a) CMOS image sensor architecture (b) Timing for rolling shutter the 3D space-time volume of scene appearance, and can thus be advantageous for computational photography. The Figure 1. The address generator in CMOS image sensors is used to required controls can be readily implemented in standard implement coded rolling shutter with desired row-reset and row- CMOS sensors by altering the logic of the control unit. select patterns for flexible space-time sampling. We propose several coding schemes and applications: and features. In fact, a few recent studies have demonstrated (1) coded readout allows us to better sample time dimen- the use of conventional rolling shutter for kinematics and sion for high-speed photography and optical flow based object pose estimation [1, 2, 3]. applications; and (2) row-wise control enables capturing In this paper, we propose a novel readout architecture motion-blur free high dynamic range images from a single for CMOS image sensors called coded rolling shutter. We shot. While a prototype chip is currently in development, we show that by controlling the readout timing and exposure demonstrate the benefits of coded rolling shutter via simu- length for each row of the pixel array, we can flexibly sam- lation using images of real scenes. ple the 3D space-time volume of a scene and take pho- tographs that effectively encode temporal scene appearance 1. Introduction within a single 2D image. These coded images are useful CMOS image sensors are rapidly overtaking CCD sen- for many applications, such as skew compensation, high- sors in a variety of imaging systems, from digital still and speed photography, and high dynamic range imaging. video cameras to mobile phone cameras to surveillance and As shown in Fig. 1, the controls of row-wise readout and web cameras. In order to maintain high fill-factor and read- exposure can be readily implemented in standard CMOS out speed, most CMOS image sensors are equipped with image sensors by altering the logic of the address generator column-parallel readout circuits, which simultaneously read unit without any further hardware modification. For con- all pixels in a row into a line-memory. The readout pro- ventional rolling shutter, the address generator is simply a ceeds row-by-row, sequentially from top to bottom. This is shift register which scans all the rows and generates row- called rolling shutter. Rolling shutter has traditionally been reset (RST) and row-select (SEL) signals. For coded rolling considered detrimental to image quality, because pixels in shutter, new logics can be implemented to generate the de- different rows are exposed to light at different times, which sired RST and SEL signals for coded readout and exposure, often causes skew and other image artifacts, especially for as shown in Fig. 2. Since the address generator belongs to moving objects [11, 13, 6]. the control unit of CMOS image sensors [9, 17], it is easy to From the perspective of sampling the space-time volume design and implement new logics in the address generator of a scene, however, we argue that the exposure discrepancy using high level tools. in rolling shutter can actually be exploited using computa- We have begun the process of developing the prototype tional photography to achieve new imaging functionalities sensor. We expect to have a fully programmable coded ∗This research was supported in part by Sony Corporation and the Na- rolling shutter sensor in 18 months. Meanwhile, in this pa- tional Science Foundation (IIS-03-25867 and CCF-05-41259). per, we demonstrated coding schemes and their applications 1 via simulations. The simulation experiments are performed with real images, i.e., full space-time volumes of scene ap- pearance recorded with high-speed cameras were used to synthesize the output images of a coded rolling shutter sen- sor. These synthesized images thus have similar character- istics as the images captured with a real sensor. 2. Rolling Shutter and Related Work (a) Global shutter (b) Flutter shutter [22] We first introduce some background related to rolling shutter. As shown in Fig. 1a, the exposure in CMOS image sensors is controlled by the row-reset and row-select sig- nals sent from the row address decoder – each row becomes photosensitive after a row-reset signal, and stops collecting photons and starts reading out data after a row-select sig- nal. Because there is only one row of readout circuits, the readout timings for different rows cannot overlap. In rolling (c) Rolling shutter (d) Coded rolling shutter shutter, as shown in Fig. 1b, the readout timings are shifted Figure 2. Timing charts for four types of camera shutter function. sequentially from top to bottom. duces high frequency components for motion deblur. Since We denote the reset time, the readout time, and the ex- the coding is fixed for all pixels, it is in effect a coded global posure time for a row with , , and , respectively. ∆ts ∆tr ∆te shutter, which is also a 1D function of time t. For conven- For typical CMOS sensors, is around and ∆ts 1 5µs ∆tr tional rolling shutter (Fig. 2c), S(x,y,t) = S(t t (y)) = is around 15 40µs. For an image sensor∼ with M rows, − r ∼ S(t y∆tr). It is still a 1D function because of the fixed we denote the reset timing (i.e., the rising edge of row- sequential− readout order. reset signals) and the readout timing (i.e., the falling edge In contrast, the proposed coded rolling shutter (Fig. 2d) of row-readout signals) for the -th row ( ) with y 1 y M extends the shutter function to 2D S(y,t) in which both the and , respectively. For rolling shutter,≤ ≤ we have ts(y) tr(y) readout timing t (y) and the exposure time ∆t (y) can be t (y)=y∆t and t (y)=t (y) ∆t ∆t ∆t . r e r r s r − e − r − s row-specific. As mentioned earlier, because there is only Recent works have modeled the geometric distortion one row of readout circuits, the readout timings for different caused by rolling shutter [1, 11], and proposed methods rows cannot overlap, which imposes a constraint on t (y). to compensate for skew due to planar motion [13, 6]. r Specifically, for an image sensor with M rows, the total Wilburn et al. [24] demonstrated the use of rolling shut- readout time for one frame is M∆tr. Each valid readout ter with a camera array for high-speed photography. Many timing scheme will correspond to a one-to-one assignment components of CMOS image sensors have also been re- of the M readout timing slots to the M rows. This is a designed for specific applications, such as HDR imag- typical assignment problem in combinatorial optimization. ing [25, 15] and multi-resolution readout [12]. These ideas Figure 3 shows one simple assignment, which is adopted in are giving rise to a new breed of image sensors, referred to the conventional rolling shutter. as “smart” CMOS sensors [20], which have spurred signifi- The remainder of the paper demonstrates several coding cant interest among camera manufactures. schemes and their applications. We limited the coding to be 3. Coded Rolling Shutter: An Overview within one frame time. 1. Coded readout timing tr(y) for better sampling over In coded rolling shutter, both ts(y) and tr(y) can be con- trolled by the address generator. As a result, the exposure the time dimension for optical flow and high-speed photography, as detailed in Sec. 4. time, ∆te(y), can also be varied for different rows. Let E(x,y,t) denote the radiance of a scene point (x,y) at time 2. Coded exposure time ∆te(y) for high dynamic range t, and S(x,y,t) denote the shutter function of a camera. (HDR) imaging. With this control, we propose a sim- The captured image I(x,y) is ple row-wise auto-exposure in Sec. 5.1, which is ef- ∞ fective for outdoor, natural scenes. Moreover, if both I(x,y) = E(x,y,t) S(x,y,t) dt. (1) tr(y) and ∆te(y) are controllable, we show in Sec. 5.2 Z−∞ · that motion-blur free HDR images can be recovered Figure 2 shows four types of shutter functions. For the from a single coded image. global shutter (Fig. 2a) widely used in CCD image sensors, 4. Coded Readout and Its Applications S(x,y,t) is a 1D rectangular function. Raskar et al. [22] In this section, we show how to use coded readout to bet- proposed the flutter shutter (Fig. 2b), which breaks the sin- ter sample the time dimension by shuffling the readout tim- gle integration time into multiple chunks and thus intro- ings tr(y) among rows. We propose two coding schemes. Figure 3. Coded rolling shutter is constrained as an assignment problem because readout timings cannot overlap between rows. (a) Interlaced readout (K =2) (b) Diagram of interpolation Here we show the assignment of a conventional rolling shutter.