Downloaded by guest on September 27, 2021 www.pnas.org/cgi/doi/10.1073/pnas.1703715114 a Freeman T. William and Freeman M. Dennis Wadhwa Neal motions quantifying small and visualizing for microscopy Motion r iutnosylclzdi pta oain(x location spatial that orientation in ), S1 and localized (Fig. sinusoids simultaneously basis of windowed are local- sum by a spatially is approximated image a over- functions, transformed to The an transform. similar (6), Fourier transform, ized pyramid The wavelet steerable waves. linear sine complex complete complex the windowed is of representation shifts rep- pixels phase are displacements frame’s by where resented each representation of wavelet-like this intensities a does It into captured phase. the local spatial transforming using by by video a in displacements magnified the both analyses. see for quantitative can allowing and variances, user qualitative with values, the their by obtain Thus, and introduced (5). motions errors noise the quantify and sensor we motions camera motions, subpixel tiny object’s visualizing the to engineer- both addition and visualiza- In scientific the analysis. to extend magnification ing in video We kind by any band. produced of frequency tion motions temporal small specified amplify a to videos processes which of effectiveness showing the hearing, verifying metamaterials. and designed mammalian structures, in of modes used vibration engineer- motions and biology optical visualizing from problems as ing: disparate three motions in demonstrate We utility tiny forms. its tiny of of inspection the inspection enables microscopy the enables microscope M visualization unknown previously of discovery of the variety to phenomena. a leading the mecha- over instances, scales, dynamics a these length hidden demonstrating In uncovers hearing. ear microscope in inner motion separation frequency the of in nism extra- found an of tissue motions cellular nanometer with metamaterial and matrix units, a elastic resonating an embedded of tem- through vibrations propagation and wave micrometer spatial demonstrating analysis, simultaneous vibra- resonant demonstrating modal to the bridge poral enough are a macro- They large of spanning scales. made tions length shown, are nanoscopic are producing motions to visualizations by scopic the scientific them which quantifies Three visualizes in see. that then video and tool new videos a computational in to a present motions precursors we tiny microscope,” Here, be failure. or “motion mechanical of mechanisms, a case the physical in reveal motions large impor- can are threshold and Although this threshold. below tant some motions can- tiny than we visualize, smaller to and are difficult perceiving sensitivity, that 2017) limited motions at 5, see March has remarkable review not it for is (received motions, 2017 system interpreting 22, August visual and approved human and TX, the Austin, Austin, Although at Texas of University Press, H. William by Edited and 21218; MD Baltimore, University, g 02139; MA Cambridge, 02139; MA 02139; Cambridge, MA Technology, Cambridge, of Institute Massachusetts Engineering, Environmental optrSineadAtfiilItliec aoaoy ascuet nttt fTcnlg,Cmrde A02139; MA Cambridge, Technology, of Institute Massachusetts Laboratory, Intelligence Artificial and Science Computer colo niern n ple cecs avr nvriy abig,M 02138; MA Cambridge, University, Harvard Sciences, Applied and Engineering of School h oinmcocp hrceie n mlfistn local tiny amplifies and characterizes microscope motion The (1–4), magnification video on based is microscope motion The z n nlz ennflbtsalmtos h motion The motions. visual- small to but meaningful technique analyze computational and ize a is microscopy otion | motion a,1 θ ahbssfnto ofcetgvsspatially gives coefficient function basis Each . utnG Chen G. Justin , | f d eateto lcrclEgneigadCmue cec,MsahstsIsiueo ehooy abig,M 02139; MA Cambridge, Technology, of Institute Massachusetts Science, Computer and Engineering Electrical of Department mg processing image eerhLbrtr fEetois ascuet nttt fTcnlg,Cmrde A02139; MA Cambridge, Technology, of Institute Massachusetts Electronics, of Laboratory Research c,d,f rlB Oral , a,e,f,2 i okn xrm aeil nttt,JhsHpisUiest,Blioe D21218 MD Baltimore, University, Hopkins Johns Institute, Materials Extreme Hopkins uy ¨ a,b uk ¨ oahnB Sellon B. Jonathan , ozt ¨ urk ¨ b a Wang Pai , , y ,scale ), g ii Sun Sijie , c,d oga Wei Donglai , r , 1073/pnas.1703715114/-/DCSupplemental at online information supporting contains article This 2 option. access open 1 PNAS the through online available Freely Submission. Direct PNAS a is article This interest. of conflict paper. no the declare wrote W.T.F. authors J.B.S., and The F.D., J.G.C., K.B., N.W., S.H.K., and S.S., data; W.T.F. P.W., analyzed O.B., and D.M.F., D.W. S.H.K., R.G., and D.W., S.S., P.W., J.B.S., R.G., J.G.C., N.W., D.W., research; J.B.S., performed J.G.C., N.W., research; designed W.T.F. and S3 (Fig. it) along those than reliable more are edge coeffi- an orthogonal the functions to to basis for proportion coefficients direct (e.g., amplitude in cient’s Dif- varies orientations, phase Phase and local Local scale spatial Between of across orienta- reliability (Relation The function’s (7) Motions). and motion basis ferences 2D corresponding the the and each of tion case, product this dot In the location. difference, spatial phase each coefficient’s at motion small single, phase amplitude a an has and information frequency local t orsodn au ntefis frame, first time the at in image value transformed corresponding the its of coefficient each of ence i.S2 produce (Fig. to video pyramid output steerable motion-magnified frame’s the each time back each at transform function then basis each to for factor phases magnification modified motion obtain desired to the by due shifts phase components filtering filtered spatially remove and and temporally interest by noise of motions isolate We uhrcnrbtos ...... F.D., K.B., S.H.K., O.B., D.M.F., R.G., M.R., D.W., J.B.S., J.G.C., N.W., contributions: Author owo orsodnesol eadesd mi:[email protected]. Email: addressed. be should 94043. correspondence CA whom View, To Mountain Inc. Google Research, Google address: Present rsne nti td,tevsaiain eelimportant eye. naked reveal the to visualizations invisible examples are the representative that study, motions the this In in signals. presented and spurious see ampli- of the avoid to these fication to analysis of enough noise careful Amplification large involves motions analysis. tiny them for motions make those rerender- to quantifies microscope by motions sequence motion video small The captured ing biolog- motion. a important in of motions understand modes amplifies and physical see and to ical possible that it microscope, motion makes the tool, visualization Here, a difficult. introduce extremely we is motions dynamic to small of visu- techniques alization microscopes), (e.g., optical features physical are static there small visualize Although threshold. amplitudes with a motions below small seeing difficulty have Humans Significance g eetmt oin ne h supinta hr sa is there that assumption the under motions estimate We oapiymtos ecmueteuwapdpaediffer- phase unwrapped the compute we motions, amplify To ugHo Kang Hoon Sung , h φ a eateto ehnclEgneig on Hopkins Johns Engineering, Mechanical of Department ∆ r ihe Rubinstein Michael , c avr-I rga nHat cecsadTechnology, and Sciences Health in Program Harvard-MIT ,θ PNAS φ (x r ,θ , (x y | ). , coe 1 2017 31, October y , t := ) g,h,i ai Bertoldi Katia , φ r ,θ . ∆ (x e φ oze Ghaffari Roozbeh , | , b r y eateto ii and Civil of Department ,θ o.114 vol. , e sapoiaeyeulto equal approximately is , t ogeRsac,Gol Inc. Google Research, Google ) www.pnas.org/lookup/suppl/doi:10. − ∆φ φ g | (3). ) r Fr , ,θ r o 44 no. ,θ (x d Durand edo ´ eapiythe amplify We . , y A , r | 0). ,θ d 11639–11644 (x , , y t t ) from We . a,f and and [1] ,

COMPUTER SCIENCES ABeach frame. We compute the sample covariance of the esti- mated motion vectors in this simulated video (Fig. S7 and Noise 1 Model and Creating Synthetic Video). We show analytically, and 0.75 via experiments in which the motions in a temporal band are 0.5 known to be zero, that these covariance matrices are accurate

Correlation 0.25 for real videos (Analytic Justification of Noise Analysis and Figs.

0 S8 and S9). We also analyze the limits of our technique by com- 10-3 10-2 10-1 100 paring to a laser vibrometer and show that, with a Phantom RMS Motion Size (px) V-10 camera, at a high-contrast edge, the smallest motion we can Fig. 1. A comparison of our quantitative motion estimation vs. a laser detect is on the order of 1/100th of a pixel (Fig. 1 and Fig. S4). vibrometer. Several videos of a cantilevered beam excited by a shaker were taken with varying focal length, exposure times, and excitation magnitude. Results and Discussion The horizontal, lateral motion of the red point was also measured with a We applied the motion microscope to several problems in biol- laser vibrometer. (A) A frame from one video. (B) The correlation between the two signals across the videos vs. root mean square (RMS) motion size ogy and engineering. First, we used it to reveal one component of in pixels (px). Only motions at the red point in A were used in our analysis. the mechanics of hearing. The mammalian cochlea is a remark- More results are in Fig. S4. able sensor that can perform high-quality spectral analysis to dis- criminate as many as 30 frequencies in the interval of a semitone (8). These extraordinary properties of the hearing organ depend Low-Amplitude Coefficients Have Noisy Phase). We combine on traveling waves of motion that propagate along the cochlear information about the motion from multiple orientations by solv- spiral. These wave motions are coupled to the extremely sensitive ing a weighted least squares problem with weights equal to the sensory receptor cells via the tectorial membrane, a gelatinous amplitude squared. The result is a 2D motion field. This pro- structure that is 97% water (9). cessing is accurate, and we provide comparisons to other algo- To better understand the functional role of the tectorial mem- rithms and sensors (Fig. 1, Synthetic Validation, and Figs. S4 brane in hearing, we excised segments of the tectorial membrane and S5). from a mouse cochlea and stimulated it with audio frequency For a still camera, the sensitivity of the motion microscope is vibrations (Movie S1 and Fig. 2A). Prior work suggested that mostly limited by local contrast and camera noise—fluctuations motions of the tectorial membrane would rapidly decay with dis- of pixel intensities present in all videos (5). When the video tance from the point of stimulation (10). The unprocessed video is motion-magnified, this noise can lead to spurious motions, of the tectorial membrane appeared static, making it difficult to especially at low-contrast edges and textures (Fig. S6). We mea- verify this. However, when the motions were amplified 20 times, sure motion noise level by computing the covariance matrix of waves that persisted over hundreds of micrometers were revealed each estimated motion vector. Estimating this directly from the (Movie S1 and Fig. 2 B–E). input video is usually impossible, because it requires observing Subpixel motion analysis suggests that these waves play a the motions without noise. We solve this by creating a simu- prominent role in determining the sensitivity and frequency lated noisy video with zero motion, replicating a static frame selectivity of hearing (11–14). Magnifying motions has provided of the input video and adding realistic, independent noise to new insights into the underlying physical mechanisms of hearing.

AB C

DE

Fig. 2. Exploring the mechanical properties of a mammalian tectorial membrane with the motion microscope. (A) The experimental setup used to strobo- scopically film a stimulated mammalian tectorial membrane (TectaY1870C/+). Subfigure Copyright (2007) National Academy of Sciences of the United States of America. Reproduced from ref. 12. (B) Two of the eight captured frames . (Movie S1, data previously published in ref. 13). (C) Corresponding frames from the motion-magnified video in which displacement from the mean was magnified 20×. The orange and purple lines on top of the tectorial membrane in B are warped according to magnified motion vectors to produce the orange and purple lines in C.(D) The vertical displacement along the orange and purple lines in B is shown for three frames. (E) The power spectrum of the motion signal and noise power is shown in the direction of least variance at the magenta and green points in B.

11640 | www.pnas.org/cgi/doi/10.1073/pnas.1703715114 Wadhwa et al. Downloaded by guest on September 27, 2021 Downloaded by guest on September 27, 2021 ae os tnaddvain.W loue accelerometers used esti- also the We and deviations. points, standard three noise at mated time versus displacements videos, nFg 3 Fig. In 400× magnified .Tetm twihthe which at time The S2 ). (Movie shown is in video points as orange motion-magnified Same and the (D) (C green, from shown. “impact.” cyan, slice are as the time line marked at A red band is the shown. 1.8-Hz at lowered also to slice fully are time 1.6- is points a a green span and in and video central motions resulting cyan for the the shown from at are frame accelerometers microscope A from motion lowered. the was span from central SD the noise while and filmed was span left The (B) 3. Fig. in revealed are shapes (Fig. video span. modal input left its away the of in the m Two still 3B). impacted completely 80 (Fig. looks and span from left lowered the bridge was this, suspension Despite span a central of The span 3A). left the ing mit- carefully. be can viewpoint only this the visible, choosing While are by motions. plane igated image the the of in interpretation motions structural easy for measurement, motion-magnified allows the the and video by dense, spatially unaltered are is measurements the structure the The affect sensors. and, tional can tedious, mass and accelerometers’ difficult the be measurement. structures, can light structure for densely but a analysis, modal instrumenting for used traditionally accelerom- been Contact have eters (17). excitation broadband con- However, operational assuming excitation. under ditions on made input be known locations can a different measurements this to approximate many Typically, response at in (16). vibrations structure structures the measuring in by damage done and or is models changes element detect (15). finite behavior validate to to dynamic mode are its and applications characterize Common frequencies to resonant measured structure’s are shapes a which biological in of analysis, multitude a of and motions see to systems. nanoscale applied the be could interpret microscope motion the Ultimately, ahae al. et Wadhwa C Displacement (mm) A eapidtemto irsoet oa nlssb film- by analysis modal to microscope motion the applied We tradi- over advantages many offers microscope motion The modal of field the to microscope motion the applied also We -0.15 -0.15 -0.15 0.15 0.15 0.15 020 0 10 h ue pn ftebig r xdwietecnrlsa oe vertically. moves span central the while fixed are bridge the of spans outer The (A) bridge. lift a of shapes modal reveals microscope motion The C Motion microscope Noise standarddeviation and Impact eso iesie rmtemotion-magnified the from slices time show we D, 16H o18H)ad250× and Hz) 1.8 to Hz (1.6 Time (s) Accelerometer lift span Central

Space (y) Time 24H o27Hz). 2.7 to Hz (2.4 x400 (1.6-1.8Hz) oi S2 Movie u o oin na24 o27H band. 2.7-Hz to 2.4- a in motions for but C, when B oisS4 (Movies if correctly determine functioning to difficult is it metamaterial making the meta- this stationary, vibrated, when appears Even con- material (24). cores beams elastic copper four of to consists nected which units, resonating embedded (23). specialized vibrometers highly laser and scanning expensive metamateri- like using of the setups possible been in use only everywhere has the motions als Visu- through mechanical measurements. gaps the point band alizing provide only the investigations which directly capturing experimental accelerometers, to on the ampli- impossible focused of small it majority have the makes the waves as them, propagating However, visualize the systems. of phenom- these wave tude in elastic observed the characterize ena been acoustic experimentally have (20), efforts to applica- Several cloaking (22). made potential reduction (19), noise their and guiding (21), and wave imaging physics include rich (18) which attention their tions, much both received propa- of have the struc- They because control waves. artificially and elastic metamaterials, of manipulate gation to elastic designed of materials tured functioning , Pipe the a ify of Shapes of (Modal hammer shapes a modal with the struck S10 show is Fig. we it example, after pipe second a a within In (Fig. accelerometers points bars. the those error matches of microscope two at motion bridge The the 3B). of motions the measure to irtos egi nih n nesadn ftesse by system the of motion. of understanding its attenuation amplifying and visually successful insight the gain verify We to vibrations. measure- difficult point it provides two making only with ments, method measured This were (24). vibrations accelerometers miniscule these Previously, D

Displacement (mm) Accelerometer2 Accelerometer 1 efcso eaaeilcmrsn neatcmti with matrix elastic an comprising metamaterial a on focus We ver- to microscope motion the used we example, final our In -0.3 -0.3 -0.3 0.3 0.3 0.3 020 0 10 and , Motion microscope Noise standarddeviation Impact oi S3 Movie PNAS Time (s) | ). coe 1 2017 31, October Accelerometer 1m

Space (y) Space (y) | Time Time o.114 vol. obyitgae data integrated Doubly B. x250 (2.4-2.7Hz) | o 44 no. Displacement ) and | 11641 S5).

COMPUTER SCIENCES B C D

A

Fig. 4. The motion microscope is used to investigate properties of a designed metamaterial. (A) The metamaterial is forced at 50 Hz and 100 Hz in two experiments, and a frame from the 50-Hz video is shown. (B) One-dimensional slices of the displacement amplitude along the red line in A are shown for both a finite element analysis simulation and the motion microscope. (C) A finite element analysis simulation of the displacement of the metamaterial. Color corresponds to displacement amplitude, and the material is warped according to magnified simulated displacement vectors. (D) Results from the motion microscope are shown. Displacement magnitudes are shown in color at every point on the metamaterial, overlayed on frames from the motion-magnified videos (Movies S4 and S5).

The elastic metamaterial was forced at two frequencies, 50 Hz We use a four-orientation complex steerable pyramid specified by Portilla and 100 Hz, and, in each case, it was filmed at 500 frames per and Simoncelli (25). We use only the two highest-frequency scales of the second (FPS) (Fig. 4A). The motions in 20-Hz bands around the complex steerable pyramid, for a total of eight subbands. We use a Gaussian forcing frequencies were amplified, revealing that the metamate- spatial smoothing kernel with a SD of 3 pixels and a support of 19×19 pixels. rial functions as expected (24), passing 50-Hz waves and rapidly The temporal filter depends on the application. attenuating 100-Hz waves (Movies S4 and S5). We also com- Noise Model and Creating Synthetic Video. We estimate the noise level pared our results with predictions from a finite element analysis function (26) of a video. We apply derivative of Gaussian filters to the image simulation (Fig. 4 B and C). In Fig. 4D, we show heatmaps of in the x and y directions and use them to compute the gradient magnitude. the estimated displacement amplitudes overlaid on the motion- We exclude pixels where the gradient magnitude is above 0.05 on a 0 to magnified frames. We interpolated displacements into texture- 1 intensity scale. At the remaining pixels, we take the temporal variance less regions, which had noisy motion estimates. The agreement and mean of the image. We divide the intensity range into 64 equally sized between the simulation (Fig. 4C) and the motion microscope bins. For each bin, we take all pixels with mean inside that bin and take the (Fig. 4D) demonstrates the motion microscope’s usefulness in mean of the corresponding temporal variances of I to form 64 points that verifying the correct function of the metamaterial. are linearly interpolated to estimate the noise level function f. Conclusion Estimating Covariance Matrices of Motion Vectors. For an input video I(x, y, t), we use the noise level function f to create a synthetic video Small motions can reveal important dynamics in a system under p study, or can foreshadow large-scale motions to come. Motion IS(x, y, t) = I0(x, y, 0) + In(x, y, t) f(I0(x, y, 0)) [4] microscopy facilitates their visualization, and has been demon- that is N frames long. We estimate the covariance matrices of the motion strated here for motion amplification factors from 20× to 400× vectors by taking the temporal sample covariance of IS, across length scales ranging from 100 nm to 0.3 mm. 1 X

T ΣV = VS(x, y, t) − V¯ S(x, y) VS(x, y, t) − V¯ S(x, y) , [5] N − 1 Materials and Methods t Quantitative Motion Estimation. For every pixel at location (x, y) and time where V¯ (x, y) is the mean over t of the motion vectors. t, we combine spatial local phase information in different subbands of the S The temporal filter reduces noise and decreases the covariance matrix. frames of the input video using the least squares objective function, Oppenheim and Schafer (27) show that a signal with independent and iden-   2 2 X ∂φr ,θ ∂φr ,θ tically distributed (IID) noise of variance σ , when filtered with a filter with arg min 2 i i , i i · ( , ) − ∆φ . Ar ,θ u v ri,θi [2] P 2 2 u,v i i ∂x ∂y impulse response T(t), has variance t T(t) σ . Therefore, when a temporal i P 2 filter is used, we multiply the covariance matrix by t T(t) . Arguments have been suppressed for readability; ( , , ) and Ari,θi x y t φ (x, y, t) are the spatial local amplitude and phase of a steerable pyramid ri,θi Comparison of Our Motion Estimation to a Laser Vibrometer. We compare representation of the image, and u(x, y, t) and v(x, y, t) are the horizontal the results of our motion estimation algorithm to that of a laser vibrometer, and vertical motions, respectively, at every pixel. The solution (V = (u, v)) is which measures velocity using Doppler shift (28). In the first experiment, a our motion estimate and is equal to cantilevered beam was shaken by a mechanical shaker at 7.3 Hz, 58.3 Hz, −1 V = (XT WX) (XT WY), [3] 128 Hz, and 264 Hz, the measured modal frequencies of the beam. The rel- ative amplitude of the shaking signal was varied between a factor of 5 and where X is N × 2 with ith row ( ∂ φ , ∂ φ ), Y is N × 1 with ith row ∂x ri,θi ∂y ri,θi 25 in 2.5 increments. We simultaneously recorded a 2,000 FPS video of the 2 ∆φr ,θ , and W is a diagonal N × N matrix with ith diagonal element A . i i ri,θi beam with a high-speed camera (VisionResearch Phantom V-10) and mea- To increase the signal-to-noise ratio, we assume the motion field is con- sured its horizontal velocity with a laser vibrometer (Polytec PDV100). We stant in a small window around each pixel. This gives additional constraints repeated this experiment for nine different excitation magnitudes, three from neighboring pixels, weighted by both their amplitude squared and the focal lengths (24 mm, 50 mm, 85 mm) and eight exposure times (12.5 µs, corresponding value in a smoothing kernel K, to the objective described in 25 µs, 50 µs, 100 µs, 200 µs, 300 µs, 400 µs, 490 µs), for a total of 20 high- Eq. 3. To handle temporal filtering, we replace the local phase variations speed videos. The beam had an accelerometer mounted on it (white object ∆φr,θ (x, y, t) with temporally filtered local phase variations. in Fig. 1A), but we did not use it in this experiment.

11642 | www.pnas.org/cgi/doi/10.1073/pnas.1703715114 Wadhwa et al. Downloaded by guest on September 27, 2021 Downloaded by guest on September 27, 2021 0 wsok J(90 iedcdso eerho ohermechanics. cochlear on research of decades Five (1980) JJ Zwislocki 10. ahae al. et Wadhwa motions. inaccu- similar to have regions pixels accurate adjacent from that field assuming We regions, motion matrix. rate the covariance the interpolate motion to to the perpendicular how in direction show edges, reflected the are like in inaccuracies accurate structures These be one-dimensional edge. only at will and, field motion all, the at motion the estimate Interpolation. Field Motion parameters. specified the with a filter with Butterworth band-passed band-pass were first-order acceleration integrated doubly the and 3 placement Fig. In was displacement. span to central gration the before s point. 5 lowest about its to started lowered was Filming reso- traffic. a with marine FPS 800 30 of at GS3-U3-23S6M-C) lution (model camera Grasshopper3 Gray Sequence. Bridge Filming S4 I (Fig. noisier are times exposure lower with S4 iue w hnsta oiieycreaewt oinsz i ies ( pixels) (in size motion with correlate positively from that signals things corre- two two nitude, lower the had prevents and video noisier with the were in Displacements pixel noise matching. signals. a that of the indicating motion 1/100th lations, between between sig- the than vibrometer misalignment in smaller is the RMS noise slight correlation in are the and noise discrepancy pixel, low-frequency nal, 0.94. of integrated a and sources signal, of Possible 0.87 microscope 0.999. 1/20th between and varies than 0.95 signals larger in two motions video the each For in between eight piece correlation into each the signal of motion correlations video’s the each S4 plot divide laser and we the pieces aver- time, motion’s equal from over the velocities varies Because integrated size displacement). (RMS age the size and motion estimate vs. vibrometer motion our of tion correlation a and S4 aligned (Fig. well signals modes motion 0.997. higher two of with the well, plot video remarkably we each agree mm), from 85 490 signals length, (exposure, video motion focal one The For noise aligned. signal. manually low-frequency vibrometer were reduce to integrated signal Hz the 2.5 vibrometer in above integration, laser high-passed Before camera were The to integration. signals to trapezoidal rotation. filter both corresponded discrete, fan’s band-stop that using cooling temporal integrated Hz accelerome- its was 80 a the by and applied of caused Hz We side motions 67 1A). left between (Fig. the motions on video remove point the red from marked, ter the of placement .Tamn 19)Clae facsoysrcue fogno Corti. of organ of structures accessory of Collagen (1993) I Thalmann (2012) local 9. from RR velocity archi- Fay image P, flexible component Dallos of A Computation 8. (1990) pyramid: AD Jepson steerable DJ, Fleet The 7. (1995) WT Freeman EP, Simoncelli 6. fast for pyramid (2005) Riesz J Nakamura (2014) WT 5. Freeman F, Durand M, Rubinstein motion video N, Phase-based Wadhwa (2013) WT 4. Freeman F, Durand M, the Rubinstein in N, changes subtle Wadhwa revealing for 3. magnification video Eulerian magnification. (2012) Motion al. (2005) et HY, EH Wu Adelson F, 2. Durand WT, Freeman A, Torralba C, Liu 1. emnmz h olwn betv function: objective following the minimize We inte- trapezoidal using integrated doubly were data accelerometer The sepce,creainicesswt oa eghadectto mag- excitation and length focal with increases correlation expected, As correla- the plot we microscope, motion the of sensitivity the show To dis- horizontal the compute to method estimation motion our used We 67:1679–1685. Res 8. Vol York), New Science, (Springer information. computation. phase derivative multi-scale 447. for tecture FL). Raton, Boca Photography magnification. video phase-based processing. world. Graph Trans ACM G E and and 29:191–201. C rn Graph Trans ACM H F 0t fapixel, a of 1/100th of order the on displacements RMS For . .Tecreainas nrae ihepsr,bcuevideos because exposure, with increases also correlation The ). C rn Graph Trans ACM × Is lcrEeto n,NwYr) p1–10. pp York), New Eng, Electron Electr (Inst X 0.Tecnrlsa ftebig itdt accommodate to lifted bridge the of span central The 600. x 24:519–526. λ (V mg esr n inlPoesn o iia tl Cameras Still Digital for Processing Signal and Sensors Image n optVis Comput J Int S S y∈N (x) X 31:1–8. − h Cochlea, The (x) h rdewsfimdwt oohoePoint monochrome a with filmed was bridge The V(x))Σ (V ntxueesrgos tmyntb osbeto possible be not may it regions, textureless In 32:80. S (x) EEItrainlCneec nComputational on Conference International IEEE C − V −1 5:77–104. and V (x)(V S pigrHnbo fAdtr Research Auditory of Handbook Springer (y))(V ohtemto irsoedis- microscope motion the both D, S (x) S (x) − ). V(x)) − n mg Proc Image J Int V S T (y)) ;ectto,2;and 25; excitation, µs; + T , cutScAm Soc Acoust J onc Tissue Connect .They B–D). 3:444– (CRC, Fig. Fig. [6] 9 hlfA hua ,Bnhbn ,DaaiRuaiB ad 20)Giigand Guiding (2004) V Laude B, and Djafari-Rouhani S, materials Benchabane A, phononic Choujaa of A, Khelif Dynamics 19. (2014) M Ruzzene MJ, Leamy MI, under structures of Hussein analysis and 18. testing A Modal (1999) frequency: H in Auweraer der changes van L, through Hermans damage 17. structural of Detection (1997) O Salawu 16. (1995) DJ Ewins trav- membrane Tectorial 15. (2010) DM Freeman GP, controls Richardson AJ, Porosity Aranyosi (2014) R, DM Ghaffari Freeman GP, 14. Richardson S, Farrahi R, Ghaffari JB, Sellon traveling 13. propagating Longitudinally (2007) DM Freeman AJ, Aranyosi R, Ghaffari mechan- of 12. spread Longitudinal (2015) DM Freeman R, Ghaffari S, Farrahi JB, Sellon 11. ueo h os ee ( level noise mea- ground-truth due beam. the a entirely us the of are gives motions of covariance sure sample (Fig. these video temporal band, Hz their a this and 700 noise, in from and to stationary motions Hz is in-band 600 beam the the between accelerometer Because motions estimated the no then used had We We beam S9B). S9A). the (Fig. that verify beam to a Data. to Video attached Real accelerometer with Analysis Noise of halfway Validation a located given metamaterial. node the a was of to bottom metamaterial and applied top the was the frequency of between forcing loading bottom the displacement at The sinusoidal condition A (24). condition. al. boundary et zero-displacement Wang in ified 10 ldtecpe oewt ha ouu 4.78 modulus shear with core 1.33 copper the mod- 7.39 shear eled C10 modulus with parameters Neo-Hookean, bulk as (Abaqus element kPa, and rubber (Abaqus nodes 443.4 the 37,660 elements modeled ulus with We quadrilateral model CPE8H). strain 2D type plane a eight-node constructed metamaterial’s We 11,809 the forcing. simulate to to analyzer, response finite-element commercial a (29), Metamaterial. Acoustic of Analysis Element Finite covari- large arbitrarily the an pixels). be square of to 10,000 pixels components used square (we set 0.1 number than also larger were We that vector. matrix ance motion with each field of motion amplitude estimated the ensure to to cessing seeks term second fields. motion The similar pixel. have each pixels adjacent at that noise that motion of ensure similar to amount have seeks expected pixels term adjacent first matching making The of vs. fields. importance field motion relative estimated the the specifies that constant user-specified Σ where cec onainGat G-111 n G-127,adNational and R01-DC00238. CGV-1122374, Grant and Health National of CGV-1111415 Computer, This Institutes Quanta bridge. Grants Research, Trans- lift Shell Foundation by Portsmouth of Science part, the Department in filming supported, Hampshire with was work New assistance their and for Hampshire portation New of University ACKNOWLEDGMENTS. corresponding points corners. at or only edges and variance, to least are of direction that the to sponding results S9 produced (Fig. The model simulation IID. noise signal-dependent model is the noise than noise accurate which less constant in much the model, noise of variance result constant simpler the than S9 (Fig. bars. error truth ground in the first shown match the are and closely which from matrices of covariance matrix parameters resulting specific covariance The the the video, the estimate of to frame model noise dependent V 7 In better performs model noise signal-dependent the that verify also We epoueteclroely yapyn h bv pro- above the applying by overlays color the produce we 4D, Fig. In edn faosi ae nhgl ofie hnnccytlwaveguides. crystal phononic confined Lett highly in waves acoustic of bending outlook. future and progress, recent 66:040802. origins, Historical structures: applications. Industrial conditions: operational review. 6. Vol UK) Baldock, Stud, mutants. tectb in hearing abnormal underlie waves eling waves. traveling membrane tectorial in excitation of spread membrane. tectorial 16515. mammalian the of waves waves. traveling membrane 112:12968–12973. tectorial through excitation ical siscovariance, its is P,D1 kPa, F × i.S9 Fig. ,soigta u iuaincnacrtl siaenielvland level noise estimate accurately can simulation our that showing ), 8 84:4400–4402. V S P,addniy890kg·m 8,960 density and kPa, n Struct Eng stedsrditroae field, interpolated desired the is = eol hwtecmoeto h oainemti corre- matrix covariance the of component the show only we , 1.5 niern yaisSre (Res Series Dynamics Engineering Practice, and Theory Testing: Modal × 19:718–723. PNAS 10 N −11 stefu-ie egbrodof neighborhood four-pixel the is (x) etakPoesrEi eladTai dm at Adams Travis and Bell Erin Professor thank We i.S9C Fig. = | Pa P,D1 kPa, 221.7 coe 1 2017 31, October −1 emtyadmtra rprisaespec- are properties material and Geometry . .W sdorsmlto ihasignal- a with simulation our used We ). × 3 10 Aau aaeesC10 parameters (Abaqus 5 V = ehSsSga Process Signal Sys Mech V S P,addniy100kg·m 1,050 density and kPa, rcNt cdSiUSA Sci Acad Natl Proc λ scoeto close is steetmtdmto field, motion estimated the is 2.71 | S = o.114 vol. × 0 n hntkn the taking then and 300 × eueAbaqus/Standard use We 10 a Commun Nat 10 eto ie fan of video a took We Biophys J 7 −9 rcNt cdSiUSA Sci Acad Natl Proc egtdb the by weighted V, P,bl modulus bulk kPa, | Pa o 44 no. −1 x plMc Rev Mech Appl 106:1406–1413. and , .W mod- We ). 13:193–216. 1:96. 104:16510– G = i.S9D. Fig. plPhys Appl | and 2.39 λ 11643 S sa is H × ). E 3

COMPUTER SCIENCES 20. Cummer S, Schurig D (2007) One path to acoustic cloaking. New J Phy 9:45. 29. Hibbett, Karlsson, Sorensen (1998) ABAQUS/Standard: User’s Manual (Hibbitt, 21. Spadoni A, Daraio C (2010) Generation and control of sound bullets with a nonlinear Karlsson & Sorensen, Pawtucket, RI) Vol 1. acoustic lens. Proc Natl Acad Sci USA 107:7230–7234. 30. Blaber J, Adair B, Antoniou A (2015) Ncorr: Open-source 2D digital image correlation 22. Elser D, et al. (2006) Reduction of guided acoustic wave Brillouin scattering in pho- MATLAB software. Exp Mech 55:1105–1122. tonic crystal fibers. Phys Rev Lett 97:133901. 31. Xu J, Moussawi A, Gras R, Lubineau G (2015) Using image gradients to improve 23. Jeong S, Ruzzene M (2005) Experimental analysis of wave propagation in periodic robustness of digital image correlation to non-uniform illumination: Effects of grid-like structures. Proc SPIE 5760:518–525. weighting and normalization choices. Exp Mech 55:963–979. 24. Wang P, Casadei F, Shan S, Weaver JC, Bertoldi K (2014) Harnessing buckling 32. Unser M (1999) Splines: A perfect fit for signal and image processing. Signal Process to design tunable locally resonant acoustic metamaterials. Phys Rev Lett 113: Mag 16:22–38. 014301. 33. Fleet DJ (1992) Measurement of Image Velocity (Kluwer Acad, Norwell, MA). 25. Portilla J, Simoncelli EP (2000) A parametric texture model based on joint statistics of 34. Hasinoff SW, Durand F, Freeman WT (2010) Noise-optimal capture for high dynamic complex wavelet coefficients. Int J Comput Vis 40:49–70. range photography. 2010 IEEE Computer Society Conference on Computer Vision and 26. Liu C, Freeman WT, Szeliski R, Kang SB (2006) Noise estimation from a single image. Pattern Recognition (Inst Electr Electron Eng, New York), pp 553–560. 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition 35. Lucas BD, Kanade T (1981) An iterative image registration technique with an applica- (Inst Electr Electron Eng, New York), pp 901–908. tion to stereo vision. Int Joint Conf Artif Intell 81:674–679. 27. Oppenheim AV, Schafer RW (2010) Discrete-Time Signal Processing (Prentice Hall, 36. Horn B, Schunck B (1981) Determining optical flow. Artif Intell 17:185–203. New York). 37. Wadhwa N, et al. (2016) Eulerian video magnification and analysis. Commun ACM 28. Durst F, Melling A, Whitelaw JH (1976) Principles and Practice of Laser-Doppler 60:87–95. Anemometry. NASA STI/Recon Technical Report A (NASA, Washington, DC), 38. Wachel JC, Morton SJ, Atkins KE (1990) Piping vibration analysis. Proceedings of the Vol 76. 19th Turbomachinery Symposium, pp 119–134.

11644 | www.pnas.org/cgi/doi/10.1073/pnas.1703715114 Wadhwa et al. Downloaded by guest on September 27, 2021