Revista Mexicana de Astronomía y Astrofísica ISSN: 0185-1101 [email protected] Instituto de Astronomía México

Watson, Alan M. A Lossy Method for Compressing Raw CCD Images Revista Mexicana de Astronomía y Astrofísica, vol. 38, núm. 2, octubre, 2002, pp. 233-249 Instituto de Astronomía Distrito Federal, México

Available in: http://www.redalyc.org/articulo.oa?id=57138212

How to cite Complete issue Scientific Information System More information about this article Network of Scientific Journals from Latin America, the Caribbean, Spain and Portugal Journal's homepage in redalyc.org Non-profit academic project, developed under the open access initiative © Copyright 2002: Instituto de Astronomía, Universidad Nacional Autónoma de México is pro els curren R evista 192 (V duce Optical eillet t MB common-user Mexic h A uge in 1998); and LOSSY size. more pro images bzip2. roughly Key adv FITS tizan b pro RMS the los promedio it are del que to compresi´ garan CCDs im´ gunzip con con tiv o primidos. ana amoun y 1. imp suitable agenes infrared os eliminate v lossless vierten ducidas an fondo. difficult cuales maxim bzip2 el INTR a emen de Wor The como im´ t difference than tage files This Se osibles single ´ Univ ıas m or A o ts to Astr CCD agenes 1/5 ´ eto on bunzip2, ds: presen k comprimidas t largest similar en ODUCTION sobre of within no simply METHOD ey sobre-m um Esta El for pap for ersidad on 1/2 los compression instrumen los do onom p or tre sin of image lo TECHNIQUES: image or exceden de feature mosaic grado lossless w-order distributing imp use absolute er es their arc CCD. en of la ta b es p m la comprimir. curren et ´ describ ´ ıa devices. adecuado erdida a b hiv imagen lo ´ a eto las R ossible una uestrean un on y w Nacional diferencia from few e data. y standard original cual de een decompressing c os dos has of que ts El eive Astr compression. ra el m bits de mejora t tens cuan difference, the comprimidos w ´ no eto using the es suc l m con ´ common-user es ımite 12k d sin ning to F 1/5 comprimida of ´ The eto images. w The a that compressed 2002 do para una OR h metho ´ el ısic compressed compress. tizaci´ of Aut´ size lossy herramien × Instituto routinely p El a do absoluta deviation del un ´ general-purp imp ´ erdida, ruido, para p te´ metho 8k a mosaic largest v o IMA ercen onoma , comprimir m with es June orico en Alan v v tama COMPRESSING on 38 d alor metho ortan er-sample pix- ´ eto the ABSTRA ta The m RESUMEN with It is comprimir , se uy ja GE t no d a do a 233–249 app y no ˜ that y 3; p M. cam de exp The m´ of tas files. te de imp is quan arc in or elige metho and la d sencillo: aparen gunzip axima, prop ac es de the Astronom ears W v PR con ected for the imagen bia m´ hiv M de ose c ery compressed metho ortan im´ con las epte atson original infrared b ett of w tion quan CT the as tization ´ exico, para The orcionan theoretical eing os OCESSING compressing orkstation-class agenes resp m´ d uso that bac (2002) compression data simple: temen de im´ las or et d as mean noise, consisten p la de te to tities cuan ´ d erdida, agenes kground. 2002 degree original; unas ecto eliminar read al. bunzip2, general de Campus en bzip2 diferencia ´ some im´ ıa is FITS from images; suc detector te en 1/2 1998), lossy agenes la tizaci´ difference, con informaci´ a of images lossy A more decenas h las bruto. distribuci´ originales las limit. of ugust compresses of tly suc RA p de that tain data. descomprimi ra como tales ero but the on los im´ and This to Morelia, quan razones these w but h la compresses quan frequen en W RMS ols computers. agenes mosaic instrumen no con no images can problems desviaci´ prop bits 7 giv El de this garan on, mak Similarly bruto tization gzip is and suc information, v on tization CCD guaran es m cuando alue p b p a de orciona y ´ ´ erdida eto de e is or y tly h guaran de the es M dramatic comprimidas the t has con la pro o ´ a ´ son ıas endolos as on ba ´ exico . compresi´ cien do de up c datos significan bzip2. generated hanges ts quan images diferencia More tees is v jo duced , exp gzip IMA est´ implican Compression 2k follo se pro erted disp for dif to. can y compression tees c ciertas orden, andar hosen cuan- luego mak ected × ´ ıciles com- tized duce its w and and than osi- con im- easily 2k b on b on Se ed GES to to y y e smaller t b pixels y a o these few v is erwhelm can size a (Bec nigh large solu- 233 im- b k- ts y © Copyright 2002: Instituto de Astronomía, Universidad Nacional Autónoma de México w metho p ages; compressing pro differences with 234 comparison § pression as ries o arc mary migh or quences compression compressing the White (1994). sists lated to in ically tized the to-digital signal tially means bits standard these and images lo eac then ages Shannon’s ds; erformance ork ws 3 electrons, the follo lossy hiv h v This Compression T v case, (in briefly e cannot is us w o t . § particular the input alues of § to Gaussian images. 10–100. on es, uniform signal d; the ere illustrate lev b ws: 8 divided that 6 (1992), of to particular The e P dep sample pap of minim in discusses and § and lossy els oisson pure the deviation impro written 2. con calculate effectiv v first § b of 5 co suc summarizes b ending estigates to the er et the and noise LOSSLESS 2 ra of can ra e presen of v de the ev 2.1 the lo The w Ho h erter reviews w Gaussian white describ H w compressed. um in other compression; reference Press CCD read theorem w-order v een en this, v the the larger noise unit use; metho the . ed; to w images ariance b e data ≡ with motiv pixels. in space The e ev theorem the n lo bandwidth on the ts fixed-length the um gains − read is metho v h noise. images and cal er, series the lossy images noise, is § ariance ugely (1992), es ∀ results from whether the suitabilit and sampled with noises ds Shannon optimal X b p 9 ation just original CCD bits i the storage (Shannon er noise observ in and COMPRESSION of to p 6=0 noise, § discusses can The b erformance limitation d metho of of detail the 10 If eing in hcomp; states p con o a the the from most ha i in to presen v of § v 512 bits for images the asso BSCALE on b log and w er-sampled. estigates presen atories), difficulties to P e v compression compression tain 4 the ere y with an other and Shannon b signal devices. 1–5 e “input oisson Limit classified × 2 ds. the lossy y an a describ signal required of remote ciated that b p relev 1948ab, 512 V created. image ho read lossy een ts § i a no , compressed ´ the ideal eran electrons. of ts are 7 t It distribution w ypical a factor of similar × if compares and noise compression; an lossless discussed a information, of remote co quan some is is the noise 16-bit the often es metho with a metho en t brief observ CCD are de of & as q organized measured to stream 1949) If σ previous uncorre- the trop ratio The c metho metho is analog- titativ , lossless lossless of W enco hanges units”, this is conse- so essen- meth- larger quan- equal FITS d sum- com- righ d This data con- t 1 y new ato- im- im- yp- the the , (1) /q for for for al- b de W of of is d d y e t . A TSON q σ Shannon sisting corrections Fig. large but 16-bit b to This where co indep input 16-bit are (the pression b compression In tion, non fraction pressible for bits Romeo σ y e tuitiv de written original bzip2, and H Figure compression still 1. en Gazta gro ratio unit / and as equation enden pixels co 16. pixel. trop of p then The ely et ws i 1.5. de of correlated; limit ratio gzip, pure i lo their is as of , size) 1 al. . naga ˜ y the as t, unit w-order compression this and for sho F losslessly the the Not The a explicitly ratio rom the (1999) for H − lzop, Gaussian determined 16-bit as w compressed equation ws finite the (log is compressed et is as ≈ normalized optimal surprisingly of resulting optimal a this as the exp is log and function al. other deriv 2 a bits since a in compressed. q FITS a . q 2 w ) longest quan ected: function v (2001) / hcomp √ ratio As e noise estigated constan are 16. (3.3), compression ed lines gro can 2 input in optimal π b image can tized image of e y in ws size small , This frequency (ratio this deriv with − sho the the log one (used calculating ha in the of b co t log as to w e 2 v Gaussian The m with min case, q Shannon the de n e o q result e for standard compression seen, the 2 of for um − original y ccupied the limit that for losslessly). sho q of unit notation, log us , compressed Shannon solid whic b ratios a q optimal an the eac wn of er 2 the the (log BSCALE is at of is q image the h the of h individual line en not size) distribu- that deviation b simply small and ac least optimal optimal the 2 pixel y tropies incom- q hiev Shan- input is these is ) com- limit ratio new; / con- bits will size the the the 16. (2) ed as q of is a , © Copyright 2002: Instituto de Astronomía, Universidad Nacional Autónoma de México As § Arithmetic Cleary the co case. to Gaussian et is Gaussian w the metho pared is their o tigate including and significan dict images tion pression. of in demonstrates deep due go o signals, the successful orbit, losslessly to bits. o o sen comp It v v ds. ds ell 20.4) 17%–32% is de er-sampled er-sampled no o 1 al. in tativ indep addition q suc Huffman The W Real The d to this image Shannon frequency 1 lik v are onen and compression real = exp unit, alternativ Figure e and estigate (2001) This 2.2 The the compression d. ely h, actual e to are 1987; statemen are results 60), of and enden suc t, often Shannon . osures, Ho if calculated tly astronomical that small the noise lossless the noise Nev to those is on the R and metho forced on exercise the 16-bit in w h co e to o co it sho ev 1). and al b Arithmetic gain limit implemen images ccupied as Press four full ertheless, none ho ding, of frequency almost b t distribution noise. etter the as er, pixels more ding t. compression e is y noise, b L is w of White w with and biases oth F but ossless suc ds range These a consisten to normally imp WFPC2 WFPC2 pixel compression ratios v or that w of futilit (pro limit ery an h factor also hcomp, (Huffman ratios et explicitly considered ell than the of to successful this infrared their with to ossible their p y b high but & F of WFPC2 al. erfect authors vided y they nearly tations Huffman v in images or or use particular conclusion it y Compr explains alues Bec calculated table). ra white is co tro images reason, images images half 1992, under-samples sample high w of short tly is nev of for of example, bac ding k bzip2, lossy a p ratio duce astronomical er to the useful agreemen 60 erform fixed images. attempting ac 1952; for ertheless optimal are compress and theoretical its kground of noise (1998) images ession metho signals, are hiev con on using compress § has ha (whic co exp of (Whitten, image Indeed, wh compression the w these these should tak lo 20.5) size; v gzip, ding that e e images input real tain images e Huffman w using Press a to osures, y in compression in need lossless en app lo Gaussian if ds large sky Metho h lossless COMPRESSING for t. w consider the this that the metho suc metho is to this if of the lev compression and images. mak as ra ear corresp information p bac brigh b co Gazta et is o large w to this erform q the w not limit quan white h e lo els, read the with e idealized de not Rice to kgrounds, ≤ m al. Neal, and ds e noise exercise lzop. w-order close as images, con ac co wish tness ds ds uc con use meth- image meth- 1 in units. equa- input 1992, there hiev repre- noise ding, flats, tized com- ratios noise. naga ˜ onds noise h com- ho tain (see v and and less but lo tra- es- on to to of of & w w in is e RA trend.) lossy will Shannon 1992). non of compress is ing compression high the in hcomp to ra tios, spatial a tures compression quire their the parts image ered high-order ing F sp tization differen and the Hcomp cien lzop instead are plex rithms the ing general v or W 3. w eed, oid bzip2 a main the Louys Fig. Bzip2 Hcomp p the lo ts, images; Shannon pixel fastest ra PRIOR b limit—they IMA executables sligh 16-bit images (Ob algorithm ortable, and b compression compression w significance. that e metho can in w y that with and Their lo with t dev (used b tain can can filtering, order Both tro b Louys 1 suggests of images, erh w-order GES t est yte trade-offs gzip limit. v the b do (Sew sho alues, exception et ducing oted the bits). quad-tree (White but con e astronomical umer appro bzip2 b a ds is W all compression v ratios ratio. streams in efficien con e limit. bzip2 imp bits—but ws losslessly), ariable-length large al. image b ORK tains somewhat co v et used ard based that eing ac of to estigation The also are w tain that optional efficien the ortan None bits) or 1998) ximate al. hieving t (1999) metho e spurious the of ypically ev Therefore, ha 1992) and p 1998), t, b m losslessly suppress freely no in ON other aluating (1999) cannot ortion in information but et surprisingly reduced compression and v compression on it ust and termediate information e t, w and of information are the hcomp that migh ts. the ds, a een bzip2, LOSSY their as ratios, is then consider differen the ha images use quan the free concen w a b pro gzip features. metho constan v sp preserv of including same can a v subsequen input compress eing ailable. t it compression v e only this dictionary-based w b for ecialized lossy images. b elet from compression the cessing tization y follo are is orst e recen gzip, giv lzop (Gailly trated omitting t, slo useful COMPRESSION (the ds ra trend in initially go and of suggestion. ratio transformation co rest other those in quite e t e ws paten w metho in w (the compression o for b the is de Ev They tly p this t and d est the hcomp oth mid-order ypically images white erfectly t ma that larger. the uses of Compressing to of en p in as on compressing units. ac co-addition, co metho close white 1993), parts but erformance ts. scrutinized guaran image y this sp ds lzop. the compress- compress- hiev the the ratio unable w relativ efficien represen exp ratios w increase eed eak noise Sources a (White consid- ac e w co to (Lzop Shan- pap quan- ed b those ected They of ds. noise hiev- algo- com- e wish eing The and and and and and effi- fea- tee. 235 the ely ra- re- b ts. b an to er in of y y t © Copyright 2002: Instituto de Astronomía, Universidad Nacional Autónoma de México (or as to ple (1999) has compression consists (2001) b Nieto-San Gazta in man pression et the co dard card carding noise lossless pression will is lated noise and exp NGST quan field sen Gazta data 236 sim extension images most quan pro y v to § quan pro ted al. er a duce ulated Comparisons The Eac More The erimen that resampling) a n Gazta normally tization tization 7; 1952; from white um reduce (1999), “metho the to and common deviation. completely in naga naga ˜ ˜ duce (1999) and lossy is h data (W compression tizing the quan first white a b of the it algorithm a metho in v quan promising tisteban er to ts naga ˜ data ery ells, con data CCD-lik teger see is et rev et quan Nieto-San noise a metho compression b b of of other the tization stage bac d”. metho is somewhat Nieto-San v b al. eac y y al. quan tized similar ersing discard Press noise erting Greisen, e the format d. et v bits quan from describ scaling tized kground. in alue n b (2001). a o to (2001) h 4.1 from differen al. v White d It = e image ro et sp tized of in quan er-sampling 4. White brigh d image. (Rice, the . is of metho devices. and tizing suggested et w b (2001) ecified the tisteban compression consists cosmic b the is the Quantization to to zero al. METHOD image. the generic for b ed white white b al. tisteban and w & tized in use y y em the tness in image, t These lossless then metho formats (1999) ork quan original + & tended a in Harten con Y tegers, 1992, & w d arc The b The barrassing fraction discard Huffmann eh, nb fraction y taking ork one to micro Greenfield terms of image. Greenfield noise, text. hiv et noise a of scale tization of using floating-p F b authors resampling p ds of aim fraction & White or follo of y § al. suggested et use ed erform compression lossy the to is Nieto-San FITS 1981). metho 20.4). , w that the Gazta Miller White definiteness, white the (1999) of of al. ob a ra from and b w of of a Decompression the v original e ed w e the to vious. co noise, in metho lossless & the quan quan w used ha square (1999), image b bac oin of d (1999) teger naga (1999) ˜ ding data. p ork This describ b Rice etter. noise Greenfield 1993) v estimated sim ermit & is y are here, quan the exp e t kground tisteban tization tization lossless all so on b images Green- d reduce ulated (Huff- image to y is et made FITS ected is stan- com- com- ro from is sim- tum pre- The The and and and ra dis- dis- but e the the the (3) re- re- ot, al. to W w it A TSON defaults where and mine b this data Here, brigh ancillary whic are engineering electrons, whose out larly to rounded Here b It file applying prior im in Certain or or Th BZER biases. noise not devices, tum data e σ um can the us, mo the an The is b treated Q en c , information h BSCALE as hanged. tness = or g in to as is the b [ related de O tirely the ha and in x b FITS quan b noise is = The ( Q ] quan zero giv e the implicitly v tegral these compression.) of the v and is Equation information, to in alues the n bias ( seen e quan and (( r 0 minim ˆ b en the = the accordingly tized /g quan image, certain in the information, and b and estimated is b b tum header) mo ˆ b BSCALE gain to scale scale ( The − b v records lev b = that m th in tized nearest in y alues v n ˆ bias del b alue the nearest tized ultiple should 1. b 1 teger us um b b bias Q el = v b q the scale zero in n v alue blank ˆ q (7). σ b p The alues σ regions, is this ( b for b quan in = ) erhaps should ˆ b are bias sp electrons, in b /b are g original /b + image suc the − teger arithmetic in in . Q teger + are standard ecified W scale v w the b Q data scale Q quan alues Th b often teger n ev pixels. and not ould tized h e the r corresp e zero enco n [ b bias 2 n/Q en then transferred [ as c us, tak ) b b/Q from of n detector. c ˆ 1 and is v ) e quan tization / c FITS alues data /b the in b image 2 to as hanged ded used in b written q q en b determined data “blank” n lev r y /g q q σ σ scale b e scale determine ] teger σ σ the ] onding (Some . deviation x b b hence is the the b the as exp b b the el. from tum scale > ≤ with in bac > ≤ the are header to . . b b v quan BLANK are b b alue original osure mean, W The ≤ > can b b blank “data” scale scale . and flag to with F kground Q to scale scale treated read pixels e the or the images half b b to b sp bias bias tized first help the ˆ b is b quan according saturated the CCD-lik an and the σ ecial time e the . noise b BZER c median, in b noise v , y hose carried y header alue image. pixels; record (those . tegers of deter- quan- a FITS max- same simi- pixel tized ha ha lev v and and the oid (6) (4) (7) (5) (9) (8) b v v to in O el is y e e e © Copyright 2002: Instituto de Astronomía, Universidad Nacional Autónoma de México Th Here quan to than of to en o noises. the it is differen (see and FITS tized on ered. 1998), this the ra dence original The bunzip2, Huffman the and tion quan determined 16-bit pression v w t er mak imp an b 2 us, There The The One W the quan quan e noise quan data. can are § metho the (see tization tized hcomp metho 5. image q FITS e b the file ortan required 7). the es and x gzip times t define distribution DISTRIBUTION c quan F adv not quan en ta FITS in tized stage tization quan sense § gunzip, urthermore, b smallest or is v is parameter Note and tro tire y d 5 as ary 32-bit ds t file should an only no (Gailly determined rev the is (White and Arithmetic tized the tized b the duce they ta tage and the y image. is in original file. that requiremen to using to significan ersing that 4.3 v setting within b lossless. largest quan lzop, 4.2 noise. § metho estigated sub enco y apply in normalized probably lik small image original FITS of 6). . the w ∆ . 1993), teger Therefore, the of De q ely a 1992). divide e the Compr ta The the if ders = or con images. lossless can c quan the eac d Suitable the the distribution on ompr ha tly are but in the enco file ˆ b is hdecomp. simplicit m OF lossless t trols is data σ teger − v h data deriv lzop (whic that a here ultiple b reco b differences e the size within quan ession constan tization to nev bac subregion. is b e ession ders) pixel-to-pixel differen Hyp DIFFERENCES difference quan compression . These the and no the compress v data e v ertheless the kground (Ob no of h v are ered tization alues useful compression alues y othetical of will smaller coarseness the the are tized o of t second greater quan erh stage; of v t bzip2 region b within giv erscan as COMPRESSING scale c the to differences quan appro b haracteristic umer for data also information Ho et e ∆ with ta a metho and real b the the quan w no standard than e the q w basis, b metho (Sew b stage and tum optimal een than regions regions ev consid- e are region, ximate et 1998), differ- biases of larger quan- hence confi- equal using w er, com- tiza- d (10) ard een the use the the 0.5 Q to as x d. of it is b . RA appro so uniformly ∆ q deep galaxy), and corresp The The nebula), 512 standard A exp Th tained small). whic the et en Nacional metho and the reference. with timal distribution of distribution tribution W is t firm compression (the us, al. w This Fiv osure × noise “digital assuming IMA h p not e images quan b exp ximately erformance 16-bit d comes 512 (2001). no e exp with (whic onding upp mean prop trial larger GES section and (d) osure of at tit difference regimes of deviations × ect distributed er σ σ differences, a y h w of with 16-bit ˆ b b Sierra a distortion” from and a osed σ (e) and images CCD b ere that σ is the ≈ | ∆ to sparse standard They ound deep ∆ 6. ∆ of and than a presen r ≈ a | 6.1 32-bit a in (a) adding corresp RMS PERF and of here, go a ≤ mean the flat-field 1 FITS at µ standard teger San √ is decompression. . relativ ∆ exp o on + 2, sho in 2 w q the a Q 12 T d the ab field σ larger difference scenes. ere ≈ ts rial b compression 12 bias q b normalized appro w qcomp, the the osure P ORMANCE enco w deviation out D 2 ≈ and the onds files 0 m ≤ e Observ edro quan tests selected ≈ ely that 0 ultiples , (a discussed exp Images 0 magnitude exp exp . bac zero v ders in 29 . 1 deviation ximation 5 standard ariances with of standard to + tization q The sparse M´ ect q magnitude osure, and osure. kground. . in a ˆ b atorio 0 . artir a (on the − serv . quan σ 042 terms ∆ to a of complex difference) images b b ratio, Hcomp comparisons is BSCALE discrete will b square-ro to e b sample field q in (b) and y σ pro T scale Astron´ tization of uncorrelated deviation compression as , star), ˆ b o quadrature b Gazta of result ∆ and giv e vided a mak p Pro /σ than w sa (a and the oin roughly shallo is field ere en v b p differ- omico sp ) vided ed small to e (c) oin ot naga ˜ ts of (13) (11) (12) of b in and dis- 237 q q eed ob- the op- y b / as (a of of of of 1. ts w is a a 2 a e © Copyright 2002: Instituto de Astronomía, Universidad Nacional Autónoma de México 238 Fig. v deviation histograms the b alues oth noise 2. at and lo The in w of the in image frequencies the trial the dashed pixel images. in righ image v line t alues and panels is The (d) on the (after a b are images exp ecause pixel-to-pixel 5 sho ected σ clipping). wn in of con structure the stretc tribution left basis. hed In panels the in from W from the A histograms, TSON are − scene noise. 4 σ ˆ stretc b and to The hed +4 the in σ ˆ real image from b solid ab distributions out − line (e) 2 σ ˆ the b b is ecause to the mo +6 de, observ are σ ˆ of b where significan structure ab ed out distribution σ ˆ the b tly is in mo the broader the de flat standard and of field, pixel than the © Copyright 2002: Instituto de Astronomía, Universidad Nacional Autónoma de México comparison t is w plying w the o pixel (large-scale what ture a ping). quency (d) histograms bution other inated of only trial images sion set regimes pression metho w v ere ere 2. 1. successful erscan een m the The An The of is pixel uc (b) (c) (e) (d) (a) Image Lossless Lossless culating 16-bit tios option. metho images, in a estimated v remo arbitrary sets ds: y the m trial h ariations pixel Figure equation b to go of is images the and uc y bias c wider ratios for regions, hoice v o the data h v noise. alues that of d eac and d of images ed, structure with v these scene wider compression scenes. compression can mean Description Bias Deep Flat Shallo Deep compression it alues, and the images, 2 h histogram the . b for 32-bit of sho and w is as of from they t ecause w (7). yp ere 6.2 pro Ho and hcomp Shannon TEST Ho field lik as trial images bac metho around eac ws and sparse complex hcomp than w but compression e along o . w ely T This w duce compressed also v sparse the metho kground h Compr of The the co ev ABLE of the regardless erscan ev images the of individual also v er, to er, the ds image. IMA width around er fair, using images determined (a), field metho mo implies b noise go with 0.5%). structure do standard pro y en w field one ds. histogram the a ession field consisten o noise de 1 es sections. hcomp ere GES trop d (b), the duce go lev is h from the The results σ d histogram in not yp ratio adv and 1.5% using of determined Th o b lik els that y d m image. o b and estimated othetical the 11.3 14.1 76.0 exp v the us, go ely ecause an for ust compression σ in 2.8 2.5 deviation with distinguish erscan range the the in tly b (with for COMPRESSING o tage if and the of ected for (c) exp the d to eac w relativ ac eac a histograms noise. go e results the the image 532.0 hiev all b osed are compres- follo h of of 17.4 94.1 flat pixel-to- o see h of of sections 5 e σ optimal ˆ 4.0 3.3 d b σ con image, b “-s of image image whole some- struc- y e e y these dom- σ noise ˆ com- wing clip- that field The b and cal- fre- not the tri- ap- b (e) ra- for 0” of e- RA v not Figures ground with the whic sho malized tios, ing noise pression pressed. are pression o ceeds original ences (9% pression the im pression slo sion 1.47 1.94 alues ds, W 5. 3. 4. um w w The The Figure not compressed lossless er), these include IMA h Lossy Lossless Lossy hcomp quan compression, metho are giv man tion. program with giv bzip2, of “-1” metho as Mpixel/s Mpixel/s a slo and the in are at absolute the of single loss es en giv these w 3 distributions lev 4.03 compression images. difference h the These ratio ratio GES ratio sp 0.5, ab tization er), (fastest/w and RMS, and yp sp uniformly roughly The b h en using compression 4 fiv d ds. el gzip, eeds b y yp out used metho eeds compression the othetical, sho eing image; Mpixel/s e 1.9 describ 1, for metho equation compression and Arithmetic 2.84 othetical qcomp 4. for ac for for for images v sp differences 2, ws size 11 alue maxim time hiev to GHz and the the for The are eeds con metho hcomp [The the 4, all qcomp/hcomp ds the calculate bzip2, q ∆ orst) Mpixel/s Mpixel/s ds 0.79 to , ed see optimal the ed 8, zero. trolled lzop of mean, sho of do the fiv sp lossy P tak (a b more w w for are (1). original they and same um using en et in for normalization this ere eeds d e ere v differences, Equation optimal actual wn Mpixel/s to not en ailable (18% w RMS tium co with qcomp/lzop with images, h using gzip, § b een as absolute, metho T eac 16. Although measured yp “-9” ding. compressed effectiv y b to option will median, in 16-bit able 4 hcomp, the RMS app quoted y and for functions othetical. h direct IV options slo size), implemen the T compression determine the v difference, T alue image (slo able ear ables from lzop, appro 16-bit noise. (13% ds qcomp/gzip w 3 the pro reduces (10).] and for difference same and e er) w compressed sho compression w are with and the as as lossy in of 2 cessor. these on application est/b or (37% is qcomp/bzip2 ximate for v 2 the the and ws quan (the 32-bit slo and the 2 of and arying their sho T a ted and v mean mo q and Ob to mean Determin- the able alue w σ the and q computer the the the compres- est). sp metho Figure wn b author), er), “-s” = slo tization stage. de ratio viously σ , decom- lossless Sp b degree eed as b differ- 3 meth- 32-bit whic y bac 1. (26% Huff- max- , w of com- com- com- com- 3 from for eeds nor- pro- and and and 239 er), op- the the the ra- do q ds at k- of of . h 3 q , © Copyright 2002: Instituto de Astronomía, Universidad Nacional Autónoma de México 240 Fig. relations of q for 3. The the predicted four RMS, images maxim for qcomp (a)–(e) um absolute, in for § 5. qcomp and (dashed mean differences lines) W A TSON and b et hcomp w een the (dotted-dashed compressed and lines). original The solid images lines as functions sho w the © Copyright 2002: Instituto de Astronomía, Universidad Nacional Autónoma de México histogram this bias due to-pixel bit ter noise. sion b bit ter and the o that that noise of cause pixel its whic (e), ages, This individually ds etter ab The In metho than input metho 16 8 4 2 1 0.5 q mean are to optimal the is image, h there ratios b out in also (a) discussed and oth con its considers that the The than hcomp, the optimal upp correlations hcomp 0.4. and qcomp qcomp qcomp qcomp qcomp qcomp Metho hcomp hcomp hcomp hcomp hcomp hcomp ds, trast w co d. explains compression hcomp bias bzip2 are of for optimal a optimal whic . deep er de the v 16-bit whic Ho (b), elets Ho pixel the section correlations structure. to 6.3 d bzip2, in unit and h 16-bit optimal w pixel w b and and h suggests the ecause § the . ev lossless can wh 0.00/0.00/+0.00 3.34/8.00/+3.06 2.97/4.00/+1.74 1.14/2.00/ 0.56/0.80/+0.02 0.29/0.40/ 1.04/4.00/ 1.19/7.60/ 1.16/6.40/ 0.17/0.40/+0.05 0.66/2.80/+0.03 0.42/1.60/ can v ac 16-bit L 2, metho 32-bit er, bzip2, alues ev to ossless hiev flat-field in and compression y noise bzip2 hcomp of metho en this ratios extend in co the pairs, 16-bit T (a) it (Another e in v d, compression the metho able that metho whereas the just er mean b comp b can can optimal Metho this − − − − − − d et whic e? and optimal than more 0.03 0.00 0.07 0.15 0.16 0.06 p w bias p images, there 3 metho o as RMS, erforms erforms exploit—hcomp een increase ds. v The d image, onen giv compression h er bzip2 for ds DISTRIBUTIONS of p w the and the considers than es erforms The adjacen more a 32-bit are 0.00/0.00/+0.00 2.95/7.86/+2.61 1.20/1.79/ 0.25/0.36/ 2.67/3.93/+1.85 0.50/0.71/+0.01 1.02/4.64/ 1.23/13.9/ 1.15/7.50/ 0.15/0.36/+0.04 0.66/2.86/ 0.38/1.43/ pure pure d explanation Maxim t y 16-bit COMPRESSING the uses (c), is of other ev ac sligh optimal shallo presumably b small b the dominan etter one en lo hiev est than Gaussian Gaussian compres- (b) (d), metho only oking um t and tly sligh for size w meth- meth- pixels e pixels ratios pixel- pixel. − − − − − − − than Absolute, b b 0.25 0.00 0.07 0.14 0.17 0.03 0.05 and one im- the the b 16- 32- tly et- et- T at d, of e- is t, ABLE OF 0.12/0.18/+0.00 6.85/7.96/+1.47 1.06/1.95/ 0.28/0.44/+0.00 1.40/3.98/ 0.56/0.97/+0.00 0.97/3.89/ 1.31/16.7/ 1.13/7.52/ 0.58/2.21/+0.00 0.16/0.71/ 0.30/1.24/+0.01 RA and DIFFERENCES b not pression bzip, significan o duce if In ratios, press dominated ages to ages tions will erately dominated o ab images images, compression.) the pressed compression bias 2 v ds W et w er-sampled. out 0.4–0.7. (c) all, w Mean (T Nev e use image; compress IMA m een compress and in w inconsisten some able b gzip, − − − − − − uc 0.4 w these an et ertheless, o compress 0.12 0.29 0.02 0.01 0.08 0.00 w b e (c), lo v h sp GES w Differences the t oth t ell er-sampled, shallo ha w-bac masks een 3 impro to eed sets b This b impro the and v to y y (d), also results ratio the e “-1” ac others 0.14/0.21/ 4.95/7.94/ 1.15/1.99/+0.00 0.29/0.50/ 2.29/3.97/+0.08 0.57/0.99/+0.00 1.01/4.33/ 1.77/16.7/ 1.34/8.16/ 0.16/0.64/+0.01 0.58/2.34/ 0.30/1.13/ deep observ badly ab pixels, no drops w of In hiev image kground t is lzop. v v histogram out v w and sho emen c data results, images, the emen ariations as and short, hoice ell for ∆ images e confirm (d) v ws ations as 0.25 exp noticeably consisten but ery (e), as in and adds t, mean individual v F t “-9” their − − − − − − − − the ery but or units ected; their but narro 0.00 1.01 0.00 0.02 0.05 0.02 0.01 0.00 p in the the and and b (a) o allo with of bzip2 and, eing orly w of information effect to what options, the noises the ell deep compression lossless of the standard w-band tly ws and noises while factors adopt . the 0.14/0.25/ 4.48/8.00/+0.41 1.15/2.00/+0.00 0.29/0.50/ 2.34/4.00/+0.02 0.58/1.00/+0.00 1.13/4.33/ 1.84/16.3/ 1.64/7.57/ 0.58/2.16/ 0.15/0.55/+0.00 0.29/1.09/ σ high (for (for to compressed F compression compression b deep go and images, for w or of and bias (b), v e are o arying example, example, they lossy the lzop are d bac already but more metho (e) of gzip filters), and flat-field compression significan stars and b on kgrounds). 2–3 only options − − − − − − − eing there will the metho with 0.00 0.00 0.00 0.02 0.00 0.00 0.00 flat-field ratio degrees, there efficien correla- ds shallo p or in knew: sp those those mo o com- com- com- ratio they pro- orly is 241 im- im- eed the the ds. tly d- to of w is a t © Copyright 2002: Instituto de Astronomía, Universidad Nacional Autónoma de México 242 16 8 4 2 1 0.5 0 q hcomp hcomp hcomp hcomp hcomp hcomp hcomp Metho qcomp/gzip qcomp/gzip qcomp/lzop qcomp/lzop lzop qcomp/hcomp qcomp/hcomp lzop qcomp/bzip2 qcomp/bzip2 qcomp/gzip qcomp/gzip qcomp/gzip qcomp/gzip lzop lzop qcomp/lzop qcomp/lzop qcomp/lzop qcomp/lzop qcomp/hcomp qcomp/hcomp qcomp/hcomp qcomp/hcomp qcomp/optimal-32 qcomp/optimal-32 lzop qcomp/bzip2 qcomp/bzip2 qcomp/bzip2 qcomp/bzip2 gzip bzip2 qcomp/optimal-16 bzip2 qcomp/optimal-16 gzip qcomp/optimal-32 qcomp/optimal-32 qcomp/optimal-32 qcomp/optimal-32 optimal-32 qcomp/optimal-16 qcomp/optimal-16 qcomp/optimal-16 qcomp/optimal-16 optimal-16 -9 -8 -7 -3 -1 -9 -1 -1 -9 d 0.005 0.226 0.010 0.233 0.047 0.057 0.092 0.156 0.052 0.064 0.099 0.163 0.011 0.256 0.217 0.221 0.247 0.239 0.002 0.243 0.060 0.070 0.108 0.176 0.007 0.330 0.007 0.499 0.225 0.232 0.120 0.137 0.186 0.259 0.250 0.285 0.373 0.442 0.235 0.004 0.242 0.241 0.387 0.386 0.383 0.498 0.498 0.296 0.005 0.035 0.098 0.167 0.253 0.329 (a) COMPRESSION 0.008 0.237 0.013 0.246 0.038 0.063 0.103 0.166 0.044 0.071 0.111 0.175 0.026 0.269 0.241 0.250 0.262 0.252 0.004 0.257 0.049 0.075 0.120 0.189 0.011 0.343 0.012 0.510 0.236 0.246 0.099 0.144 0.197 0.271 0.187 0.304 0.380 0.450 0.249 0.007 0.256 0.254 0.399 0.399 0.396 0.509 0.509 0.313 0.011 0.046 0.104 0.179 0.266 0.343 (b) Compression 0.068 0.235 0.077 0.254 0.021 0.059 0.114 0.168 0.033 0.073 0.130 0.186 0.403 0.389 0.088 0.327 0.364 0.395 0.072 0.246 0.015 0.063 0.120 0.176 0.140 0.341 0.291 0.506 0.372 0.395 0.030 0.122 0.204 0.268 0.038 0.212 0.378 0.447 0.234 0.015 0.393 0.388 0.562 0.562 0.567 0.709 0.709 0.488 0.020 0.049 0.108 0.172 0.391 0.502 (c) L L ossless T ossy W ABLE RA A 0.070 0.312 0.103 0.389 0.101 0.146 0.196 0.253 0.150 0.207 0.267 0.328 TSON 0.210 0.441 0.239 0.310 0.360 0.420 0.041 0.274 0.060 0.097 0.148 0.209 0.083 0.421 0.128 0.591 0.466 0.561 0.127 0.203 0.279 0.352 0.195 0.322 0.418 0.499 0.260 0.024 0.460 0.449 0.691 0.691 0.691 0.825 0.828 0.621 0.036 0.075 0.133 0.198 0.440 0.641 metho (d) TIOS metho Ratio 3 ds ds 0.051 0.313 0.072 0.369 0.094 0.139 0.193 0.253 0.128 0.185 0.246 0.307 0.159 0.576 0.295 0.357 0.456 0.526 0.030 0.280 0.060 0.101 0.154 0.214 0.061 0.448 0.105 0.633 0.525 0.692 0.122 0.185 0.292 0.373 0.226 0.346 0.421 0.519 0.262 0.007 0.634 0.620 0.876 0.876 0.877 0.991 0.992 0.766 0.020 0.080 0.133 0.198 0.591 0.769 AND (e) SPEEDS mean 0.040 0.265 0.055 0.298 0.060 0.093 0.140 0.199 0.081 0.120 0.171 0.232 0.162 0.386 0.216 0.293 0.338 0.366 0.030 0.260 0.049 0.081 0.130 0.193 0.060 0.377 0.109 0.548 0.365 0.425 0.100 0.158 0.232 0.305 0.179 0.294 0.394 0.471 0.248 0.011 0.397 0.390 0.583 0.583 0.583 0.706 0.707 0.497 0.018 0.057 0.115 0.183 0.388 0.517 Compression Mpixel/s 10.31 1.80 1.68 1.84 1.76 1.68 1.70 0.40 0.83 0.31 0.53 0.83 0.85 5.52 3.52 7.46 6.13 5.68 5.13 4.31 3.83 7.41 7.25 6.67 6.37 2.20 2.99 0.73 0.58 0.16 0.20 0.60 9.80 0.36 2.85 2.75 2.53 2.37 2.06 2.81 Decompression Mpixel/s 16.67 10.31 38.46 32.26 19.23 15.15 12.82 11.90 38.46 37.04 35.71 32.26 34.48 32.26 32.26 32.26 34.48 11.49 10.20 2.01 2.03 2.07 2.05 1.98 2.01 3.51 2.46 3.73 3.37 2.76 2.62 2.23 3.22 2.07 1.52 3.05 2.87 2.64 2.43 2.04 © Copyright 2002: Instituto de Astronomía, Universidad Nacional Autónoma de México q anomaly: drops giv tion metho Fig. w option those and in qcomp 0 roughly ference. larger zation of tions predicted close of to This First, qcomp . e = 5 a the giv the es q Mo T Comparison first σ 1, 4. mean sharply-p step able b to is a of e ds. for for , T ving lossless v precipitously The is maxim is should this compression and and alues not able zero, differences 0 in F to to q The qcomp. preferable, . when differences 2 or 3 in v RMS ≤ o q estigate result. the from surprising the hcomp and σ 3 larger § coarse as 2, of solid um b eak sho compression not , 5. p the the mean of and the migh q 6.4 erformance Figure ed , ws the absolute F the b line Ho the . for . bias v or RMS to e in maxim what maxim ratio distribution. that alues and for L t used w p difference It ossy distributions q b sho adequately as qcomp Figure mean erformance ev e image, hcomp ≤ 3 seems differences optimal for er, of exp ws the app um of sho um b difference 2, stage Metho 0.163, ey of qcomp the this q ected difference absolute ears hcomp the , ws are 3 ond the absolute (a), are the clear is Shannon rev b ds 16-bit reason This that RMS b sample y compression roughly to whereas m q when is eha of RMS of eals has are qcomp.) ≈ uc quan option and b differences that differences COMPRESSING the ving suggests h e 2. the ceases v compression lo difference difference t it b ery the a en w differences w b tized etter mean quan o zero. w oth w trop er equation the distribu- m similar. w as orrying quan trends. uc v as to alues y sides step, than with used tiza- that “-1” h as dif- F limit set for b ti- or as is is functions e for RA q as has ha imation (Neither ac 2 pression flat qcomp/bzip2, with v Q not of compression ably ence not of the con the tized true equation a the age sion q ≤ ery W of = indicates v b hiev v they the 0.134. 1 alue tributions Figure = e the . bias an shallo effect prop a with 0 top IMA 5. compression is ratio little due large Gaussian . Gaussian b 8. compression ed difference that scale mean in are of to image. b er teger By GES ratio qcomp/lzop b 2 to q w e The them 0.153, structure of y v b 4a significan that = comparison for the the q alues simply image structure compression equation qcomp/optimal-16, σ ratios 0.148, to 1 b m of sho explanation and /b finite noise noise noise histogram significan The optimal is ultiple whic the can scale ratio of ab ws compressing (b) ratio so Gaussian and and out σ qcomp/gzip tly remaining noise σ quan h c q with b b nor the 2, b in , ; should differen e for is is of in so to w n ratio 0.128. so 16-bit n this explained of the tly umeric m rounded orse.) umerical b qcomp/hcomp tized the Gaussian mean the (for q in compressing scale should uc 0.162. for . = w bias for h Figure noise, bias b t? truncation ould , compression example, 1 the e closer difference Gaussian with The The and the qcomp/optimal-32, exp compression should prop as Only and image to The do b exp remaining lossy ha noise y b so so erimen functions wn bias a 2a). q to erly est v corrections non-Gaussian erimen go = other the e a wh the the giv to is the compression a mean o are with small of noise is 0 image compared d y Similarly 2.5 . compres- quan ts e 2. 7, of bias presum- Q 0.163 appro are sligh sho a images ts giv b differ- whic and Th quan- ratios q of com- com- with do part tum giv = wn, 243 has im- the e us, tly to x- es of q is h 1 a e . , © Copyright 2002: Instituto de Astronomía, Universidad Nacional Autónoma de México maxim b tios or corresp qcomp pression els 244 20% RMS m image ten 0.19 F bzip2 absolute than F sion ab T hcomp q to tion w can smaller pression ences F Nieto-San compression, one although ter q metho pression tio prop qcomp/bzip2. or b roughly 7. y or or or etter able e ≤ um > o 32-bit lossy q that tly COMP bzip2, tly Appropriate A This v ha than compression q adv of 2 example, smaller 2. osed and e ≤ ratio, from t of lossless = difference app absolute is v at 3 ds that of um go the w than e is ond 2) qcomp/gzip an and are Th than sho differences), 1 that qcomp/bzip2, four orse ratios ratios ratios section difference. 0.13, prop a metho o considered, the significan the b ears tisteban and tage here, but ARISON and 0.176 d us, oth absolute and same giv ws not the to qcomp/bzip2, compression than hcomp nebula it compression compression 0.1 times 0.41 osed and en q to qcomp/gzip that q at in that that hcomp decompress of difference than are exploited ds, these compresses quan This = but qcomp, b to compares ratios b v (whic compression terms 0.13 figures et q alue 0.1 e 2 b it et tly ac esp ac 0.214 is, qcomp/bzip2 w hcomp finding y TO faster = This difference. ev the b tization lossless is (d) hiev een ac hiev al. qcomp/gzip2 METHODS qcomp/bzip2 app are White oth factors (whic w and ecially en is h b 1 b of hiev 0.31 orse of mean OTHER et to oth (1999). and tually b corresp b ed 0.5 for is the ears ed sho at ratios q in of est w y with then ratios ab the has has the at are correlations ab es not h the een either (i.e., & b b than compression. at a q wn and RMS the o b compared in of mean y y merit corresp ab out for compression v more ecomes ratio, to = Greenfield giv distribution o also t quan e surprising, the compression qcomp/bzip2; bzip2. 0.13 the similar t lossless F w either v ond out whic in w 1. (whic flat-field then er or is QUANTIZA 2), compression the o b en hcomp, three o the e difference Figure pro ac adv cases RMS is in qcomp/bzip2 tization and compression compression the consisten and to of qcomp/bzip2 mean h for ond not hieving same optimal b to the h qcomp/bzip2. lossless duces F an v the b est q quan times compression same ary urthermore, corresp three to o 0.25 et of tages: (1999) ≥ m to comparing first and image maxim coarse as 4b w ratios compres- of uc for distribu- metho the 0.19 4). een b tization and metho q w t (whic consis- signifi- and TION y h faster, and sp b again, differ- 16-bit in maxi- ratios > bzip2 e com- com- etter deep onds only b The eed. it pix- and sa (e). the um are de- ra- ra- for for for 2), et- ds W in c. w is is d h A TSON to suitable ing Greenfield ger comparison to standard grounds sigma formation has The problems w mate and quan Fig. estimated. and empirical from σ ture, images 160. the is rather b e larger = calculating the eliminate Nieto-San White the of actual has with images standard tization 5. √ lo Ob the the assuming clipping), w b empirical quan than square ) the er viously The than for or b quan empirical deviation pixel-to-pixel can standard & et en 7.1 on noise (1999) ha tized form Consider ra more w has Greenfield v is mean using the tisteban . the v een elop the ta w the deviation o ro e , Distribution somewhat ccur that q but a as data, in − standard ot with determined con noise, the e v = gain do generally 0 quan ariations difference to ra the . a of standard is deviation 15 1. a tribution in original w the the not the noise a scaled /σ not noise et but of tum The (1999) empirical quan data single an standard b pixel-b of al. noise . 1 claim flat y dominated deviation. unfair, it , and dotted of mo more of on on (1999) images in tizing data. to ∆ and in do deviation quan field w from only calculate Differ in del. the a y-pixel no es b a (in ould that whic e quan pixel-b standard an pixel-b deviation than illustrate tum read in line on significan This ab flat image units large-scale quan with h enc image This their remo the tized out that b mean is for the y noise, field. in the es 500 y-pixel is y-pixel tize the of noise. basis 75. high v eac (e), equiv metho the can images deviation e empirical White is quan difference the the tly is (after appro h b real Similar and Ev related y image ab whic struc- of image bac cause basis. alen o basis noise This dan- en tum tak- v d out the the in- er- for xi- k- & is if h 3 t © Copyright 2002: Instituto de Astronomía, Universidad Nacional Autónoma de México median of first sonian), eac b tigate p (although v gzip 8 and m sion no slo ences ertheless tion zation (1999), 16-bit gradien prop noise). v pression compressed ferences 0.202 (1998). ues pixel-to-pixel so White order difference to results tized San The sion that p alues elop y osed ort uc to 1. w that h read as Th White The Rice applying tisteban are h er” using mean assuming osed and is metho app 0.255 ac e The eac pixel images for ha smaller 1% us, here) this, in “Rice from metho 20 & app from hiev t the quan compression than and, pixel v et noise, than and h teger algorithm ev ears 7.2 Nieto-San quan gzip Greenfield e here times images of a it is strictly differences & al. pixel ears ery d calculated for an ed t a . single image 1 w and the et small tized app written the are suggested equations ds Bec their Compr compression” in using set (1993) mean basis o to tizing v (for to data. oscillatory that has b whole and Rice alue al. to sides slo y when quan ears sho a v k noise second 200 b of w alue quan sp er gzip; sligh w adv e ev 2 to b (1999) designed implemen mo tisteban ere with the results a wn difference e ession images er and eaking (1999), (the with bits with ratio en This only describ (1998) in tum of single b electrons, that the roughly an dified w b than et quan tum tly for teger (equiv b in read y 7 e in the White tages using w appropriate y 0.272 of original structure consider quan BZER the applying and ac c rep metho Figure v een differen the t Nieto-San the in hanging R the ery ypical for tized quan noise). w et distribution e White’s to and hiev rep Nieto-San tation atio squared. form, noise alen ort ere a m same the (of o − al. pixel-to-pixel tization compress 8 eac individual quan for noise sp v & ort uc sligh a O ed 0 d er tum, with 0.193 has created t t and 5. . ecial-purp order (1999) gain image. h h tly original w 15 Bec signals), of the is is gzip the b White to of with that quan equations image with o tization larger Quan tly y . /σ implemen tisteban 0 should zero. widely Gaussian differen Sp b k and Rice This tisteban Rice The q COMPRESSING the and fidelit of b other er een metho for 0.05% while . e b lo of tum gzip = a pixel rep e with 1 etter tizing quan The (1998) calculated d w-en and the mean & c quan BSCALE T pixel lo Rice compres- compres- probably but electron, 1 hange ose used b ort y referred o case metho w quan t giv et e Bec d is to Nieto- tation differ- 7 of v er of in quan- ta w mean trop mean et noise P com- alue, than tiza- nev- on that pro- al.). “far a es v and and v ois- dif- the the the en- k for ys: (of re- es- al- al. b ti- er of d y y a a RA b distribution (see pressed not suited differences. for (White limits b w et hcomp and it most duced b et With ratios pression here, ages tios the viations. more faster compression case compression v then pression differences of thermore, as tion len computing on consider alues y y y W ere w al. al. t Rice White White “-1”. compressing more Ho The It The this ould same to giv § IMA to the 8.1 § is they Rice inferior in (1999). w the (1999) qcomp, 7, quic w than to and b qcomp/gzip2 5). for in e “-6” 0.5–2, is ould teresting . 1992) ev y with task. quan compression. w b flag or suc ratios than GES the b Suitability fundamen the that RMS & & results e the If er, e kly; compression should its compressing e In sp So, and resources, p the gzip can h Greenfield Greenfield and placed of app w ossibly giv the tization eeds. sp w this ratios ratios decompression image in m or that W firm this e a w its ould the neither other eed but for difference uc en no data as mean e ear sp assume with is compression the kno range of adv 8. straigh pro h will w ecified errors to ab guaran range, is, tal example, designed on of with the pro that then to slo of but for DISCUSSION b wledge critically quan an metho on duce out compression gzip, y the and quan decompression. differences see adapt the w (1999) (1999) app 0.27–0.13 metho of duce Both tage ra at tests adv Compr w er a that tforw smaller is that are half tization w amoun “-1” ould tees as q that satellite so qcomp/bzip2 most distribution ears tizations v but qcomp/bzip2. an predictable. ds that for data ery w similar hcomp sp to qcomp and an understandably as of d tage ratio ould the and and ard ev as the to on essing w flag. eed the that compressing to is compress aluate similar 0.25–1 ould t. these image go no fast maxim use than comparisons and the (see metho metho nature b default Nieto-San Nieto-San metho lik b with F o default is e and pixel of to e for d urthermore, that qcomp compression bzip2, ely compress at ab R v distribution mo giv as use Fig. ery articles Neither either its aw of standard can qcomp similar out um compression compression Hcomp only ds ds sp d giv ot to will allo c es those differences Of suitabilit hange ra w Rice eed. Data similar is describ prop prop 4). if is es in 10 absolute as b tisteban tisteban pro w ra roughly ws equiv limited course, e c hcomp qcomp do b hange whic w in times m mean fo used, data. men- com- com- com- etter duce do osed osed F o pro- firm The uc 245 im- not cus the the de- v ur- § ra- ed to es a- er of h h y 6 © Copyright 2002: Instituto de Astronomía, Universidad Nacional Autónoma de México either tion to and deriv stalled quan brigh sion that tizes is clude has b with men the tan brigh b quan data of compressing ther no compress of data restrictiv if photometry significan White m the tion in mates tion output ac for m roughly it sion hcomp. 246 e e uc uc hiev end-to-end that hcomp b the is 0.5–2, similar suc t The The Ob It roughly single-output manner e h h metho ts the mak sp metho or tize tization sub-millimeter of ed small eac qcomp. (and w for tness tness and still separated faster astrometry smaller is es h viously drastic the bac on ere eed & almost is under-compression CCDs. hcomp from curren h requiremen 800 great adv also es lo e applications Its in t), Greenfield treats requires while ra the appropriately; kground o adequately d indeed in noise as w statistically sources. is d, studies tro adv v their w in constan than an er erscan of of only so v kpixel/s only sp metho the , When the all w tests absence c duces ery o data t tages univ an whic adv hanges compression stars, orth confidence v in Nieto-San eed. and c b implemen compressing on er-compression hanges of metho tage original noise oth the CCDs suc of disadv one that m an that at ersally (and the on h t. ds a These sections t w compressed using uc a The considering imaging stars, (1999) tage h of iden and Nieto-San t ould pixel-b other within whic Ho on third o ypical fast; h small of are in compressed v a d quan indistinguishable v it the aries the an in er more faster and claim the w unsuitable one biases, the tisteban data. tically a tation photometry the distinguishes of tests on that h ev b tage ap in of the the noise w 1.9 that determine e ratios tization its y-pixel bias of curren er, qcomp read ha p noise. only disadv 160 sligh fixed erture w a the and same significan the ortabilit slo v w orkstations. than tisteban v the an , are Suc curren bac elengths. compression individually GHz is this of e as of suc and qcomp/gzip, of wly that data in kpixel/s rate tly as et stringen that ultra of use image kground data the h t qcomp esp regions, manner hcomp, nev lossless an h basis for and metho the implemen ma hcomp. o tests the so al. 0.38–0.23 if P faster as t v tage of is ecially y; of the section. er en o er of tly use read y the implemen its b et v ga bac (1999) lo far not square PSF-fitting lo 40 et from tium erscan made) and gzip not and Suc qcomp should w the t has v al. w ds for quan impro of w compres- compres- although with t in e kground kpixel/s pro whereas to w infrared and require- een noise presen rate surface surface h imp qcomp results It should b (1999) o mak tation whic whic quad- quan- a is a other those IV e for tests duce ha tiza- esti- v ro also sec- and ra fur- v to the de- oid or- ta- in- in- ed v to ot es W of w is is h h o q e t A TSON purp distributions), is qcomp/bzip2 qcomp/gzip widespread univ needed the tion). capacities will compressed pacities. capacit in ratio. a mec b the a assuming fectiv effectiv a maxim tiv similar CD-R one compression im the sufficien Mpixel/s tium sor, widths sion on y capacit generation large a um e compressing an a If An T If hanisms devices maxim ma exp so to ersally b bandwidth drop ose able factor ely w IV OM e images early-2002 Th bandwidths um the imp e y e jor w w table to ectation presen tly RAID compression tak ork capacit of a y ere pro Th us, and to a 4 is decompress that adv effectiv um o ortan DL of on 3 quic often v e compression us, equal (it form, sho the cessor, almost zero. it with measured with erheads is could GB, of a can 400 with T an decompress ted w effectiv ma arra it 8.3 compression are already or a kly ws that y orkstations. of t ra tec tap tage that computer to 650 adv considered e 8.2 qcomp/gzip y GB. of b . w a to in large decompressing ev The 8.4 , assume a y hnology; q 5 bandwidths b e for o Me e, b a whic compression . 12 bandwidths en e . 60 device an e = P of times the o they compressed . metho will no MB e D and Distribution v constructed Th them; ap dia faster This Bandwidth curren or GB sev tage er bandwidths VD-R compression though 1, GB, forms data h ratio additional er in us, gain soft estimated CD-R at eral then an Cap a can can v with larger D ds. or I ratio of is erse single and As in essen I A to gzip ab compression OM, t sets arbitrarily w with part acities of compressing T roughly (W transp this read compress devices are for there out terms b media store bzip2 OM single and DDS-3 e of 0.3 increases an of them. and atson, than tial. is without a for compressed that disk distributed q adv of 12 0.2 the w soft and installed and 80 D for has the appropriate = ort images is ould b A of qcomp/bzip2; decompressed sev ha 1.9 Mpixel/s. equiv an their as ecomes or GB w 1. T is uses compression an qcomp/gzip tap write curren fast in an decompres- v mec capacit at are tage. eral t effectiv e tap GHz The can transp b ypical acquiring the o files prepara- e disk effectiv effectiv e ab alen v sp ra in hanism pro will almost erhead has e higher Lin band- out w ecial- allo effec- t data, max- more is their P This with with y t ces- and has en- ca- ort ely , for for (A ux ef- an b to w a 3 e e e © Copyright 2002: Instituto de Astronomía, Universidad Nacional Autónoma de México sults ren except CD-R and faster, all fastest im ficult. Mpixel/s and cate GHz will Mpixel/s 125 ing than suming in for suc lelism Ac cessors ab out um sp hieving t h reading devices T The Mpixel/s the require a optimal effectiv eed able that pro in P bzip2. OMs, (see bandwidths, 2009. compression the en for net 28 8-w curren compression an cessors ev 9. tium write read 4 bzip2 w actual the 83 times § ery and will a POSSIBLE impro e pro tap ork 9.2), sho y 32-bit Alternativ Ho bandwidth Mpixel/s read 9.1 t parallelism, 56 18 IV; cessors bandwidth es, bandwidth in effectiv ws b w connections. should is 4 D DL F 50 56 Slo Device 10 F 100 1 faster . bandwidths enefit. suc v kbps ev mon × ast ast ab A Gbps emen Compr single × b bandwidth if compression, that T w and, kbps Mbps er, T metho CD-R etter h out Mbps ratio CD-R pro RAID single ths, single DDS-3 sp only e link whic the ely write IMPR t b bandwidths 2005 link eeds for ev cessors of F link disks, e ession in link , than according d or could qcomp/bzip2 around from link en OM a whic exp 0.015 if 7 h DEVICE disk disk disk v effectiv fast that will sp should example, times ailable for O As w should no around erimen e VEMENTS eed and b h R b a con writing. w so enough oth arra allo pro approac Mpixel/s.) atio e w p 2008.) fast e faster will ould erformed impro this compression all tin b to w cessors w in y b optimal bandwidth AND ts e ould COMPRESSING 2006 e ue Mo ac but 8-w p RAID ab require ma a ab should ha ossible pro 0.003 h hieving v v (Again than Ra to out ore’s ailable ed a o b the v y 0.6 0.3 0.5 2.5 the TRANSPOR and y 25 50 v e e b cessors, w 4 7 4 5 double e b b ecome b paral- lo a 16-bit arra 2006. e max- y a etter indi- v w pro- la no cur- giv 125 dif- ery us- 1.9 as- Curren for re- er, 83 w, T in w y e ABLE Bandwidth t RA T Read noise the ac high-bac a ten pro noise ab ther This seems tio nothing pression thereb factors (a) bzip2 presumably scale job 0.193. sion again, bac forming qcomp/bzip2 age ence 16-bit 4 0.01 MECHANISM W 1.7 h hiev 12 12 12 12 12 12 o yp v kground One F of 2 8 v and information (a) and/or of IMA emen e, urthermore, impro ratio b should othetical structure with as ab e y et and compression that there’s enco Curren of are (b)) and a The w impro and v means kground out metho GES w (Mpixel/s) ery t een t compression ac v ell w optimal the of really ding emen bzip2 narro subtract images. o the are compressed 0.143, p hiev mean close the with erform as v only to t not compression d e same in to W as SPEED 0.172 flat-field ed that ts subsequen b b optimal it w 0.01 compression images the rite q is impro e oth to 1.7 w q m 18%. an is w will the compression 16-bit b gained. 3 = 3 3 2 3 3 1 3 ell = close uc ould q y optimally the easy compressed similarly flat-field and impro the 1; h ratio v histogram 0 as qcomp/bzip2 b v Maxim alues . the 8 e e to w bzip2 Ev sky 16-bit image bzip2 ac to to compression 0.143. small; ould t pixel-to-pixel the and metho hiev v en noise b compression. of sho emen optimal, e prior (whic ratios 0.01 . 150 image, um compression 1.7 a w ab is gained. q 83 23 13 13 17 and The e w b (e) compression ratios ell 2 1 8 This h the d of there e doing out a as = t that yp h, that to to compression pixel is of for compressed on mean w 0 othetical information with 0.164, . ell as and suggests only and only 7 compression. roughly bzip2 are ac of the a other compressed The for and as men v hiev v alues Ho Gaussian compres- so q ery no an bias 10% 35%. optimal ratio images an = w tioned is differ- w w ed y large- high- more com- go ev ould ould that flat- p 1 and fur- 247 im- im- the ra- for er- er, o b to in It is d y © Copyright 2002: Instituto de Astronomía, Universidad Nacional Autónoma de México gzip concatenating p is lelized to matter tunately and 248 w fixing qcomp/bzip2. men w lossless pression in w w sp those rapidly sample sible termining of migh require to qcomp tion a noise. presen en increases sections thermal the is This regions regions ground tiv ermit a ould orth ould single eed t adequately lossless decompress only ely y p qcomp, 9.3 T An Rice One This data ts bac ermit of decompression and to o t that . could of while . t is eac impro not are b b the One of obtain to is decompression ab kground (It effort ob determine recursiv are in R compressor a b gzip , problem y e bac situation infrared to bunzip2, steps. required compression y obustness section rapidly h the pixels, vious out that problem splitting compression mo the decompression to the the b image up do as determining not for b kground e compressed ha adaptiv v are e dify parallelize e and so es, file 3 c v region. noise the to noise the haracterized ac it.) priv it b further e ely significan times This adequate. con impro y in lev with hiev in can with ho differen formats 50% in assumes w the compression and could a the output quan un ate whic to for the ould against els, v e w m Ho lev as somewhat lev whic enien and 9.2 ed has b ev til sc that ust format regions the v length migh w e compression to images for w use. el could el input emen it heme er, tized h impro for . b a blo adequately the limit man ev the the tly t h v case hence already e could Sp b to and b is curren tly for from elength. qcomp/gzip of er, that That bac e e allo impro c the V t example, e b bac differen k t created. b noise ually parallelized. quan e arying qcomp/gzip. made b y p of b image w v gzip b of d the e suc the quan kgrounds.) it that e to with emen ortable, can w et e larger ould a eac kgrounds thermal parallelized b a the the reduced said, t a w w e v h single attractiv the b compression tization the compression implemen single een ed compressed ould and or h from b determined een tizing faster. that ha t ratios c ts, its compressed e Backgr whole b is pro haracterized b from ev factor). p sp e trivially data v writing The pro y erhaps but e pip bzip2 implemen and length); to en bac ectra compression b cess. only bac sub bac eac to v e in w cesses ery e (The similar ed and (It sub if the data ounds This F ould kground. migh impro and a tation zero, h kground kground (b dividing the 10% the if or do Unfor- simple a is to in region divide y differ- paral- adap- sp sp sp bac blo more more com- b only sub- sub- p cost bias pre- t files this sec- and still will oth eed not eed eed the ted the de- v os- for b b c to k- W of e- y k e A TSON and p results able White b the ensure can can optimally Mark soft image of b of b for is of compressing than gzip giv the exp This ratios is and that purp quan tees sists ages. b follo b and follo difference ort, e e e een een their adequately astronomical the real-w es presen relativ uncompressed I The The A A The Another use ected w compressed trail. distribute wing w b are in on ose tization hence of and ab o are. is thank 1/2 sho guaran presen e ed p lossy v ac Adler, k as difference. for that ossibly er-sample lossy particular to ey on out a used original mo next difficult metho hiev metho wn compression orld a . dramatic of b ted ely bzip2, Nieto-San mean This useful I the ra y b feature normal dern the metho thank ted. the a the et Enrique tees this ed w option the quan is without simple step lossless in and testing standard w fast compressed Shannon d the d data. c b is een and metho size v quan P hosen analyses. w distribution commen whic on y in data or The consisten alue, is ap tization the ac These orkstations, in d 10. Markus for using lossless Julian tro FITS on most of impro tisteban the imp lossy hiev the while w er for original to the tum Gazta h duces compression. to with metho ould the noise, his within SUMMAR the curren d to and I parallelize. maxim are deviation ossible I compressing ols. en ed do quan dev to guaran useful file ts v eliminate (W Sew suggestion but c need tly (resampling on emen Suc compression. trop metho data ols Ob es hanging naga b ˜ b on the compressed widely a elopmen d et It and atson, con images. y e t a tized not of ard, reduces h small a erh um hcomp is that whic computers writing y to giv to few app al. t exp I tees to and end-to-end tain pixel-b Y differences limit. v d compressing for umer on ha degrade in es estimate compress. ery absolute (1999), Jean-Loup install ears a the h tens ected is in and are no v bias v exp the mak the t certain his no e ra In means ailable to The and simple; of that preparation). images v the lo ev in w y-pixel for I ect that widely alue particular, of compare ra information, bac The compression original this w-order e in referee’s er thank RMS brigh images sp and for the w but it p quan doing degree v the that the data tests kground. receiv ercen alue, ecialized that b b general- suitable bzip2 guaran- data metho metho blazing y et it it results Gailly to as w tness) basis, a mean v more noise w tized Ric with they ould v con- alue bits will one 1/5 can t has has im- een the ail- the m ed, re- to of of is it d d y k , © Copyright 2002: Instituto de Astronomía, Universidad Nacional Autónoma de México the hard here Bec de pitalit Alan. Gailly Gazta Nieto-San Press, Louys, Huffman, Press, Ob erh M I. CUP), Pro nery ical Hanish, (h Murtagh, dio D. 2001, (h files” and & F k M staff ett, rancisco: w ttp://www.gzip.org/) ttp://www.ob ´ exico , w R., ´ naga, J. exico ˜ umer, W.H M. M. y W. c. ere J.-L. Engineers, ork , Data Systems M., Barnes in B. M. MNRAS, SPIE, tisteban, Ellis, Mehringer, D. of W H. 2nd conceiv F ., P R. at of atson: E., G., 1993 M. the A. Starc (a.w ebruary . 1992, T F. Analysis ASP), 1992, J., writing euk San edition R. F. 1952, 3354, (San Mac Romeo, 1999, I, Observ “gzip: [email protected]). & k, 40, olsky X. ed. Astronomical S., M. eds. 320, Instituto erh REFERENCES P Numerical Sto k 137 R. J. a edro F J.-L., Pro 1098 431 1999, A&AS, A., y Chan, umer.com/op rancisco: Soft , bzip2, , c D. 1998, 12 The L. S. kman, atorio A., C. ceedings Fixsen, A., Plan M. w M´ D., Mei., are data when Barriga, artir de S. “lzop 136, V W gzip, Recip H. te, McMahon, etterling, Astron´ ASP), J., Astronom and orrall, Data compression D.J., of S. & S., for 579 the - & ensource/lzop/) es compress and the 1999, D. Systems J., Analysis Ho their Bonnarel, in 3 omico C. Offen ideas A. Institute COMPRESSING & C W. enig, lzop. Biemesderfer, in ´ Rob R. ıa, (Cam Elizalde, w b T., Astronom- presen G., erg, or VI UNAM, arm program” Nacional M. erts Soft I & expand I bridge: thank I, of P F., J. Flan- 1998, w arry (San hos- eds. Ra- ted are D., E. & , Apartado RA Shannon, White, White, Romeo, White, Sew Rice, V W Whitten, V W eillet, ´ eran, ells, Analysis (h gram and (h Mehringer, 27, A&AS, J. 623 Conference Information sion 687 1999, Engineers, tems ceedings (ftp://www.cfh vironmen cisco: nomical & m P ard, IMA ostal unications ttp://sundog.stsci.edu/ric ttp://sources.redhat.com/bzip2/) C. R. 379 D. R. Earth R. C. R. J. A., J. for Tilton In F., ASP), I. J. GES I C. C., . . I 3–72 L. 44, ternational 1998, L., L., I, and H., P Gaztanaga, of t 1949, Y Barnes Soft 1948b, the R. E. ., 1992, Data Science and 37, eh, Greisen, eds. R. 363 (AIAA-93-4541-CP) & the & Neal, Bulletin, (ftp://stsci.edu/soft of 125 (Xangari) 1948a, & w 1998, MegaPrime: L. 10 HST Greenfield, t.ha P Bec are the library Pro Wide 9th .-S., in D. Plan Bell W R. k Analysis w Data ceedings A Journal (San righ Pro and er, aii.edu/pub/compfits/) AIAA E. Bell CM, Adv M., R. “bzip2 & E., te, Field 39 System ceedings t, I. W., 58089 Miller, Compression Systems & ance & Crabtree, Barriga, for System 30, 1998, P J. F A D. Computing of Image k/acscompression.ps.gz) Cleary of . rancisco: & and 520 New in Soft Mo T A. the Morelia, R. Camera data Harten, ec W. w of On-b Astronomical dern Rob hnical are/hcompress/) T w , VI J., Camera Institute libbzip2: 1994, the ec J. Prime-F are H. R. I hnical erts I, & oard G. W Ph 1993, compression” NASA in ASP), Mic R. orkshop, for J. Journal, Elizalde, eds. 1987, ysics and (San in Aerospace in o compres- H. Hanisc of hoac´ Journal, cus Surv in a D. CFHT Astro- C, Radio Space Com- F 1981, Data Pro- Sys- pro- ran- 249 En- 519 an, eys 10, ed. 27, M. E. h,