Japanese Virtual Observatory (JVO) and Naregi (Japanese Grid Middleware Initiative)
Total Page:16
File Type:pdf, Size:1020Kb
EURO-VO DCA Workshop Japanese Virtual Observatory (JVO) and NaReGi (Japanese Grid middleware initiative) Masatoshi Ohishi / NAOJ, Sokendai & NII 大石雅寿 / 国立天文台,総合研究大学院大学, 国立情報学研究所 [email protected] agenda • Status of the Japanese Virtual Observatory • On-demand image mosaicing service for the SuprimeCAM data • Quick Multiple Catalog access service • Prototype for a Workflow builder • NaReGi – National Research Grid Initiative in Japan • A testbed to federate NAOJ & KEK through NaReGi 2 Supported by • JSPS “Core to Core Program” (2004~2009) • MEXT Grant-in-Aid “Information Explosion” (2001~ ) • National Institute for Informatics “CSI Program” (2007~ ) • NAOJ 2008 April 10 EURO-VO DCA Workshop 3 JVO collaborators Scientists System Engineers NAOJ Fujitsu Ltd. SEC Ltd. • Ohishi • Kawarai • Morita • Mizumoto • Ishihara • Nakamoto • Oe • Machida • Kobayashi • Shirasaki • Tsutsumi • Sakamoto • Tanaka • Hiyama • Tsuyuki • (Honda) ICRR • Yasuda Supporter Aoyama Gakuin U. NII • Miura • Masunaga 4 Data Resources in NAOJ • Subaru 8.2m Optical-Infrared Telescope • Kiso 105cm Schmidt Camera Nobeyama 45m • Okayama 188cm Optical Telescope • Nobeyama 45m Radio Telescope • Nobeyama Millimeter Array • Nobeyama Radioheliograph Subaru • VSOP • VERA • ALMA Data Resources in JAXA/ISAS • ASCA X-ray astronomy satellite • YOHKO solar physics satellite • Ginga X-ray astronomy satellite • HALCA VLBI satellite • Geotail geomagnetosphere satellite ASCA • Akebono aurora observation satellite • AKARI Infrared satellite • SUZAKU X-ray satellite • HINODE YOHKO Solar sat. HALCA Japan – Its Uniqueness • Two major astronomical research institutes cover entire spectrum – NAOJ: ground-based instruments (radio ~ optical) and a supercomputer – ISAS: space-based instruments • There has been close collaboration between the two institutes – building satellites, dedicated line, personnel exchanges, etc. 7 Overview of the JVO Portal Service LDAP 解析 解析 解析Analysis Grid Servers Auth. System JVO SDSS QSO Invoke SkyNode Subaru MDS Portal 1.0 … 2MASS User Web Service Storage Data Service HTTP Get FTP S. Reg. LDAP Other VOs SIAP/SSAP P. Reg. SkyNode Registry P. Reg. 2008 April 10 EURO-VO DCA Workshop 0.7.4 8 Log(F_IR) MAG_B 2008 April 10 EURO-VO DCA Workshop 9 Spectrum Data Viewer • Display • Baseline estimation • Line detection & parameter read • ・・・ 2008 April 10 EURO-VO DCA Workshop 10 User Portal http://jvo.nao.ac.jp/portal/ 11 Simple Interface to Multiple Catalog Query 12 Access to the SuprimeCAM Data • On-Demand Mosaicing of images and calibrations • Pre-processed mosaic images are also accessible 13 Management of Multiple Servers • Data Analysis services under JVO ‒ SExtractor (Image → Catalog) ‒ HyperZ (Catalog → photometric redshift) • In operation on multiple servers → Monitor and Discovery Service (MDS) server JVO Portal MDS Server Analysis Servers # of Jobs Service Request CPU load Job status Execute Available Server Info Confirm Job Completion Analysis Server Job Completion Info Management Data Job submission Client Results Parallel Executions on Multiple Servers are available 14 – Reduction 10s (Bias+Flat +Distortion+ Astrometory) – Transfer 30s Data Reduction: ~24 hours ~1 hour 15 Pop-up Menu for various functions 16 Quick Image viewer, SED plotter, etc 17 Link to external resources 18 Interconnected VO Data services in the World Canada VO China VO Data Service Operations have just been started March, 2008. 1300+ data resources are accessible. 19 Portal VO summer school SuprimeCAM mosaic 20 Quick Multiple Catalog Search Toward higher access efficiency to “Frequently Used DBs” -- ~ 109 records 21 Table Design Extract the minimum set of meta data from each DB: Position, Wavelength & Intensity Other info is shown as LINKs to reference 22 Acceleration by means of Table Partitioning Table design: Search Result # of HTM Elaspted time (sec) • Partitioning with HTM radius objects conditions (Hierarchical Postgre Our Postgre Our Triangular Mesh) arcmin # ratio SQL method SQL method • Upper level : level 6 6 : 8×4 = 32768 1 2 6.460 0.042 154 32 32 segments 10 165 3.807 0.030 127 16 16 • Tables are partitioned by Upper HTM level : 60 6697 6.468 0.107 60 32 32 32768 tables 100 26720 2.016 0.307 6.6 4 16 180 57246 9.044 0.709 12.8 48 72 (by Tanaka-san) 23 Improvement of Work Flow system Items in FY2007 • Improvement of WFDL – Shorten the volume of DL – Array definition – Awk-like description (loop WF Editor description in a single line) – and so on • Built-in functions – Available to register and use built-in functions Execution monitor • Status and Log system • Need a GUI builder – Investigating reference systems 2008 April 10 EURO-VO DCA Workshop 24 WF builder prototype by means of Taverna • Good graphical interface together with corresponding script • Builder prototype in 2007 • Easy construction of WFs w/ WFDL • Visual display of execution status 25 National Research Grid Initiative (NAREGI) Project : Overview - Originally started as an R&D project funded by MEXT (FY2003-FY2007) extended until FY2010 2 B Yen(~17M$) budget per year - One of Japanese Government’s Grid Computing Projects Visualization Grid, Grid Technology Research Center, BioGrid, ITBL (IT-Based Laboratory), etc. - Collaboration of National Labs. Universities and Industry in the R&D activities (IT and nano-science apps.) - NAREGI Testbed Computer Resources (since FY2003) MEXT:Ministry of Education, Culture, Sports,Science and Technology 26 (1) To develop a Grid Software System (R&D in Grid Middleware and Upper Layer) as the prototype of future Grid Infrastructure in scientific research in Japan (2)To provide a testbed to prove that the High-end Grid Computing Environment (100+Tflop/s) can be practically utilized in the nano-science applications over the Super SINET. (3) To Participate in International Collaboration /Interoperability (U.S., Europe, Asian Pacific) GIN (4) To Contribute to Standardization Activities, e.g., OGF NAREGI Software Stack Grid-Enabled Nano-Applications Grid Grid Visualization Grid PSE Programming Data Packaging -Grid RPC Grid Workflow Grid -Grid MPI Super Scheduler Distributed Information Service Grid VM High-Performance & Secure Grid Networking SINET3 Research NII etc IMS Organizations Computing Resources Japanese Academic(Science) Information Network( SINET/ SuperSINET SINET3) 29 Circuit Speed SINET (44 nodes) 1 Gbps Super SINET (35 nodes) 10 Gbps Kitami Institute of Technology International Line U.S.A: 10 Gbps×1 2.4 Gbps×1 Hokkaido Singapore: 622 Mbps×1 University Hong Kong: 622 Mbps×1 Collaboration with other global network Hirosaki University (1) “lifeline” for educaional and research activities in Japan (2) Indispensable for advanced research Niigata University Tohoku University cooperation Gunma University Japan Advanced Institute of High-energy Accelerator Research Saitama University (3) Continuous development towards Science and Technology Toyama Univ. Organization Kanazawa Univ. Univ. of Electro-Communications the establishment of the full-fledged Shinshu Univ. Univ. of Ttskuba Fukui Univ. National Institute Yamanashifor National Astronomical Observatory of Japan Tokyo Univ. Waseda Univ. CSI Tottori Univ. Fusion Science Agriculture Univ. Univ. of Tokyo Kyoto Univ. And Technology Hiroshima Univ. NII National Institute of Informatics Shizuoka Univ. Chiba University Tokyo Institute of Technology Okayama Univ. Kobe Univ. Osaka Univ. Nagoya Univ. Kyushu Institute National Institute Institute of Statistical Mathematics Kansai Univ.Doshisha Univ. Yokohama National Univ. of Technology Yamaguchi Univ. Kagawa Univ. Of Natural Science Keio Univ. Kyushu Univ. Japan Aerospace ExplorationJapan Agency for Univ. of Tokushima Agency Institute of Space Marine-Earth Science Ehime Univ. and Astronautical Science Technology Nagasaki Univ. Oita Univ. Kumamoto Univ. Kagoshima Univ. Participation SINET Institutions Univ. of the Ryukyu National Public Private Junior Technical Inter-univ. Others Total Univ. Univ. Univ. College College institute 82 48 266 78 42 14 184 713 Collaboration in Data Grid Area • High Energy Physics(GIN) - KEK (High Energy Accelerator Research Organization) - under collaboration with EGEE • Astronomy - National Astronomical Observatory of Japan (Virtual Observatory) • Bio-informatics - BioGrid Project 30 Large Hadron Collider at CERN Detector for LHCb experiment Detector for ALICE experiment 31 ~1850 ATLAS Detector physicists from 33 countries 32 Visit to CERN 33 Multi-TMulti-Layeredier Regional Regional Center Analysis 34 Scheme Center for the LHC 1 TIPS = 25,000 SpecInt95 ~PBytes/sec Online System ~100 MBytes/sec PC (1999) = ~15 SpecInt95 Offline Farm ~20 TIPS Bunch crossing per 25 nsecs. ~100 MBytes/sec 100 triggers per second CERN Computer Event is ~1 MByte in size ~622 Mbits/sec Center >20 TIPS or Air Freight France Regional Germany Regional Italy Regional US Regional Center Center Center Center ~4 TIPS ~2.4 Gbits/sec Tier2 Center Tier2 CenterTier2 CenterTier2 CenterTier2 Center ~1 TIPS ~1 TIPS ~1 TIPS ~1 TIPS ~1 TIPS ~622 Mbits/sec Physicists work on analysis “channels”. InstituteInstitute Institute Institute ~0.25TIPS Each institute has ~10 physicists working Physics data cache 100 - 1000 on one or more channels Mbits/sec Data for these channels should be cached by the institute server Workstations Subaru SuprimeCam Single Shot Raw Image • 10~20 minutes exposure 160 MB / shot • Postprocesses (Bias+Flat+Distortion+Astrometory, mosaicing to fill the gaps, coadding ) are needed 2008 April 10 EURO-VO DCA Workshop 35 Testbed for NAOJ-KEK federation User