1 Evolving Software Rep ositories http://www.netli b.org/utk/pro ject s/esr/ Jack Dongarra UniversityofTennessee and Oak Ridge National Lab oratory Ron Boisvert National Institute of Standards and Technology Eric Grosse AT&T Bell Lab oratories 2 Pro ject Fo cus Areas NHSE Overview Resource Cataloging and Distribution System RCDS Safe execution environments for mobile co de Application-l evel and content-oriented to ols Rep ository interop erabili ty Distributed, semantic-based searching 3 NHSE National HPCC Software Exchange NASA plus other agencies funded CRPC pro ject Center for ResearchonParallel Computation CRPC { Argonne National Lab oratory { California Institute of Technology { Rice University { Syracuse University { UniversityofTennessee Uniform interface to distributed HPCC software rep ositories Facilitation of cross-agency and interdisciplinary software reuse Material from ASTA, HPCS, and I ITA comp onents of the HPCC program http://www.netlib.org/nhse/ 4 Goals: Capture, preserve and makeavailable all software and software- related artifacts pro duced by the federal HPCC program. Soft- ware related artifacts include algorithms, sp eci cations, designs, do cumentation, rep ort, ... Promote formation, growth, and interop eration of discipline-oriented rep ositories that organize, evaluate, and add value to individual contributions. Employ and develop where necessary state-of-the-art technologies for assisting users in nding, understanding, and using HPCC software and technologies. 5 Bene ts: 1. Faster development of high-quality software so that scientists can sp end less time writing and debugging programs and more time on research problems. 2. Less duplication of software development e ort by sharing of soft- ware mo dules. 3. Less time and e ort sp ent in lo cating relevant software and in- formation through the use of appropriate indexing and search mechanisms and domain-sp eci c exp ert help systems. 4. Reducing information overload through the use of lters and au- tomatic search mechanisms. 6 Intended Audience: HPCC application and computer science community { Source of material for NHSE Users of NASA, NSF, DOE and other sup ercomputer centers { Go o d targets for NHSE { Natural supp ort organization: sup ercomputer center sta Other users of high p erformance computers { Current and p otential industrial users { No natural supp ort organization Applicable to other domains 7 NHSE Comp onents: Discipline Oriented Rep ositories Submission and Review Common Infrastructure { Resource cataloging and distributed system { Rep ository to ols and resource center { Naming and authentication { Publishing to ols HPCC Sp eci c Searching Outreach and technology transition { To the HPCC user community and industry Measurement Hyp ertext Road Map Selective Capitalization of Emerging Technologies Collection Management 8 Physical Physical Repository Physical Repository 2 Repository 1 n catalog catalog info info catalog info file request retrieved file NHSE Search/Browse Interface search results search request Virtual Repository Architecture 9 NHSE Based on Existing Technologies { WWW Browser Mosaic / Netscap e / etc Distributed / Scalable URL: http://www.netlib.org/nhse/ { Netlib Rep ository for math software since 1985 Rep ositories Currently Available { Netlib, Softlib, CITlib { ASSET - Asset Source for SW Engineering Tech. { CARDS - Comprehensive Approachto Reusable Defense SW { ELSA - Electronic Library Services and Appl. { GAMS Virtual Software Rep ository { STARS - SW Technology for Adaptable, Reliable Systems { Many examples related to GC problems Currently Available Information { NHSE currently p oints to 350+ mo dules software catalog, tech rep orts and pap ers parallel pro cessing to ols numerical libraries Grand Challenge prototyp e co des data analysis and visulization b enchmarks 10 Discipline Oriented Rep ositorie s Interop eration The NHSE will catalog and provide access to software and software- related artifacts from all the HPCC software rep ositories. Assets accessible from other existing software rep ositories, such as ASSET, CARDS, DSRS, and ESLA, may also b e of interest to NHSE users. The NHSE will b e participating in a small-scale interop erability exp eriment with the ab ove rep ositories to help de ne require- ments for further interop eration e orts. The NHSE will also b e working with the Reuse Library Interop er- ability Group RIG on establishing standards for unique naming, asset description and classi cation, and asset evaluation. In the future, the NHSE will interop erate with these other rep os- itories so that software from them may b e retrieved directly from the NHSE interface. 11 Netlib - Network Access to Mathematical Software and Data Began in 1985 { JD and Eric Grosse, AT&T Bell Labs Motivated by the need for cost-e ective, timely distribution of high-quality mathematical software to the community. Designed to send, by return electronic mail, requested items. Automatic mechanism for the disseminate of public domain soft- ware. { Still in use and growing { Mirrored at a numb er of sites netlib2.cs.utk.edu netlib1.epm.ornl.gov research.att.com netlib.no unix.hensa.ac.uk ftp.zip-b erlin.de nchc.edu.tw Mo derated collection of high-quality math software Distributed maintenance Mo del for domain-sp eci c rep ositories 12 Netlib { Network access to mathematical software and data Jack Dongarra Univ. Tenn. and ORNL Eric Grosse AT&T Bel l Labs, Murray Hil l NJ 13 netlib Started in 1985. Motivated by the research community Uses email for the distribution. Has grown in p opularity and scop e. Funded by the NSF and Bell Labs. 14 The development of NETLIB was motivated by the need for cost- e ective, timely distribution of high-quality mathematical soft- ware to the research community at large. The system was designed to send, by return electronic mail, re- quested routines together with subsidiary routines and any re- lated do cuments or test programs supplied by the authors. Automatic mechanism for the disseminate of public domain soft- ware. 15 Try: mail [email protected] mail [email protected] send index send rs from eispack who is Golub Collection includes: Linpack Eispack Fishpack Odepack ACM TOMS Benchmark Bihar Blas BMP Conformal f2c FMM Fnlib Fftpack Hompack Lanczos LP/data Minpack Napack NL2SOL Odepack Paranoia Pltmg Polyhedra Port Pppack Quadpack SIAM memship Sparspak Typ esetting Vanhu el Voronoi ... 16 Netlib provides the following features: There are no administrativechannels to go through. Since no human pro cesses the request, it is p ossible to get software at any time, even in the middle of the night. The most up-to-date version is always available. Individual routines or pieces of a package can b e obtained instead of a whole collection. Over around 15000 requests a day. Software collection ab out 1 Gbytes 21K le in 330 libraries/directories 17 Interdisciplinary resource Software Parallel pro cessing collection Data To ols Rep orts Do cumentation Benchmarks Journal information 18 Synchronization and Netlib Sites: Still in use and growing Mirrored at a numb er of sites { netlib2.cs.utk.edu { netlib1.epm.ornl.gov { research.att.com { netlib.no { unix.hensa.ac.uk { ftp.zip-b erlin.de 19 O sho ots Other sites running the netlib pro cessor, but to supp ort other databases: statlib@temp er.stat.cmu.edu statistics [email protected] T X E [email protected] Reduce symbolic algebra [email protected] Maple symbolic algebra [email protected] benchmarks Well overahundred copies of netlib itself have b een shipp ed. 20 NETLIB do es not o er Technical assistance in determining and correcting problems with library software. Pro cedures for testing or validating co des. A uniform style for programming and do cumentation. Uniform error handling within the library. 21 Requests Made to the Netlib Repositories at the Univ. of Tennessee & ORNL 8,946,816 total requests to these repositories as of Feb 15, 1996 4000 3000 2000 Requests x 1,000 1000 785,025 0 1985 1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 Year HTTP Gopher FTP XNetlib EMail Data as of - Feb 15, 1996 at 02:07:27 22 Breakdown of requests to each Netlib library (an alphabetical listing is also available.) Data as of 02/15/96 at 02:09:20 Library Name Number of accesses lapack 475,979 pvm3 379,849 linpack 256,403 slatec 248,292 blas 178,728 clapack 129,256 linalg 127,022 eispack 126,116 slatec/src 118,366 toms 117,152 f2c 98,025 c++ 96,774 benchmark 85,552 master 69,997 f2c/src 67,415 minpack 60,632 fn 59,781 fftpack 58,805 na-digest 50,970 port 49,496 slatec/lin 46,800 hence 45,229 confdb 43,118 slatec/chk 37,640 c++/answerbook 37,524 napack 36,719 23 Discipline Oriented Rep ositorie s Di erent discipline s will maintain their own software rep ositories Users should not need to access each of these rep ositories sepa- rately NHSE will provide a uniform interface to a virtual HPCC software rep ository which will b e built on top of the distributed set of discipline-oriented rep ositories. The interface will assist the user in lo cating relevant resources and in retrieving these resources. A combined browse/searchinterface will allow the user to explore the various HPCC areas and b ecome familiar with the available resources. A longer term goal of the NHSE is to provide users with domain- sp eci c exp ert help in lo cating and understanding relevant re- sources. 24 Discipline Oriented Rep ositorie s HPCC Cataloging To enable searching, cataloging information must b e made avail- able for NHSE assets. Eachphysical rep ository will b e resp onsible for maintaining a network-accessible le containing such cataloging information. These les will b e retrieved and indexed by an NHSE indexer on a regular basis, and the resulting searchable index will b e replicated for reliability.