SoftwareSoftware ProductivityProductivity ToolsTools Purdue School on HPC Sept. 4-5, 2008 Dave Seaman ([email protected]) Rosen Center for Advanced Computing AA SelectionSelection ofof ToolsTools TarTar MakeMake PBSPBS VersionVersion controlcontrol systemssystems •• CVSCVS •• subversionsubversion TarTar SimpleSimple ArchivesArchives CompressedCompressed ArchivesArchives ((gzipgzip)) CompressedCompressed ArchivesArchives ((bzip2bzip2)) SimpleSimple ArchivesArchives CreatingCreating anan ArchiveArchive ExractingExracting fromfrom anan ArchiveArchive LookingLooking beforebefore YouYou LeapLeap CreatingCreating anan ArchiveArchive tartar cfcf myarchive.tarmyarchive.tar filesfiles UseUse cc optionoption toto createcreate anan archivearchive UseUse ff optionoption toto specifyspecify aa diskdisk filefile (default(default isis tape)tape) ExtractingExtracting fromfrom anan ArchiveArchive tartar xfxf myarchive.tarmyarchive.tar [files[files……]] UseUse xx optionoption toto extractextract files.files. DefaultDefault isis toto extractextract allall filesfiles inin archive.archive. LookingLooking beforebefore YouYou LeapLeap ManyMany archivesarchives willwill extractextract intointo aa subdirectory,subdirectory, butbut somesome writewrite filesfiles inin currentcurrent directory.directory. ToTo bebe safe,safe, extractextract intointo anan emptyempty directory.directory. UseUse tt optionoption toto seesee what'swhat's there:there: tartar tftf myarchive.tarmyarchive.tar CompressedCompressed ArchivesArchives ((gzipgzip)) WeWe maymay useuse gzipgzip toto compresscompress anan archive:archive: tartar cfcf mystuff.tarmystuff.tar mystuffmystuff gzipgzip mystuff.tarmystuff.tar AA shortcutshortcut isis availableavailable whenwhen usingusing GNUGNU tar:tar: tartar zcfzcf mystuff.tar.gzmystuff.tar.gz mystuffmystuff UncompressingUncompressing withwith gunzipgunzip AA compressedcompressed archivearchive namename maymay endend inin ..tar.gztar.gz,, .tar.Z,.tar.Z, oror ..tgztgz.. WeWe maymay extractextract filesfiles byby usingusing aa twotwo--stepstep process:process: gunzipgunzip mystuff.tar.gzmystuff.tar.gz tartar xfxf mystuff.tarmystuff.tar UncompressingUncompressing ShortcutsShortcuts WeWe maymay useuse zcatzcat toto unzipunzip andand pipepipe thethe resultresult intointo tar:tar: zcatzcat mystuff.tar.gzmystuff.tar.gz || tartar xfxf -- GNUGNU tartar hashas thethe zz optionoption forfor zip/unzip:zip/unzip: tartar zxfzxf mystuff.tar.gzmystuff.tar.gz Bzip2Bzip2 CompressionCompression Bzip2Bzip2 isis usedused muchmuch likelike gzipgzip,, butbut tendstends toto havehave betterbetter compressioncompression ratiosratios atat thethe costcost ofof slightlyslightly moremore time.time. bzip2 mystuff.tar bunzip2 mystuff.tar.bz2 bzcat mystuff.tar.bz2 | tar xf - GNUGNU tartar andand bzip2bzip2 GNUGNU tartar usesuses thethe jj optionoption toto invokeinvoke bzip2/bunzip2:bzip2/bunzip2: tar jcf mystuff.tar.bz2 mystuff tar jxf mystuff.tar.bz2 MakeMake AutomatesAutomates maintainingmaintaining filesfiles thatthat dependdepend onon otherother files.files. MayMay bebe usedused forfor programs,programs, documents,documents, oror arbitraryarbitrary developmentdevelopment projects.projects. InstructionsInstructions containedcontained inin MakefileMakefile controlcontrol stepssteps neededneeded toto keepkeep filesfiles upup toto date.date. SimpleSimple MakefileMakefile ## variablevariable definitionsdefinitions PROGPROG == graphgraph SRCSRC == main.cmain.c graph.cgraph.c OBJOBJ == main.omain.o graph.ograph.o CCCC == iccicc CFLAGSCFLAGS == --O2O2 LOADLIBESLOADLIBES == --lmlm SimpleSimple MakefileMakefile (cont.)(cont.) all:all: $(PROG)$(PROG) $(PROG):$(PROG): $(OBJ)$(OBJ) $(LINK.c)$(LINK.c) --oo $@$@ $(OBJ)$(OBJ) $(LOADLIBES)$(LOADLIBES) ## additionaladditional dependencydependency rulerule main.omain.o graph.o:graph.o: graph.hgraph.h RulesRules target:target: prereq1prereq1 prereq2prereq2 …… command1command1 command2command2 [[……]] EachEach commandcommand mustmust bebe precededpreceded byby aa tab.tab. InvokingInvoking makemake make [-f makefile] [target] [var=defs]… DefaultDefault makefilemakefile namesnames (in(in order):order): •• GNUmakefileGNUmakefile •• makefilemakefile •• MakefileMakefile DefaultDefault targettarget isis firstfirst oneone inin makefilemakefile.. VariableVariable definitionsdefinitions onon commandcommand lineline overrideoverride valuesvalues givengiven inin makefilemakefile.. SimpleSimple MakefileMakefile ## variablevariable definitionsdefinitions PROGPROG == graphgraph SRCSRC == main.cmain.c graph.cgraph.c OBJOBJ == main.omain.o graph.ograph.o CCCC == iccicc CFLAGSCFLAGS == --O2O2 LOADLIBESLOADLIBES == --lmlm SimpleSimple RuleRule OurOur simplesimple makefilemakefile containscontains asas itsits firstfirst rule:rule: all:all: $(PROG)$(PROG) ThisThis meansmeans make isis equivalentequivalent toto make all. ThereThere isis oneone prerequisite:prerequisite: thethe targettarget allall dependsdepends onon $(PROG),$(PROG), whichwhich involvesinvolves aa variablevariable substitution:substitution: PROG = graph VariableVariable SubstitutionSubstitution VariableVariable namename isis precededpreceded byby $$ andand isis enclosedenclosed inin parentheses:parentheses: $(PROG) ParenthesesParentheses maymay bebe omittedomitted onlyonly ifif thethe namename isis aa singlesingle character.character. SomeSome havehave standardstandard meanings,meanings, suchsuch asas $@ forfor thethe targettarget currentlycurrently beingbeing updated.updated. BracesBraces maymay bebe usedused insteadinstead ofof parentheses:parentheses: ${PROG} DefaultDefault RulesRules TheThe MakefileMakefile sayssays $(PROG)$(PROG) dependsdepends onon $(OBJ)$(OBJ) andand givesgives aa commandcommand toto useuse forfor creatingcreating $(PROG):$(PROG): $(PROG): $(OBJ) $(LINK.c) -o $@ $(OBJ) $(LOADLIBES) However,However, nono rulesrules areare statedstated forfor producingproducing thethe $(OBJ)$(OBJ) files.files. TheThe DefaultDefault RulesRules SinceSince nono commandscommands werewere foundfound forfor makingmaking thethe objectobject files,files, makemake usedused thethe defaultdefault rule:rule: %.o:%.o: %.c%.c ## builtbuilt--inin commandscommands toto makemake aa .o.o filefile $(COMPILE.c)$(COMPILE.c) $(OUTPUT_OPTION)$(OUTPUT_OPTION) $<$< TheThe rulerule beginningbeginning withwith %.o: %.c isis anan exampleexample ofof aa patternpattern rulerule.. DefaultDefault CompilationCompilation RuleRule $(COMPILE.c)$(COMPILE.c) isis predefinedpredefined byby makemake:: COMPILE.c = $(CC) $(CFLAGS) $(CPPFLAGS) \ $(TARGET_ARCH) -c YouYou maymay definedefine thethe variablesvariables (CC,(CC, CFLAGS,CFLAGS, ……)) asas neededneeded inin orderorder toto customizecustomize thethe command.command. SomeSome StandardStandard VariablesVariables $(COMPILE.c) Command for compiling C programs $(LINK.c) Command for linking C programs $< First prerequisite $@ Target being updated $(CC) C compiler $(CFLAGS) Flags for C compiler $(CPPFLAGS) Flags for C preprocessor $(CXX) C++ compiler CommandsCommands ProducedProduced RunningRunning makemake withwith thesethese rulesrules maymay produceproduce severalseveral commands:commands: iccicc --O2O2 --cc --oo main.omain.o main.cmain.c iccicc --O2O2 --cc --oo graph.ograph.o graph.cgraph.c iccicc --O2O2 --oo graphgraph main.omain.o graph.ograph.o --lmlm OnlyOnly thethe necessarynecessary stepssteps areare actuallyactually performed,performed, dependingdepending onon whichwhich filesfiles areare olderolder thanthan theirtheir prerequisites.prerequisites. SpecifyingSpecifying DependenciesDependencies TheThe filesfiles main.cmain.c andand graph.cgraph.c bothboth #include#include "graph.h""graph.h".. IfIf wewe makemake changeschanges inin graph.hgraph.h,, thenthen bothboth main.omain.o andand graph.ograph.o needneed toto bebe updatedupdated (recompiled).(recompiled). NoNo commandscommands areare neededneeded here.here. WeWe onlyonly needneed thethe dependencydependency toto forceforce recompilationrecompilation usingusing thethe builtbuilt--inin rule.rule. OOtherther UseUsefulful FlagsFlags withwith makemake --nn DisplayDisplay thethe commandscommands thatthat willwill bebe generated,generated, butbut don'tdon't execute.execute. --vv ShowShow verboseverbose output.output. --pp ShowShow completecomplete databasedatabase ofof rules,rules, includingincluding builtbuilt--inin rules.rules. ForFor MoreMore InformationInformation http://www.gnu.org/software/make/manual/make.htmlhttp://www.gnu.org/software/make/manual/make.html PortablePortable BatchBatch SSystemystem (PBS)(PBS) SubmitsSubmits jobsjobs toto runrun onon aa cluster.cluster. SchedulesSchedules jobsjobs toto runrun whenwhen resourcesresources becomebecome available.available. ManagesManages runningrunning ofof jobsjobs andand returnsreturns outputoutput toto user.user. AllowsAllows interactiveinteractive execution.execution. UsefulUseful PBSPBS CommandsCommands qsubqsub SubmitSubmit PBSPBS jobjob qdelqdel DeleteDelete PBSPBS jobjob qstatqstat DisplayDisplay statusstatus ofof PBSPBS jobs,jobs, queues,queues, oror serversservers pbsnodespbsnodes QueryQuery PBSPBS hosthost statusstatus TheThe qsubqsub CommandCommand ToTo submitsubmit thethe scriptscript myscriptmyscript asas aa batchbatch job,job, use:use: qsub [options] myscript OptionsOptions maymay bebe embeddedembedded inin aa scriptscript filefile oror specifiedspecified onon thethe commandcommand line.line. ToTo submitsubmit anan interactiveinteractive job,job, use:use: qsub -I [options] UseUse man qsub toto