DOCSLIB.ORG

Using GNU Make for Neuroimaging Workflow

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Using GNU Make for Neuroimaging Workflow Using GNU Make for Neuroimaging Workflow University of Washington Using GNU make for Neuroimaging Workflow Mary K. Askren∗, Trevor K. McAllister-Day, Natalie Koh, Zoé Mestre, Jennifer N. Dines, Benjamin A. Korman, Susan J. Melhorn, Daniel J. Peterson, Matthew Peverill, Swati D. Rane, Melissa A. Reilly, Maya A. Reiter, Kelly A. Sambrook, Karl A. Woelfer, Xiaoyan Qin, Thomas J. Grabowski, and Tara M. Madhyastha† University of Washington, Seattle May 26, 2016 ∗[email protected] †[email protected] i Contents Contents ii List of Figures vi I Manual1 1 Introduction to make 2 Conventions Used in This Manual ...................... 3 Quick Start Example.............................. 3 A More Realistic Example........................... 5 What is the Difference Between a Makefile and a Shell Script?...... 7 Make Built-In Functions............................ 8 The shell and wildcard Functions .................. 8 Text and Filename Manipulation.................... 8 Logging Messages from make ...................... 10 Additional Resources For Learning About make . 11 2 Running make in Context 13 File Naming Conventions ........................... 13 Subject Identifiers ............................ 13 Filenames................................. 15 Directory Structure............................... 16 Setting up an Analysis Directory....................... 18 Defining the basic directory structure ................. 18 Creating the session-level makefiles................... 19 Creating the common subject-level makefile for each session . 20 Creating links to the session-level makefile............... 21 Creating links to subject-level makefile................. 21 Running analyses............................. 21 Setting Important Variables.......................... 22 ii Contents Variables that control make’s Behavior................. 22 Other important variables........................ 23 Variable overrides............................. 24 Suggested targets............................. 25 3 Running make in Parallel 27 Guidelines for Writing Parallelizable Makefiles ............... 27 Each line in a recipe is, by default, executed in its own shell . 27 Filenames must be unique........................ 28 Separate time-consuming tasks ..................... 29 Executing in Parallel.............................. 30 Using multiple cores ........................... 30 Using the grid engine.............................. 30 Setting FSLPARALLEL for FSL jobs ................... 31 Using qmake ................................ 31 How long will everything take? ..................... 32 Consideration of memory ........................ 32 Troubleshooting make ............................. 33 Find out what make is trying to do................... 33 Use the trace option in make 4.1 .................... 33 Check for line continuation ....................... 34 No rule to make target! ......................... 34 Suspicious continuation in line # .................... 35 make keeps rebuilding targets...................... 35 make doesn’t rebuild things that it should............... 35 4 Recipes 36 Obtaining a List of Subjects.......................... 36 Setting a variable that contains the subject id ............ 37 Using Conditional Statements......................... 37 Setting a conditional flag ........................ 37 Using a conditional flag ......................... 38 Conditional execution based on the environment ........... 38 Conditional execution based on the Linux version........... 39 II Practicals 41 1 Overview of make 43 Lecture: The Conceptual Example of Making Breakfast.......... 43 Practical Example: Skull-stripping...................... 45 iii Contents Manipulating a single subject...................... 45 Pattern rules and multiple subjects................... 47 Phony targets............................... 48 Secondary targets............................. 49 make clean ................................ 50 2 make in Context 52 Lecture: Organizing Subject Directories................... 52 Practical Example: A More Realistic Longitudinal Directory Structure . 53 Organizing Longitudinal Data...................... 53 Recursive make .............................. 55 Running make over multiple subject directories ............ 56 Running FreeSurfer............................ 57 3 Getting Down and Dirty with make 58 Practical Example: Running QMON..................... 58 Practical Example: A Test Subject...................... 59 Estimated total intracranial volume .................. 61 Hippocampal volumes .......................... 63 Implementing a Resting State Pipeline: Converting a Shell Script into a Makefile.................................. 64 4 Advanced Topics & Quality Assurance with R Markdown 68 Creating a make help system ......................... 68 Makefile Directory Structure ......................... 69 The clean target................................ 70 Creating New Makefile Rules On The Fly.................. 71 Incorporating R Markdown into make .................... 71 III Examples 77 1 Downloading Data From XNAT 79 2 Running FreeSurfer 82 3 DTI Distortion Correction with Conditionals 85 4 Quantifying Arterial Spin Labeling Data 90 5 Processing Scan Data for a Single Test Subject 95 Testsubject Main Makefile........................... 95 iv Contents Testsubject FreeSurfer............................. 99 Testsubject Transformations.......................... 101 Testsubject QA Makefile............................ 103 6 Preprocessing Resting State Data 108 7 Generating A Methods Section 114 8 Seed-based Functional Connectivity Analysis I 117 9 Seed-based Functional Connectivity Analysis II 119 Group Level Makefile - tapping/Makefile . 119 Subject Level Makefile - tapping/lib/resting/subject.mk . 120 Preprocessing - tapping/lib/resting/Preprocess.mk . 121 Generation of Nuisance Regressors - tapping/lib/resting/Regressors.mk123 ROI Timeseries Extraction - lib/resting/timeseries.mk . 125 First Level Analysis - lib/resting/fsl.mk . 126 Quality Assurance Reports - lib/resting/qa.mk . 126 10 Using ANTs Registration with FEAT 128 Group Level Makefile.............................. 128 Subject Level Makefile............................. 129 Preparatory Registrations - Prep.mk . 130 Running Feat and Applying ANTs Registrations - Feat.mk . 131 11 Creating Result Tables Automatically Using Make 135 Simple Result Tables.............................. 135 Multiple Group Analyses ........................... 136 12 Plotting Group FEAT Results Against Behavioral Measures 138 Bibliography 146 Other Resources ................................ 147 v List of Figures 1.1 Basic make syntax. ............................. 4 1.2 A very basic make recipe. .......................... 4 1.3 Automatic make variables.......................... 5 1.4 A more realistic example........................... 5 1.5 An expansion of Figure 1.4 ......................... 6 1.6 A makefile expressed in bash ........................ 7 1.7 make filename manipulation functions ................... 9 1.8 How to use subst .............................. 9 1.9 How to use word ............................... 10 1.10 Using strip .................................. 10 1.11 make filename manipulation functions ................... 11 1.12 make filename manipulation functions ................... 11 2.1 Example of good file naming conventions ................. 15 2.2 An example of a project directory. ..................... 17 2.3 A longitudinal analysis directory....................... 19 2.4 Session-level Makefile ............................ 20 2.5 Subject-level makefile ............................ 20 2.6 Creating symbolic links to the session-level makefile ........... 21 2.7 Making test ................................. 21 2.8 Output of recursive make .......................... 22 2.9 Setting $(SHELL) ............................... 22 2.10 Running DTIPrep in make ......................... 24 2.11 Controlling software version......................... 24 2.12 Specifying BET flags in make........................ 25 3.1 A non-functional multi-line recipe ..................... 28 3.2 A now-functional multi-line recipe ..................... 28 3.3 A multi-line recipe using “&&\”....................... 28 3.4 How not to name files ............................ 28 vi List of Figures 3.5 The wrong way to run two long tasks ................... 29 3.6 The right way to run two long tasks .................... 29 3.7 Pattern-matching error handling ...................... 35 4.1 Obtaining a list of subjects from a file.................... 36 4.2 Obtaining a list of subjects using a wildcard. ............... 37 4.3 Determining the subject name from the current working directory and a pattern..................................... 37 4.4 Setting a variable to determine whether a FLAIR image has been acquired. 38 4.5 Setting a variable to indicate the study site................. 38 4.6 Testing the site variable to determine the name of the T1 image. 39 4.7 Testing to see whether an environment variable is undefined . 39 4.8 /etc/os-release .............................. 40 4.9 Testing the Linux version to determine whether to proceed . 40 1.1 Creating a waffwich ............................. 44 1.2 How to make a waffwich........................... 45 1.3 Pattern-matching as a sieve......................... 48 2.1 Script to create symbolic links within a longitudinal directory structure 54 3.1 resting-script file............................. 66 4.1 make’s Help System ............................. 69 4.2 Example of a R Markdown File......................
Load more

Recommended publications
  • Project1: Build a Small Scanner/Parser
    Project1: Build A Small Scanner/Parser Introducing Lex, Yacc, and POET cs5363 1 Project1: Building A Scanner/Parser Parse a subset of the C language Support two types of atomic values: int float Support one type of compound values: arrays Support a basic set of language concepts Variable declarations (int, float, and array variables) Expressions (arithmetic and boolean operations) Statements (assignments, conditionals, and loops) You can choose a different but equivalent language Need to make your own test cases Options of implementation (links available at class web site) Manual in C/C++/Java (or whatever other lang.) Lex and Yacc (together with C/C++) POET: a scripting compiler writing language Or any other approach you choose --- must document how to download/use any tools involved cs5363 2 This is just starting… There will be two other sub-projects Type checking Check the types of expressions in the input program Optimization/analysis/translation Do something with the input code, output the result The starting project is important because it determines which language you can use for the other projects Lex+Yacc ===> can work only with C/C++ POET ==> work with POET Manual ==> stick to whatever language you pick This class: introduce Lex/Yacc/POET to you cs5363 3 Using Lex to build scanners lex.yy.c MyLex.l lex/flex lex.yy.c a.out gcc/cc Input stream a.out tokens Write a lex specification Save it in a file (MyLex.l) Compile the lex specification file by invoking lex/flex lex MyLex.l A lex.yy.c file is generated
    [Show full text]
  • GNU M4, Version 1.4.7 a Powerful Macro Processor Edition 1.4.7, 23 September 2006
    GNU M4, version 1.4.7 A powerful macro processor Edition 1.4.7, 23 September 2006 by Ren´eSeindal This manual is for GNU M4 (version 1.4.7, 23 September 2006), a package containing an implementation of the m4 macro language. Copyright c 1989, 1990, 1991, 1992, 1993, 1994, 2004, 2005, 2006 Free Software Foundation, Inc. Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license is included in the section entitled “GNU Free Documentation License.” i Table of Contents 1 Introduction and preliminaries ................ 3 1.1 Introduction to m4 ............................................. 3 1.2 Historical references ............................................ 3 1.3 Invoking m4 .................................................... 4 1.4 Problems and bugs ............................................. 8 1.5 Using this manual .............................................. 8 2 Lexical and syntactic conventions ............ 11 2.1 Macro names ................................................. 11 2.2 Quoting input to m4........................................... 11 2.3 Comments in m4 input ........................................ 11 2.4 Other kinds of input tokens ................................... 12 2.5 How m4 copies input to output ................................ 12 3 How to invoke macros........................
    [Show full text]
  • GNU M4, Version 1.4.19 a Powerful Macro Processor Edition 1.4.19, 28 May 2021
    GNU M4, version 1.4.19 A powerful macro processor Edition 1.4.19, 28 May 2021 by Ren´eSeindal, Fran¸coisPinard, Gary V. Vaughan, and Eric Blake ([email protected]) This manual (28 May 2021) is for GNU M4 (version 1.4.19), a package containing an implementation of the m4 macro language. Copyright c 1989{1994, 2004{2014, 2016{2017, 2020{2021 Free Software Foundation, Inc. Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.3 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license is included in the section entitled \GNU Free Documentation License." i Table of Contents 1 Introduction and preliminaries ::::::::::::::::: 3 1.1 Introduction to m4 :::::::::::::::::::::::::::::::::::::::::::::: 3 1.2 Historical references :::::::::::::::::::::::::::::::::::::::::::: 3 1.3 Problems and bugs ::::::::::::::::::::::::::::::::::::::::::::: 4 1.4 Using this manual :::::::::::::::::::::::::::::::::::::::::::::: 5 2 Invoking m4::::::::::::::::::::::::::::::::::::::: 7 2.1 Command line options for operation modes ::::::::::::::::::::: 7 2.2 Command line options for preprocessor features ::::::::::::::::: 8 2.3 Command line options for limits control ::::::::::::::::::::::: 10 2.4 Command line options for frozen state ::::::::::::::::::::::::: 11 2.5 Command line options for debugging :::::::::::::::::::::::::: 11 2.6 Specifying input files on the command line :::::::::::::::::::::
    [Show full text]
  • An AWK to C++ Translator
    An AWK to C++ Translator Brian W. Kernighan Bell Laboratories Murray Hill, NJ 07974 [email protected] ABSTRACT This paper describes an experiment to produce an AWK to C++ translator and an AWK class definition and supporting library. The intent is to generate efficient and read- able C++, so that the generated code will run faster than interpreted AWK and can also be modified and extended where an AWK program could not be. The AWK class and library can also be used independently of the translator to access AWK facilities from C++ pro- grams. 1. Introduction An AWK program [1] is a sequence of pattern-action statements: pattern { action } pattern { action } ... A pattern is a regular expression, numeric expression, string expression, or combination of these; an action is executable code similar to C. The operation of an AWK program is for each input line for each pattern if the pattern matches the input line do the action Variables in an AWK program contain string or numeric values or both according to context; there are built- in variables for useful values such as the input filename and record number. Operators work on strings or numbers, coercing the types of their operands as necessary. Arrays have arbitrary subscripts (‘‘associative arrays’’). Input is read and input lines are split into fields, called $1, $2, etc. Control flow statements are like those of C: if-else, while, for, do. There are user-defined and built-in functions for arithmetic, string, regular expression pattern-matching, and input and output to/from arbitrary files and processes.
    [Show full text]
  • Make a Jump Start Together with Software Ag
    Software AG Fast Startup FAST STARTUP PROGRAM MAKE A JUMP START TOGETHER WITH SOFTWARE AG You have great ideas, a clear vision and the drive to build something really cool. You are smart, you are fast, you are innovative and you are determined to become a rock star of the startup community. Need some free software to build your solution on? Here’s the secret. When it comes to software partnership, the key differentiators are: technology leadership, ease of use, total cost of ownership and—most importantly—reliability. Why Software AG? Why not just freeware? The key to your success is a brilliant business idea, a sustainable For a homegrown, non-mission-critical application, there is business model, openhanded investors and finally paying nothing wrong with choosing freeware. But if you need best-in- customers. class functionality and reliability to build your business on, you It’s a strong partnership that can be the differentiator in the should think twice about the software you use. success of your business. What is your innovation? What do you require as the foundation That’s why you should partner with Software AG. for your intellectual property? Why should you spend time re- Building on 45 years of customer-centric innovation, we are inventing what professional developers invented and optimized ranked as a leader in key market categories, year after year. Our over many years for their paying customers? Do you want to grow solutions are faster to implement and easier to use than anything your business or just your development team? How reliable are else on the market.
    [Show full text]
  • The Cool Reference Manual∗
    The Cool Reference Manual∗ Contents 1 Introduction 3 2 Getting Started 3 3 Classes 4 3.1 Features . 4 3.2 Inheritance . 5 4 Types 6 4.1 SELF TYPE ........................................... 6 4.2 Type Checking . 7 5 Attributes 8 5.1 Void................................................ 8 6 Methods 8 7 Expressions 9 7.1 Constants . 9 7.2 Identifiers . 9 7.3 Assignment . 9 7.4 Dispatch . 10 7.5 Conditionals . 10 7.6 Loops . 11 7.7 Blocks . 11 7.8 Let . 11 7.9 Case . 12 7.10 New . 12 7.11 Isvoid . 12 7.12 Arithmetic and Comparison Operations . 13 ∗Copyright c 1995-2000 by Alex Aiken. All rights reserved. 1 8 Basic Classes 13 8.1 Object . 13 8.2 IO ................................................. 13 8.3 Int................................................. 14 8.4 String . 14 8.5 Bool . 14 9 Main Class 14 10 Lexical Structure 14 10.1 Integers, Identifiers, and Special Notation . 15 10.2 Strings . 15 10.3 Comments . 15 10.4 Keywords . 15 10.5 White Space . 15 11 Cool Syntax 17 11.1 Precedence . 17 12 Type Rules 17 12.1 Type Environments . 17 12.2 Type Checking Rules . 18 13 Operational Semantics 22 13.1 Environment and the Store . 22 13.2 Syntax for Cool Objects . 24 13.3 Class definitions . 24 13.4 Operational Rules . 25 14 Acknowledgements 30 2 1 Introduction This manual describes the programming language Cool: the Classroom Object-Oriented Language. Cool is a small language that can be implemented with reasonable effort in a one semester course. Still, Cool retains many of the features of modern programming languages including objects, static typing, and automatic memory management.
    [Show full text]
  • Usability Improvements for Products That Mandate Use of Command-Line Interface: Best Practices
    Usability improvements for products that mandate use of command-line interface: Best Practices Samrat Dutta M.Tech, International Institute of Information Technology, Electronics City, Bangalore Software Engineer, IBM Storage Labs, Pune [email protected] ABSTRACT This paper provides few methods to improve the usability of products which mandate the use of command-line interface. At present many products make command-line interfaces compulsory for performing some operations. In such environments, usability of the product becomes the link that binds the users with the product. This paper provides few mechanisms like consolidated hierarchical help structure for the complete product, auto-complete command-line features, intelligent command suggestions. These can be formalized as a pattern and can be used by software companies to embed into their product's command-line interfaces, to simplify its usability and provide a better experience for users so that they can adapt with the product much faster. INTRODUCTION Products that are designed around a command-line interface (CLI), often strive for usability issues. A blank prompt with a cursor blinking, waiting for input, does not provide much information about the functions and possibilities available. With no click-able option and hover over facility to view snippets, some users feel lost. All inputs being commands, to learn and gain expertise of all of them takes time. Considering that learning a single letter for each command (often the first letter of the command is used instead of the complete command to reduce stress) is not that difficult, but all this seems useless when the command itself is not known.
    [Show full text]
  • The AWK Pattern Processing Language
    000.book.fm Page 531 Wednesday, September 30, 2009 10:13 AM 12 The AWK Pattern Processing Language In This Chapter AWK12Chapter12 is a pattern-scanning and processing language that searches one or more files for records (usually lines) that Syntax . 532 match specified patterns. It processes lines by performing Arguments . 532 actions, such as writing the record to standard output or Options . 533 incrementing a counter, each time it finds a match. Unlike procedural languages, AWK is data driven: You describe the Patterns . 534 data you want to work with and tell AWK what to do with Actions. 535 the data once it finds it. Variables . 535 You can use AWK to generate reports or filter text. It works Functions. 536 equally well with numbers and text; when you mix the two, Associative Arrays. 538 AWK usually comes up with the right answer. The authors of AWK (Alfred V. Aho, Peter J. Weinberger, and Brian W. Control Structures . 539 Kernighan) designed the language to be easy to use. To Examples . 541 achieve this end they sacrificed execution speed in the origi- getline: Controlling Input . 558 nal implementation. Coprocess: Two-Way I/O. 560 Getting Input from a Network. 562 Copyright © 2010 Mark G. Sobell 531531 000.book.fm Page 532 Wednesday, September 30, 2009 10:13 AM 532 Chapter 12 The AWK Pattern Processing Language AWK takes many of its constructs from the C programming language. It includes the following features: • A flexible format • Conditional execution • Looping statements • Numeric variables • String variables • Regular expressions • Relational expressions •C’s printf • Coprocess execution (gawk only) • Network data exchange (gawk only) Syntax A gawk command line has the following syntax: gawk [options] [program] [file-list] gawk [options] –f program-file [file-list] The gawk utility takes its input from files you specify on the command line or from standard input.
    [Show full text]
  • Unix-And-Radiance.Pdf
    UNIX and RADIANCE Giulio Antonutto & Andrew McNeil Unix Terminal 3 Starting the Terminal Application 3 Useful Unix Commands 3 Wildcard Characters 4 Directory Characters 4 Process control 4 Pipes and Redirection 4 Shell Redirection Operators 5 Makefile 5 X11 - X Window Server 7 Launching X11 7 The First Time 7 SCRIPTING 8 HOW TO START 8 HOW TO CONTINUE 9 Create the file ‘run’ 9 Give the proper permissions 9 Run the new tool 9 Future 9 UNIX and RADIANCE 2 Unix Terminal Starting the Terminal Application On a Mac the terminal program can be found in the /Applications/Utilities folder. When launched a window titled “Terminal - Bash - 80 x 24” will appear. This window contains a Unix prompt (which is in some ways similar to a DOS prompt for those familiar with windows). At the prompt you can type commands that will be executed within the terminal window. Useful Unix Commands The following UNIX commands will allow the user to navigate through the file system, manipulate files, check and terminate running processes. Command Description Usage Useful options ls list directory contents ls -l lists additional file info ls folder/ -a lists all files (including hidden files) cd change directory cd folder/folder2/ cd .. more view a text file one page at a time more file.txt cp copy file cp file.txt newfile.txt mv move a file mv oldloc.txt newloc..txt rm remove a file rm file DANGER, DO NOT USE: rm * ps list currently running processes ps top list the currently running processes top -u -s 5 5 and various system statistic UNIX and RADIANCE 3 Command Description Usage Useful options kill kills a running process kill pid pid is the process id number which can be found by using ps.
    [Show full text]
  • Make Tool in Unix
    Make Tool in Unix Prof. P.C.P. Bhatt P.C.P Bhatt OS/M15/V1/2004 1 Introduction ¾ Make is both a productivity enhancement utility as well as a program management tool.. ¾ Individual as well as team productivity enhancement tool. ¾ make tool avoids wasteful re-compilations ¾ It is also useful in the context of software installations. P.C.P Bhatt OS/M15/V1/2004 2 When to use make? ¾ Make can be used in a context where tasks can be expressed as (UNIX) shell commands which need to be executed in the context of certain forms of dependencies amongst file. ¾ Make also helps to determine what definitions the new software should assume during installation. P.C.P Bhatt OS/M15/V1/2004 3 How make works? ¾Makefile structure: the basic structure of makefile is a sequence of targets, dependencies, and commands. ¾Linking with libraries: helps in creating a customized computing environment P.C.P Bhatt OS/M15/V1/2004 4 Makefile Structure P.C.P Bhatt OS/M15/V1/2004 5 Example - 1 We will demonstrate the use of a make file through this simple example. Step1: Create a program file helloWorld.c as shown below: #include<stdio.h> #include<ctype.h> main { printf("HelloWorld \n"); } P.C.P Bhatt OS/M15/V1/2004 6 Example – 1 continues Step2: Prepare a file called “Makefile” as follows: # This may be a comment hello: helloWorld.c cc -o hello helloWorld.c # this is all in this make file P.C.P Bhatt OS/M15/V1/2004 7 Example - 1 Step3: Now give a command as follows: make Step4: Execute helloWorld to see the result.
    [Show full text]
  • Semantics of Programming Languages Computer Science Tripos, Part 1B 2008–9
    Semantics of Programming Languages Computer Science Tripos, Part 1B 2008–9 Peter Sewell Computer Laboratory University of Cambridge Schedule: Lectures 1–8: LT1, MWF 11am, 26 Jan – 11 Feb Lectures 9–12: LT1, MWF 11am, 27 Feb – 6 March Time-stamp: <2009-01-04 22:22:54 pes20> c Peter Sewell 2003–2009 1 Contents Syllabus 3 Learning Guide 4 Summary of Notation 5 1 Introduction 8 2 A First Imperative Language 12 2.1 Operational Semantics . 13 2.2 Typing........................................... 32 2.3 L1:CollectedDefinition ............................... 39 2.4 Exercises ........................................ 41 3 Induction 42 3.1 Abstract Syntax and Structural Induction . 44 3.2 InductiveDefinitionsandRuleInduction. 46 3.3 ExampleProofs...................................... 49 3.4 Inductive Definitions, More Formally (optional) . 61 3.5 Exercises ........................................ 62 4 Functions 63 4.1 Function Preliminaries: Abstract Syntax up to Alpha Conversion, and Substitution . 65 4.2 FunctionBehaviour.................................. 70 4.3 FunctionTyping .................................... 74 4.4 Local Definitions and Recursive Functions . 76 4.5 Implementation..................................... 79 4.6 L2:CollectedDefinition ............................... 82 4.7 Exercises ........................................ 85 5 Data 86 5.1 Products,Sums,andRecords . 86 5.2 MutableStore....................................... 90 5.3 Evaluation Contexts . 94 5.4 L3:CollectedDefinition ............................... 95 5.5 Exercises .......................................
    [Show full text]
  • Debugging Makefiles with Remake
    Debugging Makefiles with remake Rocky Bernstein 1 Remake Autotools[Fou09, Fou10a] is still very popular as a framework for configuring and building open-source software. Since it is a collection of smaller tools, such as autoconf, automake, libtool, and m4, debugging code that it generates can be difficult. When I wrote my first POSIX shell debugger for bash, one of my initial goals was to be able to debug autotools configure scripts, and I was rather pleased when it worked. It required, however, writing a custom bash module to read the 20,000 lines of shell script into an array much faster than bash was able to. (This module has since been incorporated into bash as built-in function readarray.) It was only after completing this task that I realized a POSIX shell debugger was just one part of the bigger problem of debugging autotools script. Here, I describe the next step in that endeavor, adding debugging to GNU Make[Fou10b, Ber11]. We will see how to use remake and a POSIX shell debugger (the one for bash) together. Makefiles have been around for quite a while, and over time, largely through the success of automake, they have gotten more complex. Make can be somewhat opaque, but after writing the debugger component of remake, I can usually solve make problems very quickly and easily. In many programming languages, such as POSIX shell, Perl, Python, Ruby, and Lisp, type expressions or statements have interactive shells to see what happens when they run. Although GNU Make is every bit as dynamic as these other languages, currently there is no such interactive shell.
    [Show full text]