Use Perl Regular Expressions in SAS® Shuguang Zhang, WRDS, Philadelphia, PA
Total Page:16
File Type:pdf, Size:1020Kb
Load more
Recommended publications
-
A First Course to Openfoam
Basic Shell Scripting Slides from Wei Feinstein HPC User Services LSU HPC & LON [email protected] September 2018 Outline • Introduction to Linux Shell • Shell Scripting Basics • Variables/Special Characters • Arithmetic Operations • Arrays • Beyond Basic Shell Scripting – Flow Control – Functions • Advanced Text Processing Commands (grep, sed, awk) Basic Shell Scripting 2 Linux System Architecture Basic Shell Scripting 3 Linux Shell What is a Shell ▪ An application running on top of the kernel and provides a command line interface to the system ▪ Process user’s commands, gather input from user and execute programs ▪ Types of shell with varied features o sh o csh o ksh o bash o tcsh Basic Shell Scripting 4 Shell Comparison Software sh csh ksh bash tcsh Programming language y y y y y Shell variables y y y y y Command alias n y y y y Command history n y y y y Filename autocompletion n y* y* y y Command line editing n n y* y y Job control n y y y y *: not by default http://www.cis.rit.edu/class/simg211/unixintro/Shell.html Basic Shell Scripting 5 What can you do with a shell? ▪ Check the current shell ▪ echo $SHELL ▪ List available shells on the system ▪ cat /etc/shells ▪ Change to another shell ▪ csh ▪ Date ▪ date ▪ wget: get online files ▪ wget https://ftp.gnu.org/gnu/gcc/gcc-7.1.0/gcc-7.1.0.tar.gz ▪ Compile and run applications ▪ gcc hello.c –o hello ▪ ./hello ▪ What we need to learn today? o Automation of an entire script of commands! o Use the shell script to run jobs – Write job scripts Basic Shell Scripting 6 Shell Scripting ▪ Script: a program written for a software environment to automate execution of tasks ▪ A series of shell commands put together in a file ▪ When the script is executed, those commands will be executed one line at a time automatically ▪ Shell script is interpreted, not compiled. -
Lecture 18: Theory of Computation Regular Expressions and Dfas
Introduction to Theoretical CS Lecture 18: Theory of Computation Two fundamental questions. ! What can a computer do? ! What can a computer do with limited resources? General approach. Pentium IV running Linux kernel 2.4.22 ! Don't talk about specific machines or problems. ! Consider minimal abstract machines. ! Consider general classes of problems. COS126: General Computer Science • http://www.cs.Princeton.EDU/~cos126 2 Why Learn Theory In theory . Regular Expressions and DFAs ! Deeper understanding of what is a computer and computing. ! Foundation of all modern computers. ! Pure science. ! Philosophical implications. a* | (a*ba*ba*ba*)* In practice . ! Web search: theory of pattern matching. ! Sequential circuits: theory of finite state automata. a a a ! Compilers: theory of context free grammars. b b ! Cryptography: theory of computational complexity. 0 1 2 ! Data compression: theory of information. b "In theory there is no difference between theory and practice. In practice there is." -Yogi Berra 3 4 Pattern Matching Applications Regular Expressions: Basic Operations Test if a string matches some pattern. Regular expression. Notation to specify a set of strings. ! Process natural language. ! Scan for virus signatures. ! Search for information using Google. Operation Regular Expression Yes No ! Access information in digital libraries. ! Retrieve information from Lexis/Nexis. Concatenation aabaab aabaab every other string ! Search-and-replace in a word processors. cumulus succubus Wildcard .u.u.u. ! Filter text (spam, NetNanny, Carnivore, malware). jugulum tumultuous ! Validate data-entry fields (dates, email, URL, credit card). aa Union aa | baab baab every other string ! Search for markers in human genome using PROSITE patterns. aa ab Closure ab*a abbba ababa Parse text files. -
Shell Scripting, Scripting Examples
Last Time… on the website Lecture 6 Shell Scripting What is a shell? • The user interface to the operating system • Functionality: – Execute other programs – Manage files – Manage processes • Full programming language • A program like any other – This is why there are so many shells Shell History • There are many choices for shells • Shell features evolved as UNIX grew Most Commonly Used Shells – /bin/csh C shell – /bin/tcsh Enhanced C Shell – /bin/sh The Bourne Shell / POSIX shell – /bin/ksh Korn shell – /bin/bash Korn shell clone, from GNU Ways to use the shell • Interactively – When you log in, you interactively use the shell • Scripting – A set of shell commands that constitute an executable program Review: UNIX Programs • Means of input: – Program arguments [control information] – Environment variables [state information] – Standard input [data] • Means of output: – Return status code [control information] – Standard out [data] – Standard error [error messages] Shell Scripts • A shell script is a regular text file that contains shell or UNIX commands – Before running it, it must have execute permission: • chmod +x filename • A script can be invoked as: – ksh name [ arg … ] – ksh < name [ args … ] – name [ arg …] Shell Scripts • When a script is run, the kernel determines which shell it is written for by examining the first line of the script – If 1st line starts with #!pathname-of-shell, then it invokes pathname and sends the script as an argument to be interpreted – If #! is not specified, the current shell assumes it is a script in its own language • leads to problems Simple Example #!/bin/sh echo Hello World Scripting vs. -
Unix (And Linux)
AWK....................................................................................................................................4 BC .....................................................................................................................................11 CHGRP .............................................................................................................................16 CHMOD.............................................................................................................................19 CHOWN ............................................................................................................................26 CP .....................................................................................................................................29 CRON................................................................................................................................34 CSH...................................................................................................................................36 CUT...................................................................................................................................71 DATE ................................................................................................................................75 DF .....................................................................................................................................79 DIFF ..................................................................................................................................84 -
Chapter 10 SHELL Substitution and I/O Operations
Chapter 10 SHELL Substitution and I/O Operations 10.1 Command Substitution Command substitution is the mechanism by which the shell performs a given set of commands and then substitutes their output in the place of the commands. Syntax: The command substitution is performed when a command is given as: `command` When performing command substitution make sure that you are using the backquote, not the single quote character. Example: Command substitution is generally used to assign the output of a command to a variable. Each of the following examples demonstrate command substitution: #!/bin/bash DATE=`date` echo "Date is $DATE" USERS=`who | wc -l` echo "Logged in user are $USERS" UP=`date ; uptime` echo "Uptime is $UP" This will produce following result: Date is Thu Jul 2 03:59:57 MST 2009 Logged in user are 1 Uptime is Thu Jul 2 03:59:57 MST 2009 03:59:57 up 20 days, 14:03, 1 user, load avg: 0.13, 0.07, 0.15 10.2 Shell Input/Output Redirections Most Unix system commands take input from your terminal and send the resulting output back to your terminal. A command normally reads its input from a place called standard input, which happens to be your terminal by default. Similarly, a command normally writes its output to standard output, which is also your terminal by default. Output Redirection: The output from a command normally intended for standard output can be easily diverted to a file instead. This capability is known as output redirection: If the notation > file is appended to any command that normally writes its output to standard output, the output of that command will be written to file instead of your terminal: Check following who command which would redirect complete output of the command in users file. -
Perl Regular Expressions Tip Sheet Functions and Call Routines
– Perl Regular Expressions Tip Sheet Functions and Call Routines Basic Syntax Advanced Syntax regex-id = prxparse(perl-regex) Character Behavior Character Behavior Compile Perl regular expression perl-regex and /…/ Starting and ending regex delimiters non-meta Match character return regex-id to be used by other PRX functions. | Alternation character () Grouping {}[]()^ Metacharacters, to match these pos = prxmatch(regex-id | perl-regex, source) $.|*+?\ characters, override (escape) with \ Search in source and return position of match or zero Wildcards/Character Class Shorthands \ Override (escape) next metacharacter if no match is found. Character Behavior \n Match capture buffer n Match any one character . (?:…) Non-capturing group new-string = prxchange(regex-id | perl-regex, times, \w Match a word character (alphanumeric old-string) plus "_") Lazy Repetition Factors Search and replace times number of times in old- \W Match a non-word character (match minimum number of times possible) string and return modified string in new-string. \s Match a whitespace character Character Behavior \S Match a non-whitespace character *? Match 0 or more times call prxchange(regex-id, times, old-string, new- \d Match a digit character +? Match 1 or more times string, res-length, trunc-value, num-of-changes) Match a non-digit character ?? Match 0 or 1 time Same as prior example and place length of result in \D {n}? Match exactly n times res-length, if result is too long to fit into new-string, Character Classes Match at least n times trunc-value is set to 1, and the number of changes is {n,}? Character Behavior Match at least n but not more than m placed in num-of-changes. -
Bash Guide for Beginners
Bash Guide for Beginners Machtelt Garrels Garrels BVBA <tille wants no spam _at_ garrels dot be> Version 1.11 Last updated 20081227 Edition Bash Guide for Beginners Table of Contents Introduction.........................................................................................................................................................1 1. Why this guide?...................................................................................................................................1 2. Who should read this book?.................................................................................................................1 3. New versions, translations and availability.........................................................................................2 4. Revision History..................................................................................................................................2 5. Contributions.......................................................................................................................................3 6. Feedback..............................................................................................................................................3 7. Copyright information.........................................................................................................................3 8. What do you need?...............................................................................................................................4 9. Conventions used in this -
Introduction to Perl
Introduction to Perl Science and Technology Support Group High Performance Computing Ohio Supercomputer Center 1224 Kinnear Road Columbus, OH 43212-1163 Introduction to Perl • Setting the Stage • Functions • Data Types • Exercises 3 • Operators • File and Directory Manipulation • Exercises 1 • External Processes • Control Structures • References • Basic I/O • Exercises 4 • Exercises 2 • Some Other Topics of Interest • Regular Expressions • For Further Information 2 Introduction to Perl Setting the Stage • What is Perl? • How to get Perl • Basic Concepts 3 Introduction to Perl What is Perl? • Practical Extraction and Report Language –Or: Pathologically Eclectic Rubbish Lister • Created by Larry Wall • A compiled/interpreted programming language • Combines popular features of the shell, sed, awk and C • Useful for manipulating files, text and processes – Also for sysadmins and CGI • Latest official version is 5.005 – Perl 4.036 still in widespread use –Use perl -v to see the version • Perl is portable •Perlis free! 4 Introduction to Perl How to Get Perl • Perl’s most natural “habitat” is UNIX • Has been ported to most other systems as well – MS Windows, Macintosh, VMS, OS/2, Amiga… • Available for free under the GNU Public License – Basically says you can only distribute binaries of Perl if you also make the source code available, including the source to any modifications you may have made • Can download source code (C) and compile yourself (unlikely to be necessary) • Pre-compiled binaries also available for most systems • Support available -
Unicode Regular Expressions Technical Reports
7/1/2019 UTS #18: Unicode Regular Expressions Technical Reports Working Draft for Proposed Update Unicode® Technical Standard #18 UNICODE REGULAR EXPRESSIONS Version 20 Editors Mark Davis, Andy Heninger Date 2019-07-01 This Version http://www.unicode.org/reports/tr18/tr18-20.html Previous Version http://www.unicode.org/reports/tr18/tr18-19.html Latest Version http://www.unicode.org/reports/tr18/ Latest Proposed http://www.unicode.org/reports/tr18/proposed.html Update Revision 20 Summary This document describes guidelines for how to adapt regular expression engines to use Unicode. Status This is a draft document which may be updated, replaced, or superseded by other documents at any time. Publication does not imply endorsement by the Unicode Consortium. This is not a stable document; it is inappropriate to cite this document as other than a work in progress. A Unicode Technical Standard (UTS) is an independent specification. Conformance to the Unicode Standard does not imply conformance to any UTS. Please submit corrigenda and other comments with the online reporting form [Feedback]. Related information that is useful in understanding this document is found in the References. For the latest version of the Unicode Standard, see [Unicode]. For a list of current Unicode Technical Reports, see [Reports]. For more information about versions of the Unicode Standard, see [Versions]. Contents 0 Introduction 0.1 Notation 0.2 Conformance 1 Basic Unicode Support: Level 1 1.1 Hex Notation 1.1.1 Hex Notation and Normalization 1.2 Properties 1.2.1 General -
A Crash Course on UNIX
AA CCrraasshh CCoouurrssee oonn UUNNIIXX UNIX is an "operating system". Interface between user and data stored on computer. A Windows-style interface is not required. Many flavors of UNIX (and windows interfaces). Solaris, Mandrake, RedHat (fvwm, Gnome, KDE), ... Most UNIX users use "shells" (or "xterms"). UNIX windows systems do provide some Microsoft Windows functionality. TThhee SShheellll A shell is a command-line interface to UNIX. Also many flavors, e.g. sh, bash, csh, tcsh. The shell provides commands and functionality beyond the basic UNIX tools. E.g., wildcards, shell variables, loop control, etc. For this tutorial, examples use tcsh in RedHat Linux running Gnome. Differences are minor for the most part... BBaassiicc CCoommmmaannddss You need these to survive: ls, cd, cp, mkdir, mv. Typically these are UNIX (not shell) commands. They are actually programs that someone has written. Most commands such as these accept (or require) "arguments". E.g. ls -a [show all files, incl. "dot files"] mkdir ASTR688 [create a directory] cp myfile backup [copy a file] See the handout for a list of more commands. AA WWoorrdd AAbboouutt DDiirreeccttoorriieess Use cd to change directories. By default you start in your home directory. E.g. /home/dcr Handy abbreviations: Home directory: ~ Someone else's home directory: ~user Current directory: . Parent directory: .. SShhoorrttccuuttss To return to your home directory: cd To return to the previous directory: cd - In tcsh, with filename completion (on by default): Press TAB to complete filenames as you type. Press Ctrl-D to print a list of filenames matching what you have typed so far. Completion works with commands and variables too! Use ↑, ↓, Ctrl-A, & Ctrl-E to edit previous lines. -
Regular Expressions with a Brief Intro to FSM
Regular Expressions with a brief intro to FSM 15-123 Systems Skills in C and Unix Case for regular expressions • Many web applications require pattern matching – look for <a href> tag for links – Token search • A regular expression – A pattern that defines a class of strings – Special syntax used to represent the class • Eg; *.c - any pattern that ends with .c Formal Languages • Formal language consists of – An alphabet – Formal grammar • Formal grammar defines – Strings that belong to language • Formal languages with formal semantics generates rules for semantic specifications of programming languages Automaton • An automaton ( or automata in plural) is a machine that can recognize valid strings generated by a formal language . • A finite automata is a mathematical model of a finite state machine (FSM), an abstract model under which all modern computers are built. Automaton • A FSM is a machine that consists of a set of finite states and a transition table. • The FSM can be in any one of the states and can transit from one state to another based on a series of rules given by a transition function. Example What does this machine represents? Describe the kind of strings it will accept. Exercise • Draw a FSM that accepts any string with even number of A’s. Assume the alphabet is {A,B} Build a FSM • Stream: “I love cats and more cats and big cats ” • Pattern: “cat” Regular Expressions Regex versus FSM • A regular expressions and FSM’s are equivalent concepts. • Regular expression is a pattern that can be recognized by a FSM. • Regex is an example of how good theory leads to good programs Regular Expression • regex defines a class of patterns – Patterns that ends with a “*” • Regex utilities in unix – grep , awk , sed • Applications – Pattern matching (DNA) – Web searches Regex Engine • A software that can process a string to find regex matches. -
Regular Expressions
CS 172: Computability and Complexity Regular Expressions Sanjit A. Seshia EECS, UC Berkeley Acknowledgments: L.von Ahn, L. Blum, M. Blum The Picture So Far DFA NFA Regular language S. A. Seshia 2 Today’s Lecture DFA NFA Regular Regular language expression S. A. Seshia 3 Regular Expressions • What is a regular expression? S. A. Seshia 4 Regular Expressions • Q. What is a regular expression? • A. It’s a “textual”/ “algebraic” representation of a regular language – A DFA can be viewed as a “pictorial” / “explicit” representation • We will prove that a regular expressions (regexps) indeed represent regular languages S. A. Seshia 5 Regular Expressions: Definition σ is a regular expression representing { σσσ} ( σσσ ∈∈∈ ΣΣΣ ) ε is a regular expression representing { ε} ∅ is a regular expression representing ∅∅∅ If R 1 and R 2 are regular expressions representing L 1 and L 2 then: (R 1R2) represents L 1⋅⋅⋅L2 (R 1 ∪∪∪ R2) represents L 1 ∪∪∪ L2 (R 1)* represents L 1* S. A. Seshia 6 Operator Precedence 1. *** 2. ( often left out; ⋅⋅⋅ a ··· b ab ) 3. ∪∪∪ S. A. Seshia 7 Example of Precedence R1*R 2 ∪∪∪ R3 = ( ())R1* R2 ∪∪∪ R3 S. A. Seshia 8 What’s the regexp? { w | w has exactly a single 1 } 0*10* S. A. Seshia 9 What language does ∅∅∅* represent? {ε} S. A. Seshia 10 What’s the regexp? { w | w has length ≥ 3 and its 3rd symbol is 0 } ΣΣΣ2 0 ΣΣΣ* Σ = (0 ∪∪∪ 1) S. A. Seshia 11 Some Identities Let R, S, T be regular expressions • R ∪∪∪∅∅∅ = ? • R ···∅∅∅ = ? • Prove: R ( S ∪∪∪ T ) = R S ∪∪∪ R T (what’s the proof idea?) S.