Linux Systems Administration
Trainer: Ken Marr
LADM (12/02) © Flying Pigs Training Overview
This course gives an introduction to Systems Administration on Linux, a versatile and popular UNIX like operating system used in business, on high-end workstations and on many of the servers on the internet. Linux is easy to learn and use and we expect this course to be a fun and interesting way to master it.
The course includes the installation and configuration of the operating system, installing packages, startup and shutdown, managing users and the network, running backups, the partitioning of file systems and the use of many advanced commands.
Although based on Red Hat (CentoS) Linux, the commands and principles learnt may be applied to other versions of Linux including Ubuntu and SUSE. Aims and Objectives
The main aim is that the delegate leaves the course feeling comfortable with the operating system, ready and able to administer the system on a daily basis.
At the end of this course the delegate will be able to:
describe the main Linux concepts and commands
use the more advanced features of the editors
use the main tools available to the administrator
describe how file permissions are implemented
install Linux and start up and shutdown the system
install and upgrade software packages
create and manage user accounts
backup and recover files and partitions
manage and troubleshoot the network
manage files systems and logical volumes
LADM (12/02) 2 © Flying Pigs Training Schedule
The times shown may be changed as agreed with the tutor:
Start time is 9.30am; end time is 4.15pm approx
Break times, morning and afternoon
Lunch will be for 1 hour at 12.30pm approx
Please ensure that you adhere to the agreed times
For an on-site course, interruptions should be kept to a minimum
If you have a mobile phone, please turn it off now
Your tutor will point out the location of the following:
The men's and women's toilets
The fire exit and the action to be taken in the event of a fire
LADM (12/02) 3 © Flying Pigs Training Getting Value
The course is divided between lecture, demos and exercises. The idea behind this is three fold:
listen and learn see and understand do and remember
A Broad Picture The aim of the course is to provide a broad picture for all but it is useful for us all to be aware of levels of experience. People in the group may have specialised experience that will be helpful to others. It is also worth highlighting particular interests, as the tutor may be able to address them during the course.
Questions Courses are more interesting if there is dialogue, so please feel free to make comments or ask questions.
Practicals In the practical sessions, you are given the opportunity to experiment and consolidate what has been taught during the lectures. Please tell your tutor if you have problems as it is sometimes difficult for them to see that someone is struggling.
We are all individuals, work at different paces and may have special interests in particular topics. Do not be dismayed if you do not appear to complete exercises as fast as the next person.
Hacking Time The exercises are there to give plenty of practical opportunities; they do not have to be finished and you may even choose to focus for a long period on the topic that most interests you. If you have finished early, there is a great deal to investigate. Such hacking time is very valuable; you may not get the opportunity to do this back in the office!
Collaborate Do work with other people during practical exercise sessions. The person next to you may have the answer or you may know the remedy for the problem that your neighbour is having.
The tutor is not the sole provider of answers; indeed, they do not know everything and on occasions may have to find an answer and come back to you.
Contacts The rest of the group may not know you. One of the great benefits of a public course is meeting other people. They may have similar interests, have encountered similar problems and may even have found the solution to yours. The contacts made on the course can be very useful.
LADM (12/02) 4 © Flying Pigs Training Introductions
Before the course begins, your tutor will ask each delegate to introduce themselves to the rest of the class.
In your introduction, please include the following:
Name and company
Job title and responsibilities
Previous experience with the software to be used
Also consider answers to the following questions:
I have come on the course because?
What would I like to gain from the course?
What would I like to be able to do by the end of the course? Pre-requisites
Previous knowledge of UNIX or Linux is a help but is not essential.
LADM (12/02) 5 © Flying Pigs Training Responsibilities
There are many exercises and workshops throughout this course. Please note that collaboration whilst undertaking these is perfectly acceptable.
COOPERATION is encouraged, competition is not
all the exercises may be undertaken in PAIRS
delegates may, if they prefer, work TWO to a terminal
if sharing try to achieve a 50% - 50% SPLIT at the keyboard
During the course, in order to aid the learning experience, the delegates should endeavour to:
try out the course EXAMPLES
attempt to complete each EXERCISE
EXPERIMENT and try out new things
not be afraid to ask QUESTIONS
CONFER with their neighbours
LEAN and be leaned on; the best way to learn is to help others
learn by your (and their) MISTAKES
ENJOY themselves!
LADM (12/02) 6 © Flying Pigs Training Contents
ACCESSING THE SYSTEM ...... 8
COMMANDS AND CONCEPTS ...... 24
DIRECTORY COMMANDS...... 42
FILE COMMANDS ...... 54
FINDING FILES AND JOB CONTROL ...... 69
THE VI EDITOR ...... 91
FINDING TEXT AND SORTING ...... 105
BACKUP COMMANDS ...... 120
FILE PERMISSIONS ...... 131
INSTALLATION ...... 144
PACKAGE MANAGEMENT ...... 149
STARTUP AND SHUTDOWN ...... 156
SERVICE AND PRINTER MANAGEMENT ...... 167
USER MANAGEMENT ...... 177
NETWORK MANAGEMENT...... 188
FILE SYSTEM BACKUP ...... 199
FILE SYSTEM MANAGEMENT ...... 211
LOGICAL VOLUME MANAGEMENT ...... 221
HANDOUTS ...... 229
LADM (12/02) 7 © Flying Pigs Training Accessing the System
Objectives
At the end of this section the delegate will be able to:
describe the history and structure of the operating system
describe the advantages and disadvantages
log on to the system and run some simple commands
change the password and confirm the change
describe the various shells available
LADM (12/02) 8 © Flying Pigs Training What are UNIX and Linux?
Most people are familiar with versions of Microsoft Windows. These are popular computer operating systems with a simple Graphical User Interface or GUI used both at work and at home. UNIX and Linux are versatile and popular computer operating systems found on many business and scientific machines. They have some advantages over Windows but are often accessed via a command line interface very similar to the DOS prompt on Windows rather than through a GUI interface. Both UNIX and Linux are available from many different sources and in many different flavours.
UNIX Solaris or SunOS is a version available from Sun Microsystems. One advantage is that as well as running on Sun’s own hardware it can be installed on a laptop or PC and unlike some UNIX systems it does incorporate a GUI, the Java Desktop Environment.
Linux Linux is a free UNIX like operating system originally developed to run on a PC. Linux has subsequently been developed with the assistance of hundreds of users world-wide and has evolved into many independent distributions. Each includes several different GUI desktops, including Gnome and KDE and popular versions include Redhat, CentOS, Ubuntu and SUSE.
Live CD, DVD, USB Drive
Both UNIX and Linux are available as a compilation of free GNU software that runs completely from a CD, DVD or a USB pen drive. These systems automatically recognise and support many types of graphic card, sound card, network card, SCSI device and other peripherals and allow the user to try out the system without having to install.
Cloud Computing
Many companies now offer server instances ‘in the cloud’. For example Amazon Elastic Compute Cloud (Amazon EC2) is a web service that provides resisable computing capacity. It is designed to make web-scale computing much easier for developers.
Amazon EC2’s simple web service interface allows the user to obtain and configure a Linux server with minimal effort. It provides complete control of computing resources and reduces the time required to obtain and boot new server instances from hours to minutes.
LADM (12/02) 9 © Flying Pigs Training Logging On
A command-line connection to the server can be made directly if access to the console is available by opening a terminal window. If accessing the server remotely from a desktop PC running Microsoft Windows a Secure Shell, ssh, may be used. Free versions available for download include Putty and SSH Secure Shell. In either case the user is prompted to log in with a user name and password. For example, for the user train1 with a password: Login: train1 Password: ******
Exercise
Log onto the server using the emulation software, user name and password allocated by your tutor.
Note: Although the course notes use the user name train1, the actual user name allocated by your tutor may be different. The Prompt
If the user name and password are valid a session is started and the default prompt, which may be similar to the following, will appear:
$ or bash-3.00$
Once logged in the system is case sensitive and all user names and commands are in lower case. However, file names and passwords may be in mixed case. For example the files bye and BYE are two different files.
When creating files and folders it is best not to use spaces in the filename. For example, the file name Red Hat is a valid Windows filename, but a user might have problems accessing it from the command line in UNIX or Linux. Rename the file to redhat to resolve this problem.
The user will normally be using either the bash or the korn shell. A shell is a command interpreter that sits between the user and the server. It reads from the keyboard, interprets what is typed and displays its output on the screen.
Every system has a system administrator user called root. This user should be used with care as the root user has permissions to delete any file. Note that the default prompt for the root user contains a hash (#) symbol.
LADM (12/02) 10 © Flying Pigs Training Command Syntax
Most commands have the general syntax: command options filenames where each element is separated by a space or spaces and the options and filenames are optional.
Command Alone
For example, a command may be entered on its own, without options or filenames. The command ls may be used to list files: ls Your home directory should include files similar to the following: alias bye BYE linux logfile nice oldfile passwd people temp text1 Note that file names are case sensitive. Unlike many other operating systems such as DOS and Windows, UNIX and Linux systems are case-sensitive. All commands are in lower-case. For example, the command ls lists file names but there is no command LS. Therefore it is best to always work in lowercase.
Command With Options
One or more options may be used to modify the behaviour of the command. These options often take the form of a hyphen followed by a letter. For example, this command lists files in a long format: ls -l This lists those files in the home directory whose names start with a full stop (.). ls -a and this lists both: ls -al or ls -la or ls -a -l On some systems useful commands such as ll, which runs the command ls –l, are available.
LADM (12/02) 11 © Flying Pigs Training Command With Filename
One or more filenames may also be specified with many commands and both options and filenames may be specified together: ls -l oldfile text1 uname - Where Am I?
The commands uname and hostname may be used to find out information about the system being used. For example, try the following: uname -a hostname id - Who Am I?
The command id may be used to find out your user and user id.
For example, try the following: id Associated commands include whoami and who am I. who - Who is Logged On?
The command who may be used to find out who else is logged on to the server.
For example: who Associated commands include w and finger.
Exercise
Try out some of the above simple commands.
LADM (12/02) 12 © Flying Pigs Training date - What is the Date/Time?
The command date may be used display the date and time.
For example: date Note that the options for this command do not take the form of a hyphen followed by a letter. For example: date +%m%d%H%M cal – Days and Months
The command cal may be used display a calendar.
For example: cal cal 9 1752 echo - Displaying a Message
The command echo may be used to display output on the screen.
For example: echo Hello there banner - Displaying a Message
On some systems the command banner may also be used to display output on the screen.
For example on UNIX: banner hello
On Linux use: figlet hello
Exercise
Try out some of the above simple commands.
LADM (12/02) 13 © Flying Pigs Training man - Getting Help
The command man may be used to invoke the online manual pages for a given command.
For example, to find out the available options for the command ls:
man ls
Commands available within man include:
View the next page press space bar
Move down a line press return
Go forward a page press f
Go back a page press b
/-a search for a string
n find next occurrence
View the help screen press h
Exit press q
To show all commands with the word edit in their description use:
man -k edit
Associated commands include whatis and which.
Note that on most Linux systems, the help and info commands are also available: help cd
info ls exit – Leaving the System
To end a user session use the command: exit or the alternative, CONTROL & D.
LADM (12/02) 14 © Flying Pigs Training Important Keys
The following are some important keys:
Backspace erase the character before the cursor
Control & C Interrupt a hung command
Function Keys not normally used
Number keys number pad keys will work with Num Lock on stty – Showing Keys
Modifications can be made to the working environment by using the command stty. It is commonly used to set input/output mapping and special keys.
The following command will give a full list of settings:
stty -a passwd - Changing the Password
The command passwd, note the spelling, may be used to change your password.
To do this, enter the command:
passwd
When prompted enter your old password, and then your new password.
Try both a short password of only three characters and one of six alphabetic characters. What is the result? A new password must be of six or more characters and contain either numbers or special characters.
Exercise
Alter your password and test the change by logging out and logging in again.
LADM (12/02) 15 © Flying Pigs Training History of UNIX
UNIX was originally developed in 1969 at AT&T Bell Laboratories in America and was written in PDP-7 assembler code, It was later re-written in the C programming language and this made UNIX quite unique in that now it was written in a high-level language it was portable to different computer platforms. This version developed into what we now know as System V Release 4, SVR4. During the mid-seventies, UNIX was licensed almost free of charge by AT&T to universities in source code form. Many enhancements were incorporated, particularly by the University of California at Berkeley (UCB), which subsequently released its own version of UNIX. This version is known as the Berkeley Standard Distribution or BSD. Microsoft created XENIX, a UNIX-like operating system for Intel x86 microprocessors but decided that the future of XENIX was limited and sold the rights to the Santa Cruz Operation who now market SCO Unix. Many other computer manufacturers acquired UNIX and developed their own proprietary versions. Hewlett Packard and IBM produced their own version of SVR4, HP-UX and AIX and Sun Microsystems successfully marketed their version of BSD, SunOS, now known as Solaris. One of the problems with UNIX is that there are so many different flavours. Today, in an effort to provide some sort of conformity, most are based on a mix of SVR4 and BSD. They are very similar at the user level and if a particular command does not exist there is usually an alternative available. However, there are significant differences between them at the system administration level.
LADM (12/02) 16 © Flying Pigs Training History of Linux
Linux is an operating system that evolved from a kernel created by Linus Torvalds when he was a student at the University of Helsinki. He was using a version of the UNIX operating system called 'Minix' but wanted to create his own operating system that would take into account users' comments and suggestions for improvements.
In essence, Linus had a kernel but no programs of his own, Richard Stallman and GNU had programs but no working kernel. By combining the two Linux was born. Due to the physical distances involved, the means used to get Linus' kernel together with the GNU programs was the Internet. The Internet would also be crucial in Linux's subsequent development as the means of coordinating the work of all its’ developers.
Developers began writing drivers so that different video cards, sound cards and other gadgets inside and outside any computer work with use Linux. But for Linus Torvalds to start with anyway, Linux was just a hobby. Later on, companies like Red Hat made it their goal to bring Linux to the point where it could be installed just like any other operating system; by anyone who can follow a set of simple instructions, and they have succeeded.
Today, Linux is enjoying a favourable press. This comes from the fact that Linux has proven to be a tremendously stable and versatile operating system, particularly as a network server. When Linux is deployed as a web server or in corporate networks, its down-time is almost negligible. There have been cases when Linux servers have been running for more than a year without re-booting and then only taken down for a brief period for routine maintainance. Its cost effectiveness has sold it more than anything else. Linux can be installed on a home PC as well as a network server for a fraction of the cost of other companies' software packages.
More reliability and less cost - it's ideal. If you're reading this, you're obviously here to learn how to use Linux. Any learning experience means opening up to new ideas and new ways of doing things. As mentioned before, Linux is in the UNIX family of operating systems. UNIX is primarily designed to be used by professionals. You will have to learn some UNIX concepts, but that doesn't mean that Linux is a professionals only operating system, quite the contrary. Most major versions of Linux are designed to be as user-friendly and as easy to install as any other operating system on the market today.
LADM (12/02) 17 © Flying Pigs Training Advantages
Multitasking The system is designed to support many tasks running at once, with each process sharing time on the computer’s CPU. These tasks fall into two main types: foreground tasks, such as editing, where the input is from the keyboard and the output is displayed on the screen
background tasks, such as printing or backing up files, which run on the server disconnected from the users screen
Multiuser The system is also designed to support many users at once, with each running many processes.
Flexible Utilities may be used as commands themselves or may be combined to form more complex commands. This combined with the fact that all peripheral devices, such as terminals, printers, disc and tape drives, are treated as simple files makes the system very flexible.
Secure Log on and password facilities are provided and control access to a user’s resources through file permissions. This allows users and applications to access files and peripherals in a controlled and secure manner.
Networked Networking is provided, the standard protocol for both Local and Wide Area Network environments being TCP/IP. All systems have always provided mail facilities. Most of the distribution and routing of e-mail through the Internet is performed by UNIX or Linux servers.
LADM (12/02) 18 © Flying Pigs Training Features Some two hundred utility programs are provided with the system. Many more applications are available via the Internet. For example, for Solaris many free utilities are available from www.sunfreeware.com
Portable The system is written in ‘C’, which is a third generation programming language available on virtually all computer systems. This means that it can run on a wide range of platforms, including Intel x86-compatible PCs, RISC and supercomputer processors.
Disadvantages
Not User Friendly The system is very flexible. Commands typically perform some small task and may be combined to perform a more complex task. However, it was written by technicians and commands rarely display succinct error messages. Most users perceive this as unfriendly.
No GUI Interface UNIX is basically a command line operating system with limited access to the GUI interface. For this reason is not as easy to learn as its Microsoft Windows counterparts. However, many UNIX versions, including Solaris from SUN, overcome this problem by supplying a default GUI such as the Java Desktop Environment. All versions of Linux come with a default GUI, Gnome being the most popular.
LADM (12/02) 19 © Flying Pigs Training Basic Structure
The structure of the operating system may be viewed as a series of layers.
The Hardware The computer hardware itself is the first layer.
The Kernel The kernel is the very core of the system. The kernel performs low-level operations such as scheduling and memory allocation and interacts between the hardware, the discs and printers etc., and the shell. The kernel contains a fixed number of commands and procedures and can be thought of as the skeleton of the system.
The Shell The shell is a utility program which users access when they log in. It is a command interpreter that sits between the user and the operating system kernel and provides the command prompt from which the user can run commands. It interacts between the kernel and the user, reading input from the keyboard and displaying output on the screen. If the kernel can be thought of as the skeleton then the shell is the flesh added to this skeleton.
There are many shell programs available and they can be tailored to provide a customised user interface. Shell commands can also be stored in a file or shell script for later execution.
The User The final layer is the user. The X Windows System, if installed, sits between the user and the shell and supplies a GUI which has a desktop similar to Windows. ps - Which Shell Am I Using
The command ps may be used to find out what default shell is currently running for the user. For example: ps
LADM (12/02) 20 © Flying Pigs Training The Different Shells
The following are the major shells available on modern systems:
User Bash Shell Korn C
Kernel Bourne
Hardware
Bourne Shell
The Bourne shell is the original shell available on all servers. Unfortunately this shell cannot be customised to any real extent and for this reason is little used except by the system administrator user root. C Shell
The C shell originates from Berkeley University and is similar to the Bourne shell where basic commands are concerned. However, this shell may be customised and contains facilities such as a command history, command aliases and job control capabilities.
By default, the prompt is a % sign. For example, to run a C shell:
csh ps
Note that there are significant differences between the SVR4 Bourne shell and the BSD C shell such that many Bourne shell scripts will not run in the C shell and vice-versa.
LADM (12/02) 21 © Flying Pigs Training Korn Shell
The Korn shell is very similar to the Bourne shell. However, it contains many of the facilities contained in the C shell and is widely used today.
This shell may be customised and also contains facilities such as a command history, command aliases and job control capabilities.
By default, the prompt is a $ sign but it may be customised in this shell.
For example, to run a Korn shell:
ksh ps
Note that in the Korn shell, the arrow keys may not be used at the command prompt.
Bash Shell
The Bash is the newest shell and is very similar to the Korn shell. On Linux systems it is the default and it is now available for both Solaris and AIX.
This shell may be customised and also contains facilities such as a command history, command aliases and job control capabilities.
However, its biggest advantage is that by default, the arrow keys may be used to recall and edit commands.
The prompt may also be customised.
For example, to run a Bash shell:
bash ps
LADM (12/02) 22 © Flying Pigs Training Which Shell to Use?
There are many other shells available including posix, rsh, zsh and tcsh.
If available, the Bash shell is the easiest shell to use when learning. This course covers both the Korn and Bash shells and points out the simple differences between them.
Handouts
The section at the end of the notes contains a summary of many of the commands covered during the course.
LADM (12/02) 23 © Flying Pigs Training Commands and Concepts
Objectives
At the end of this section the delegate will be able to:
display processes running on the system
display files and directories
describe the concepts of piping and redirection
use the switch user commands
LADM (12/02) 24 © Flying Pigs Training Processes and Files
On UNIX and Linux servers, something is usually either a file or a process.
Processes A process may be defined simply as a running program such as the default shell.
The command ps may be used to list the processes running for the user. For example: ps However, the command ps may also be used to find out what processes or daemons are currently running on the server.
For example:
ps -ef
Process Execution
Whenever a command is entered at the prompt, it is the shell which interprets what has been typed, substituting appropriate values and/or file names as necessary. Once the initial evaluation of a command line is complete, the shell will attempt to find the command to be executed. The shell looks in two places:
Hash table This is held in memory and contains a list of commands which have previously been executed in the current session.
PATH variable This environment variable is checked and each directory is searched in turn for the specified command.
Whenever a new shell is generated a new hash table is created. Also, if the PATH variable is changed, the hash table will be reset. The contents of the hash table may be displayed as follows:
hash
LADM (12/02) 25 © Flying Pigs Training The contents of the PATH variable can be displayed thus:
echo $PATH
If no command is found then the shell returns the appropriate error. If a command is located then it is loaded into memory and executed. This new program becomes known as a child process, and the original shell is the parent. Once the child process terminates, the parent process then resumes control.
Files Many different things are treated as files. Simple files include text and binary files. Directories are treated as special files which contain other files and directories.
However, an attached device or peripheral such as a printer, disc drive, keyboard or terminal is also treated as a file.
Note that file names can include ANY characters including a space, but are best restricted to numbers, letters, full stop and underscore.
LADM (12/02) 26 © Flying Pigs Training System Directories
The file system has an upside down tree like structure and a number of special directories are always present on the system. The root directory, /, is at the top:
The standard set of directories will include of the following:
/ root, the parent of all directories
/bin program files
/dev special device files
/etc system configuration and administration files
/lib shared executable and procedural libraries
/sbin system administration utilities
/tmp temporary files for users and applications
/home home directories for users on Linux
/usr user files and programs
/var print and mail files
LADM (12/02) 27 © Flying Pigs Training Listing Directories
The command ls may be used to list files in the current directory:
ls
It may also be used to list other directories, for example the root directory:
ls / or to list the contents of any of these system directories or sub-directories, for example
ls /etc
ls -l /dev
ls /usr/lib
Other options to the ls command include the following:
List sizes in human readable format:
ls -lh
List all files and sub-directories:
ls -R
List in reverse order:
ls -lr
Exercise
Try out some of the above commands.
When using ls –l the default is to display the details of a directories’ content. If details of the directory entry are required then use -d option as follows:
ls -l temp
ls -ld temp
LADM (12/02) 28 © Flying Pigs Training cat, more - Displaying Files
The command cat may be used to list the contents of files. However, since the output scrolls across the screen until the end of the file is reached, it is best used only for small files.
For example:
cat BYE people
cat linux
The command more may also be used to list the contents of files. However, unlike the cat command, this command holds the output on the screen until the user chooses to display more of the file or to quit.
For example:
more BYE people
more linux
Commands available within more include:
View the next page press space bar
Move down a line press return
Go forward a page press f
Go back a page press b
/Unix search for a string
n find next occurrence
N find previous occurrence
View the help screen press h
Exit press q
Associated commands include pg and less.
LADM (12/02) 29 © Flying Pigs Training wc - Counting lines in Files
The command wc may be used to count the number of lines, words and characters in a file.
For example, try:
wc linux
The command may also take the arguments:
-l number of lines
-w number of words
-c number of characters
Now try:
wc -l linux nl - Numbering Files
The command nl may be used to list the contents of a file in numbered form. For example:
nl linux
The -ba option to the nl command numbers all lines including those that are blank.
nl -ba linux
Exercise
Use the commands cat and more to view the file logfile.
Use the command nl to produce a numbered list of the file logfile, including blank lines.
Count how many lines there are in the file logfile.
LADM (12/02) 30 © Flying Pigs Training file - Displaying File Types
The commands to display the contents of a file should only be used on files containing text. Files which are in a binary format should not be displayed.
To illustrate this try the following:
cat linux
cat nice
The last command was an attempt to display a binary file.
The command file may be used to find out the given type of a file.
For example:
file linux
file nice
If the file type contains the words text, commands or English then it may safely be displayed.
LADM (12/02) 31 © Flying Pigs Training Piping - using | symbol
Piping is the process of sending the output of one command as input into another command and is one of the most powerful tools available.
The syntax is:
command | command | command
The command ls may be used to list the contents of a directory. However, if the output is large it will scroll across the screen until the end of the directory is reached.
For example:
ls -l /etc
The technique of piping may be used to keep the output of the command on the screen until the user chooses to display more of the file or to quit.
Here, the output from the ls command is piped into the command more:
ls -l /etc | more
Note that if the command less is used instead of more, the user can move back and forth within the output by use of the arrow keys.
Other commands may also be used with a pipe. For example:
man -k edit | more
Exercise
Display a long listing of all files in the /usr/bin directory.
Now display the same listing of the /usr/bin directory but use a pipeline to hold the output on the screen.
Display a list of all processes running on the server using the command ps and hold the output on the screen.
LADM (12/02) 32 © Flying Pigs Training Redirection - using > symbol
Redirection allows the output of a command to be saved in a file. The file will be created if it does not exist or overwritten if it does. This too is one of the most powerful tools.
The syntax is:
command > filename
The command ls may be used to list the contents of a directory. As we have seen, if the output is large it will scroll across the screen until the end of the directory is reached. It is then lost.
For example:
ls -l /dev
The technique of redirection may be used to save the output of the command in a file and the user may then choose to display the file at a later time.
Here, the output from the ls command is redirected to a file.
ls -l /dev > devfile
more devfile
Exercise
Display a list of all processes running on the server.
Now save a list of all processes on the system in a file called psfile.
Display a list of all users logged on to the system.
Now save a list of all users in a file called whofile.
LADM (12/02) 33 © Flying Pigs Training cat - Concatenating Output
The command cat may be used with redirection to concatenate or join together several files into one file.
For example, to create a file containing the contents of 3 other files:
cat BYE oldfile people > newfile Redirection - using >> symbol
Redirection also allows the output of a command to be appended to the end of a file. The file will be created if it does not exist.
The syntax is:
command >> filename
Here, the output from the echo command is appended to the end of the file devfile:
echo End of file >> devfile
LADM (12/02) 34 © Flying Pigs Training tail – End of File Display
We have seen that the commands cat and more may be used to list the contents of a file. The associated command tail may be used to list a set of lines from the end of a file.
For example, to list the last 10 lines of the file:
tail devfile
To list only the last line of the file:
tail -1 devfile
To “follow” a file as it changes use the -f option to the tail command. head – Start of File Display
The command head may be used to list a set of lines from the start of a file.
To list the first 10 lines of the file:
head devfile
To list only the first line of the file:
head -1 devfile
Exercise
List the first 20 lines of the logfile file.
Append the message End of File to the file logfile.
Now list the last 20 lines of the logfile file.
LADM (12/02) 35 © Flying Pigs Training Part of File Display
The commands head and tail may be used with a pipe to list any set of lines from a file.
For example, first display a numbered list of lines in the passwd file:
nl passwd
Then, display the first 10 lines of the file:
nl passwd | head -10
Now to display only lines 6 to 10 use:
nl passwd | head -10 | tail -5
As a second example, display a numbered list of lines in the file linux:
nl linux
nl -ba linux
The -ba option to the nl command numbers all lines including those that are empty.
Now extract only lines 97 to 107 from the file:
nl -ba linux | head -107 | tail -11 > nlinux
Exercise
Create a new file, nlfile, which contains lines 3114 to 3124 from the file logfile.
LADM (12/02) 36 © Flying Pigs Training tee - Combining Redirection and Piping
Redirection allows the output of a command to be saved in a file. But the output is no longer displayed on the screen.
The command tee may be used to both save the output of the command in a file and to display it on the screen. To append to a file use the option -a.
Here, the output from the ls command is piped into the tee command which redirects the output to a file.
ls -l /usr/bin | tee teefile | more
Exercise
List the files in the /etc directory user showing details such as permissions, owner, group and date and time updated.
Now list the same files on the screen and save the output in a file called listfile.
Protecting Files - noclobber
The command set may be used to prevent the overwriting of an existing file when using the > or >> signs. For example, try the following:
set -o noclobber echo Clobber file > teefile
Reset the noclobber option with:
set +o noclobber
To view the full list of options enter:
set -o
LADM (12/02) 37 © Flying Pigs Training Script – Saving the Session
The command script may be used to save a compete log of all commands, and their output, entered during a session.
For example, to script to a file called log:
script log
who
cat people
ls /etc
exit
Now view the file log:
more log Two Commands at Once - ;
Two or more commands may be entered at once if they are separated on the command line by a semicolon.
For example:
ls ; who Multi-line Command /
A command may be continued on a second line as follows:
nl -ba linux | head -107 | tail \ -11 > nlinux
LADM (12/02) 38 © Flying Pigs Training su - Switching Users
Logging in and out to change users can be inconvenient. To avoid this the command su may be used to switch users. For example, to switch to user traina: su - traina To switch to user root (in Solaris 11 this requires the root role): su - When prompted enter the password. When the command exit is entered, the user is returned to the original login session. exit Note that if the hyphen is not used with this command, as in: su traina this is not the same as logging in as the new user. This format of the command does not set the new user environment but retains some of the settings of the original user.
Exercise
Switch into the user traina and list the files there.
LADM (12/02) 39 © Flying Pigs Training sudo – Commands as Root
Rather than switching into the user root, the sudo command allows a permitted user to execute a command as the superuser provided:
this is specified in the /etc/sudoers file
## Allows people in group wheel to run all commands %wheel ALL=(ALL) ALL
they are a member of the wheel group in /etc/group
wheel:x:10:root,train1
By default, sudo requires that users authenticate themselves with a password (by default this is the user's password, not the root password). Once a user has been authenticated, a timestamp is updated and the user may then use sudo without a password for a short period of time. sudo – Example
The command more may be used to list the contents of file /etc/passwd, a special file which contains an entry for every user login on the system.
However, this file does not contain encrypted passwords. These are held in a restricted file which cannot be viewed by a user other than root: cat /etc/shadow The sudo command, if available, may be used to view the file: sudo cat /etc/shadow
LADM (12/02) 40 © Flying Pigs Training Piping and Redirection
In summary, piping is the process of sending the output of one command as input to another command, which may in turn pass it's input to another command:
command | command | command
Redirection allows the output of a command to be saved in a file:
command > filename
Data can also be appended to the end of a file:
command >> filename
However, if the two are combined, redirection must be the last thing on the command line:
command | command | command > filename
LADM (12/02) 41 © Flying Pigs Training Directory Commands
Objectives
At the end of this section the delegate will be able to:
find out their home and working directory
change directories to move around the file system
customise the prompt and create an alias
customise the user environment
create and remove directories
LADM (12/02) 42 © Flying Pigs Training Home Directory - HOME
The server uses a set of upper case system variables, rather like pidgin holes, to store information. The complete list can be viewed as follows:
set | more
The name of a user’s home directory is stored in the system variable, HOME; note that this is in upper case.
To view this variable enter:
echo $HOME or echo ~
Working Directory - pwd
The command pwd may be used to find out which is the current WORKING directory. When a user logs in, this is their HOME directory as defined in the /etc/passwd file:
cat /etc/passwd pwd cd - Changing Directories
The command cd may be used to change the current WORKING directory. For example, to move to the root directory:
cd /
pwd
To move to the directory /etc:
cd /etc
pwd
To move back to the previous use the command:
cd -
To move back to the HOME directory, use the command cd without any options:
cd
LADM (12/02) 43 © Flying Pigs Training Command Prompt - PS1
The default command prompt is stored in a system variable, PS1; note this is entered in upper case. To view the variable enter:
echo $PS1
Some shells, but not the Bourne and C shells, allow the prompt to be customised. In the Bash and Korn shells it may be set to the full path name of the current directory, a concept familiar to DOS users. To do this, carefully type the following:
PS1=’$PWD: ‘ and note how the prompt changes. If the prompt does not change, use the ps command to check which is the current shell.
Now try the following:
cd /
cd /etc
cd
cd temp
In the Bash shell the following special characters may also be used:
\w current working directory, same as $PWD
\h@\u host and user name
\d@\t current date and time
\s-\v current shell and version
For example:
PS1='\u@\h \w '
Exercise
1. Change the prompt PS1 as in the above example.
LADM (12/02) 44 © Flying Pigs Training Secondary Prompt - >
The secondary prompt is also stored in a system variable, PS2, also in upper case. To view this variable enter:
echo $PS2
This prompt may be seen if an incomplete command is entered, for example:
echo “hello there where the second quote symbol is missing. To correct this situation, complete the command or enter CONTROL & C and re-enter the command.
Files or Directories?
When the contents of a directory are displayed, it is not always obvious if the entries displayed are files or directories. With the option -F, the entries displayed are followed by an extra character. For example, try the following: ls ls -F
ls / ls -F /
This extra character may include the following:
for a directory /
for a linked file @
for an executable file *
Solaris 11 and Linux systems also support colour coding of directories and files. Try the following (use a double hyphen):
In Solaris 11: /usr/gnu/bin/ls - -color=auto In Linux: ls - -color=tty
LADM (12/02) 45 © Flying Pigs Training Aliases
On some systems useful commands such as lsf, which runs the command ls –F and ll which runs the command ls –l are available. However, if these commands are not available, an alias can be created to achieve the same result. An aliases can be created for any frequently used command. This saves having to type the full command and its options. For example, for Unix try: alias ls=’ls -F’ or if using Linux try:
alias ls=’ls - -color=tty’
The command is then run as:
ls /
However, the command name does not necessarily have to be used for the alias name. For example: alias la=’ls -a’ can be run as: la If there are only a few users logged on, the following alias may be used to simulate the command who:
alias who=’cat /scr/whofile’
who
Exercise
1. Create and test several new aliases as in the above examples.
Using Aliases
To display a list of command aliases, use the alias command: alias A command can be invoked without the use of the alias by prefixing the command with a backslash, \ . \ls /
To remove one or more command aliases use:
unalias ll la
LADM (12/02) 46 © Flying Pigs Training Customising the Shell
One problem new users encounter is that any settings or aliases created during a session are lost when the user logs out. This problem can be overcome by saving the settings in a set up file.
Both the Bash and Korn shells store variables and aliases permanently in the hidden file .profile which is executed at login.
Bash Shell When using the Bash shell, the hidden file .bash_profile, if it exists, will be used in preference to the file .profile on login. The bash shell may also use the file .bashrc which is run every time a Bash shell is invoked.
Korn Shell The korn shell may use another file, for example .kshrc, if the environment variable ENV is set to the file name.
The simple editor nano, if available, may be used to edit hidden set up files.
Exercise
1. With the help of your tutor, edit the hidden file in use and alter the prompt as follows:
PS1=’$PWD: ‘
Add new aliases to the file, for example:
alias who=’cat /scr/whofile’ alias ls=’ls -F’ or alias ls=’/usr/gnu/bin/ls - -color=auto’ or alias ls=’ls –color=tty’
2. Logout and login again to test the changes to the file.
. – dot Command
As an alternative to logging out and in again, the file may be tested thus:
. .bashrc
The dot (.) command runs the contents of the file in the current shell.
LADM (12/02) 47 © Flying Pigs Training type - Type of command
The type command may be used to determine the type of a command. This might be a keyword, a built-in, an external or an alias. For example: type if
type ls
type cat Command History
The command history mechanism maintains a list of recently used commands and provides a shorthand for re-executing any of these commands. The number of commands returned is held in the variable HISTSIZE.
To view the list of most recent commands, type:
history
In the Bash shell, to view the last 20 commands, type:
history 20
A command is re-executed by typing an ! followed by the number. For example:
!25
In the Bash shell to do a reverse search through the history file use CTRL&R.
Note that in the Korn shell these commands differ slightly, for example:
history -20
r 25
Line Completion
In the Bash shell the key sequence TAB TAB may be used to complete a file name used in a command provided that an exact match exists. In the Korn shell use the sequence ESC ESC.
LADM (12/02) 48 © Flying Pigs Training mkdir - Make Directory
The command mkdir may be used to create single or multiple new directories.
For example, to create two new directories in your user: cd
mkdir newdir1 newdir2
ls
To create sub-directories or a tree structure, use the –p option. mkdir -p newdir1/subdir1
mkdir -p top/middle/bottom
ls -R
On Linux, to display a message use –v.
Exercise
1. Change to the root directory and attempt to create a new directory newdir1. This will result in an error.
2. Change to your home directory and create the new directory newdir1.
3. List any files in the new directory.
LADM (12/02) 49 © Flying Pigs Training Special Files - . and ..
All new directories contain two special files . and ..
To see the new files enter:
cd newdir
ls -a
These two files are special links to other directories.
The . file is a link to the current working directory. This is especially useful when copying files as in the following example:
cp /scr/logfile .
The .. file is a link to the parent directory of the current working directory. This is useful when specifying path names.
From the current directory, the following may be used to change to the parent directory: cd ..
The following illustrates the use of both special dot files:
cd newdir
ls ..
cp ../text1 .
ls
LADM (12/02) 50 © Flying Pigs Training rmdir - Remove Directory
The command rmdir may be used to remove single or multiple empty directories.
For example, to remove a directory in your user:
cd
rmdir temp
rmdir newdir1
Exercise
1. List all the files in your home directory.
2. Create two new directories, mydir1 and mydir2 and change into the new directory mydir1.
3. List details of all files, including hidden files, in mydir1.
4. In one command, change into the directory mydir2.
Hint: use the characters .. in the path name.
5. Now change back into your home directory and remove the empty directories mydir1 and mydir2.
LADM (12/02) 51 © Flying Pigs Training Using Path Names
As we have seen, files are stored in an upside down tree like structure fanning out from the root, the / directory. The path names used are similar to those used in DOS, except that DOS uses a backslash \ in directory path names and here we use a forward slash /.
Each file has an absolute path name which begins at root with a /.
For example, you can refer to the passwd file in /etc as follows by using the absolute path name:
cat /etc/passwd
A file also has a relative path name which does NOT begin with a / but assumes that the file is below the current working directory.
To use the relative path name of the file passwd enter:
cd /etc
cat passwd
In general, if you do not own a file and are in your home directory, you must refer to the file by using an absolute path name.
If you own a file and are in your home directory, or are in the directory containing the file, you can refer to the file by using the relative path name without the /.
LADM (12/02) 52 © Flying Pigs Training Changing Directories – pushd and popd
A user may need to jump back and forth between two or more directories. One way to do this is to use the command history. The alternative is to use these commands to enable quick traversal between directories.
The commands pushd and popd work based on the concept of a stack and the user can cd into directories using pushd, which adds the new directory to the stack and return to previously visited directories using popd. An example is shown below. First create a simple directory structure:
cd
mkdir -p parent/child
Next use the pushd command to change directories and build the stack:
pushd parent
pushd child
The above shows a change from the home directory to the parent and then to the child directory.
With each pushd command the stack is displayed as the command prompt with the most recently executed target on top (top being left most). The stack can also be viewed as follows:
dirs -v
Now use pushd to navigate between the two directories:
pushd
pushd
pushd
Use popd to remove entries from the stack:
popd -1
Also use popd to navigate back up the stack:
popd
LADM (12/02) 53 © Flying Pigs Training File Commands
Objectives
At the end of this section the delegate will be able to:
copy, move and remove files
use wild cards in file name generation
LADM (12/02) 54 © Flying Pigs Training cp - Copying Files
The command cp may be used to make a copy of a file. The original file remains unchanged.
For example, to copy a file:
cd
cp text1 fish
cp text1 shark
ls
Multiple files may be copied but this must be into a directory:
cp fish shark text1
mkdir newdir
cp fish shark newdir
ls newdir
The following illustrates the use of the special dot files when copying files:
cd newdir
cp /scr/logfile .
cp ../text1 .
LADM (12/02) 55 © Flying Pigs Training Interactive Copy - cp -i
If a copy command is rerun it will overwrite the output file. The option -i may be used to prompt the user if a file will be overwritten by a copy.
For example, the following will create a new file:
cd
cp text1 text2
If the command is rerun:
cp text1 text2 it will overwrite the new file but the -i option will prompt the user:
cp -i text1 text2 Exercise
Copy the files credit and keys from /scr into your home directory.
Create an alias for the cp command which uses the interactive option –i.
Now copy the same two files again.
LADM (12/02) 56 © Flying Pigs Training mv - Moving Files
The command mv may be used to move or rename a file. Here, the original file is removed.
For example, to rename a file:
mv fish shark
ls
Here, if the command is rerun it will fail as the original file no longer exists. Interactive Move - mv -i
As with the copy command the option -i may be used to prompt the user if a file will be overwritten by a move.
For example:
mv -i text1 text2
Multiple files may also be moved into a directory:
mv -i text2 shark newdir
ls newdir Exercise
Rename the file credit as mycredit and the file keys as yourkeys.
LADM (12/02) 57 © Flying Pigs Training Copying Directories
The command cp may also be used to copy the contents of a directory. Again the original files remain unchanged.
For example, to copy a directory:
cp -r temp temp2
ls
ls temp2 Exercise
1. Copy the directory temp belonging to traina and it’s contents into your own home directory as tempa. Moving Directories
The command mv may also be used to move or rename a directory. Here, the original file is again removed.
For example, to rename a directory:
mv temp2 temp1
ls
LADM (12/02) 58 © Flying Pigs Training touch - Creating Files
The command touch may be used to update the date and time that a file was last updated. It may also be used to create an empty file or files.
For example:
cd newdir
touch file1 file2 file3 file4
ls rm - Removing Files
The command rm may be used to remove a file or files
For example, to remove files:
rm file1 file2
ls
Again, If the command is rerun it will fail as the original file no longer exists.
Exercise
1. Create several new files using the touch command and then remove the files.
LADM (12/02) 59 © Flying Pigs Training Interactive Remove - rm -i
As with the copy and move commands the option -i may be used to prompt the user, in this case to confirm deletion of a file.
For example:
rm -i file3 file4
ls Exercise
Interactively remove the files mycredit and yourkeys.
Check that these new files do not now exist in your home directory.
Removing Unusual Files
Some files with unusual names may be difficult to remove. If a file is created thus:
> -c > ‘my file' ls then the following will fail:
rm -c
However, these files may be removed by using the absolute path name:
rm ./-c rm ./‘my file'
LADM (12/02) 60 © Flying Pigs Training Read Only Files – rm -f
The rm command option -f may be used to suppress the warning message issued if a user tries to remove a file for which they do not have write permissions:
For example:
rm people
rm -f people
Removing Directories - rm -r
The command rmdir may only be used to remove an empty directory.
For example, this command returns an error:
cd
rmdir temp1
However, the rm command with the option -r may be used to remove a directory and all of its contents. The option -i may be used to prompt the user.
For example:
rm -ir temp1
Exercise
1. Create a new directory below your home directory called newtemp.
2. Now list all the files in your directory temp.
3. Copy the files atemp and btemp from temp into the new directory newtemp.
4. List the contents of the directory newtemp.
5. Remove interactively the new directory newtemp and its contents.
6. Check that the new directory has been deleted.
LADM (12/02) 61 © Flying Pigs Training Wild Cards - File Name Generation
The special characters *. ? [ and ] are known as wild cards or meta-characters.
Meta-characters are expanded by the shell to match a list of file names in the current directory.
* matches any number of characters, including none
? matches any single character
[ ] matches characters within a range
[! ] matches characters NOT within a range
Meta-characters may be used in commands to perform operations on several files at once, or just to simplify a command line.
Wild Cards - using *
Change into and list the files in the directory temp:
cd ; cd temp
ls temp
The output should include some of the following files:
a abtemp acfile atemp b btemp cfile filea temp tempab
Now display all file names in the temp directory beginning with an a:
ls a*
Display all file names ending with an p:
ls *p
Display all file names beginning with an a and ending with an p:
ls a*p
LADM (12/02) 62 © Flying Pigs Training Exercise
List all the files in the /scr directory that begin with the character c.
Now list all the files in the /scr directory that contain the characters as.
List all the files that begin with the character s from the /etc directory.
Hint: Without the -d option the ls command displays the contents of directories, not the directory entry itself. Wild Cards with cp
Wildcards may be use to copy multiple files into a directory:
mkdir newdir
cp b* *file* newdir
ls newdir
Wild Cards with rm
Wildcards may also be used to remove multiple files:
rm -i newdir/*
Note that the rm command options -fr may be used to suppress warning messages and remove directories and files.
For example, the following command will remove ALL files and directories, excluding hidden files:
rm -fr *
LADM (12/02) 63 © Flying Pigs Training Wild Cards - using ?
In the directory temp try the following.
Display all files with a five character filename:
ls ?????
Display all file names starting with any character and then the word temp:
ls ?temp
Exercise
List all the files in the /scr directory that begin with the character b followed by any 3 characters and then the characters rc.
List all the files in directory /etc that begin with the character p and have only seven characters in their file name.
LADM (12/02) 64 © Flying Pigs Training Wild Cards - using [ ]
In the directory temp try the following.
Display all file names starting with a, b or c:
ls [abc]* or ls [a-c]*
Display all file names NOT starting with a, b or c:
ls [!a-c]*
Display all file names containing the characters c or f:
ls *[cf]*
Display all files for all delegates:
ls ../../train[1-6x]
Exercise
List all the file names in directory /scr that contain the characters x, y or z.
Change into your directory temp and remove all files beginning with the character a or b.
LADM (12/02) 65 © Flying Pigs Training Word count - wc
The command wc may be used to count the number of lines, words and characters in a file or in its input.
For example, try:
wc *
The command may also take the arguments:
-l number of lines
-w number of words
-c number of characters
Now try:
wc -l *
The command may be combined with a pipeline to count the number of files in the current directory:
ls | wc -l or to count the number of files found in the above example:
ls -d /etc/p?????? | wc -l
Echo with Commands
The command echo may be used to display text on the screen and this may be combined with the output of a command in order to display the number of files found as follows:
echo Number of files found = $(ls -d /etc/p?????? | wc -l)
Any sequence of characters found within the $(…) characters is assumed to be a command and the command is executed and its output passed to the echo command.
LADM (12/02) 66 © Flying Pigs Training Exercise
1. Change into your home directory and from here list all the files that begin with the letters d, e, f or g in the /scr directory.
2. Count all files that begin with the letters d - g in the /scr directory.
Hint: Use the commands ls, wc and a pipeline.
3. Now copy all files that begin with the letters d - g from the /scr directory into your own home directory. Ignore any warning messages that are displayed.
4. Display a list of users logged on to the system.
5. Create a file called wholist containing a list of all users logged on to the system. Display the file wholist.
Hint: Use the command who and redirection with >.
6. Now count the number of users logged on to the system.
Hint: Use the commands who, wc and a pipeline.
7. Display a message of the form Number of users logged on = N where N is the actual number of users logged on.
Hint: Use the commands echo, who, wc, a pipeline and the format $(…).
8. Remove all the files, but not the hidden ones, and directories in your home directory, suppressing any warning messages.
Hint: Use a wildcard with the command rm and the options –r and –f .
9. Confirm that you now have no files or directories.
10. Now restore all the files and directories from the user traina.
Hint: Use the command cp with the –r option
LADM (12/02) 67 © Flying Pigs Training Exercise – File Commands
1. List all files that begin with the letters d, e, f or g in the /scr directory.
ls /scr/[d-g]*
2. Count all files that begin with the letters d - g in the /scr directory.
ls /scr/[d-g]* | wc -l
3. Now copy all files that begin with the letters d - g from the /scr directory into your own home directory.
cp /scr/[d-g]* .
4. Display a list of users logged on to the system.
who
5. Create a file called wholist containing a list of all users logged on to the system. Display the file wholist.
who > wholist ; more wholist
6. Now count the number of users logged on to the system.
who | wc -l
7. Display a message of the form Number of users logged on = N where N is the actual number of users logged on.
echo Number of users logged on = $(who | wc -l)
8. Remove all the files, but not the hidden ones, and directories in your home directory, suppressing any warning messages.
rm -rf *
9. Confirm that you now have no files or directories.
ls
10. Now restore all the files and directories from the user traina.
cp -r ../traina/* .
LADM (12/02) 68 © Flying Pigs Training Finding Files and Job Control
Objectives
At the end of this section the delegate will be able to:
find files on the system and execute commands on those files
redirect output and error messages to a file
run and stop background jobs
describe how a command may take input from a file
transfer files from one server to another
format files for printing and print files
view the print queue and cancel print jobs
LADM (12/02) 69 © Flying Pigs Training find - Finding Files
The command find may be used to find a particular file. The simple format of the command is:
find startpoint option filename command
For example, to find the file nice in your user:
cd
find . -name nice -print
To find all occurrences of the file below a directory:
find /home -name nice -print or find .. -name nice -print
Associated commands include locate.
Exercise
Find all occurrences of the file passwd in your PARENT directory.
LADM (12/02) 70 © Flying Pigs Training More on Redirection
One problem with the above commands is that both the output and any errors are displayed on the screen.
The terminal by default is the standard output, STDOUT. It displays output from a command.
Error messages from a command are also sent by default to standard error, STDERR. By default, this is also the terminal.
Both standard output and standard error are referred to by numbers as follows:
STDOUT 1
STDERR 2
The output from the find command can be redirected to a file, instead of the terminal as follows:
find .. -name nice -print 1>plist
However, the number 1 is assumed, so the command may be written as:
find .. -name nice -print >plist
more plist Exercise
Find all occurrences of the file passwd in your PARENT directory, this time redirecting the output to a file.
LADM (12/02) 71 © Flying Pigs Training Redirecting Errors
Errors can also be redirected to a file, instead of the terminal. However, here the number 2 must be used.
For example, the errors from the find command can be redirected as follows:
find .. -name nice -print >plist 2>perror
more perror Using /dev/null
If the error messages do not need to be saved they may be redirected to /dev/null and lost:
find .. -name nice -print 2>/dev/null
Redirecting to the Same File
Both output and error messages can be directed to the same file:
find .. -name nice -print >plist 2>&1
Exercise
Find all occurrences of the file passwd in your PARENT directory, this time redirecting the output and the errors to a file.
Find all occurrences of the file ping under the ROOT directory, redirecting the output and discarding any errors.
LADM (12/02) 72 © Flying Pigs Training Foreground Jobs
This find command will run in foreground until it ends or it is interrupted by the key sequence CONTROL & C.
find / -name ifconfig -print >plist 2>perror
No other commands may be entered while this command is running. Background Jobs - &
The following runs the find command, but here the command is run in background, disconnected from the terminal:
find / -name ifconfig -print >plist 2>perror &
This means that other commands can be run at the terminal whilst the background job is running.
For example:
ls -l
The command ps can be used to find the job number of the background job:
ps
When the background job has finished, a message is sent to the terminal.
Exercise
Find all occurrences of the file ifconfig under the ROOT directory. Redirect the output and any errors to a file and run the command as a background job.
LADM (12/02) 73 © Flying Pigs Training kill - Stopping Jobs
The command kill can be used to stop a background job that is looping or has run for too long. Here, the sleep command is used as an example:
sleep 600 & ps
The above job may now be stopped by using the PID number as a parameter to the kill command:
kill nnnn
Some jobs, such as a log in shell, may not be stopped in this way. This is because they trap the default signal sent to them by the kill command.
However, any job may be stopped by use of the kill command with the signal –9.
For example, to kill off a hung user session first find the PID:
ps -u traina and then kill the session using the signal –9:
kill -9 nnnn
Exercise
Try out some of the above commands. at - Scheduling a Job
The commands at and crontab (covered later) can be used to schedule a background job to run at a specified time on a specified day. The date and time may be entered in a variety of formats.
For example, to schedule a command to run at 16:00:
at 16:00
Then at the at> prompt:
at> banner lunch break > /dev/pts/1 at> CONTROL & D
LADM (12/02) 74 © Flying Pigs Training jobs - Job Control
Using job control, commands running in foreground can be suspended and can be moved from foreground to background and back again. Here a command is assigned both a PID and a job number and the job number can be used to work with the command.
For example, if the following job is run in foreground:
sleep 600 it can be suspended and put into background by typing:
CONTROL-Z
The following job when run in foreground:
sleep 900 can also be suspended in background by typing CONTROL-Z
Now the jobs command can be used to list jobs in background by job number.
jobs -l
A suspended job may be restarted by job number by typing:
bg %1 or may be brought back into the foreground by typing:
fg %1
LADM (12/02) 75 © Flying Pigs Training nohup Command
The nohup, no hang up. command can be used to keep background processes running even if the original login is terminated. Any output from the command is directed to a file rather than to the terminal.
nohup find / -name ping -print & nice Command
The nice command alters the priority of a command for scheduling. All processes have a default priority of twenty. However, normal users can only decrease priority.
This example adds ten to the default priority, which decreases the priority:
nice -10 find / -name ping -print >plist 2>/dev/null &
The super user can increase the priority on a process using the syntax shown below (with a double hyphen):
nice - -10 find / -name ping -print >plist 2>/dev/null &
LADM (12/02) 76 © Flying Pigs Training find - Finding Multiple Files
The command find may be used to find a set of files. For example, to find all files in your user beginning with a p try:
find . -name p* -print
If there are two or more files beginning with a p this will fail because the * is expanded by the shell before the command is run. To illustrate this try:
set -x and then rerun the find command. The command set -x turns on a debugging facility which shows how the command line is expanded. It may be turned off with: set +x
The find command will work however, if the * is protected from the shell as follows:
find . -name “p*” -print
Files can be found irrespective of case as follows:
find . -name “[Bb][Yy][Ee]*” -print
Exercise
Find all occurrences of files starting with the character p in your PARENT directory, discarding any errors.
LADM (12/02) 77 © Flying Pigs Training find - Find and Execute Command
In place of the –print option, the default, two other options may be used with find. The first –exec, executes a given command on the file found.
To find all occurrences of the file passwd below the directory home and run the wc command on each file found use:
find .. -name passwd -exec wc -l {} \;
The second, -ok, uses interactive mode. This example finds all files in your user beginning with a p and runs the rm command in interactive mode on each file found:
find . -name “p*” -ok rm {} \;
Exercise
Try out the above find commands. find - Examples
The find command may also be used to find files by user, permissions, type, inode or size. Use the command:
man find to find out more details.
The following are simple examples of the use of find:
By user: find /tmp -user train1 -exec ls -l {} \;
By permissions: find . -perm -u=x -print
By type: find . –type d
By inode: find .. -inum 355 -print 2>/dev/null
By size, bigger: find . –size +12m
By size, smaller: find . –size -2m
To files newer than another file: find . –newer oldfile
LADM (12/02) 78 © Flying Pigs Training find – Date/Time Examples
Assuming that today is 19th Jan, this example updates the given files, using the dates of 18th and 19th Jan, and then finds the files by time modified:
touch -t 01191150 less ol* pa* touch -t 01181150 text1 BYE
ls -l
find . -mtime +0 find . -mtime +1
The find options –a and –o may also be used to give an AND or OR condition.
This example removes all files in the users home directory named p* or t* that have not been accessed for a week. Note the use of the escape character \:
find $HOME \( -name ‘p*’ -o -name 't*' \) \ -atime +7 -exec rm {} \;
Note that the descriptions of -atime and -mtime use the terminology of a number of 24 hour periods. For example, a file accessed at 23:59 will be selected by:
find . -atime -1 -print at 00:01 the next day, less than 24 hours later, not more than one day ago. The midnight boundary between days has no effect on the 24-hour calculation.
LADM (12/02) 79 © Flying Pigs Training ln - Linking Files
The command ln may be used to create a link to a file. A link is like a shortcut and allows a file to be accessed via a different name.
For example:
ln ../traina/oldfile oldfile1
This will create a hard link to the file oldfile1 owned by user traina. The number of links that a file has may be viewed thus:
ls -l
Note that both links are in fact the same file:
cat ../usera/oldfile cat oldfile1 and that if usera changes their file, the other file reflects the change:
echo end of file >> oldfile cat oldfile1
If the files are viewed with the ls command and the option -i, the inode, a unique number which identifies a file is displayed for each file:
ls -i Exercise
Create your own link to a file owned by another user.
Use the find command with the inum option to find the names of the two linked files. ln - Linking login Files
The command ln may be used to link the .bashrc and .profile files to ensure that the same commands are executed on both login and when a shell is forked:
rm .profile
ln .bashrc .profile
LADM (12/02) 80 © Flying Pigs Training Linking Directories - ln -s
The default type of link created with the ln command is a hard link, which cannot be used to link directories.
A symbolic or soft link allows a directory to be accessed via a different name. Note that both links are in fact the same directory.
For example, to create and view a soft link:
ls ../traina/temp
ln -s ../traina/temp temp1
ls -l
ls temp1
Exercise
Create a link to the directory /usr/bin and use the new link to view the files in that directory
LADM (12/02) 81 © Flying Pigs Training write - Sending a Message
The command write may be used to send a message to another user provided that the user is logged on and permissions allow.
For example, to send a message to train1 type the following:
write train1 and enter lines of text. When input is complete press RETURN and enter CONTROL & D to signify end of data.
If the user is logged on more than once, specify the terminal to be used as follows: write train1 pts/3
The above may fail if you do not have permissions to write to the other users terminal.
Associated commands include wall, which may be used by the root user to send a message to all users logged on. mesg - Setting Permissions
The command mesg may be used to set and check permissions on the terminal device file. This may be used to inhibit the use of the write and talk commands.
To check the permissions enter:
mesg
If the command returns y then the terminal may be written to; if n is returned it may not.
To set permissions to n:
mesg n
LADM (12/02) 82 © Flying Pigs Training mailx - Mailing Users
The command mailx may be used to send a mail message to other users who may or may not be logged on.
The command may be used at the prompt in the same way as the write command. Lines of text are entered and when input is complete, CONTROL & D is used to signify end of data.
To view your own mail, enter the command with no user name:
mailx
To open a saved mail box file, enter the command as:
mailx -f mbox
Note that on most systems, the command mail is less sophisticated than mailx. Redirecting Input - using < symbol
Some commands such as write and mail, cannot take a file name as a parameter. To overcome this input can be redirected, so that input comes from a file instead of the keyboard.
For example, the following command will prompt the user for input:
write train1 pts/3
But input may be redirected by use of the < symbol.
Here a user is sent a file to be displayed on their screen:
write train1 pts/3 < /scr/message
This command will send the file linux as a mail message to several users:
mail train1 train2 train3 < linux
LADM (12/02) 83 © Flying Pigs Training Redirecting Input - using << symbol
The here document facility allows redirection of input to a command from the command line.
The following example uses the write command to send a message to a user:
write train1 pts/3 << XXX a multi line message sent as a here document XXX
The << symbol causes re-direction of input up to the marker, the text string XXX. Any string of characters may be used to mark the start and end of the here document.
Piping into Commands
The fact that write cannot take a file name as a parameter can also be overcome by use of a pipeline, piping being the process of sending the output of one command as input into another command.
For example:
banner "Hello there" | write train2
This will also work with the mail command:
banner "Hello there" | mail train1 train2
LADM (12/02) 84 © Flying Pigs Training File Transfer Between Servers
File transfer involves gaining access to the files on a target server and then transferring a file or files from or to the host server. The files will be transferred to and from the current working directory on the host. Three steps are involved here:
Find the IP address of the target server
Contact the target server to confirm it is available
Transfer files between the two servers ifconfig The command ifconfig can be used to view the current IP address of a server. However, the command may only be available to the root user. For example, to list all interfaces on Linux use:
ifconfig
On Solaris use:
ifconfig -a
The name and IP Address of the current server may also be held in the file /etc/hosts. ping The command ping may be issued by a user to confirm that another server is on the network. For example, to send a message to a server with a given IP Address:
ping 192.168.1.150
LADM (12/02) 85 © Flying Pigs Training sftp With sftp, the user name and target server are supplied as follows:
sftp [email protected]
A list of commands available on the target server may be displayed with the command:
help
The commands available include cd and ls.
To run a command on the host server, use the following syntax:
!ls
Exercise
With the help of your tutor, use sftp to login to the given target server as trainN. put The command put may be used to move files to the target machine from the current working directory on the host server.
To move a single file or multiple files use:
put text1 or put p* get The command get may be used to fetch files from the target machine into the current working directory on the host server.
For example, to fetch a single file or multiple files use:
get text1 or get p*
Exercise
Transfer several files from the host to the target server.
LADM (12/02) 86 © Flying Pigs Training ssh The command ssh allows a user to logon to another server, much like Putty on Windows:
Associated commnds include ftp and scp. Most Windows operating systems also offer a version of ftp which can be run from within a command window.
An alternative to the command line is one of the many Windows based alternatives such as Filezilla, SSH Secure Shell and Tunnelier. These allow files to be transferred between a server and a PC using simple drag and drop commands.
LADM (12/02) 87 © Flying Pigs Training Print System
There are two different methods or commands used to print files.
SVR4 - lpstat, lp and cancel
BSD - lpq, lpr and lprm
UNIX systems such as Solaris and HP-UX tend to use SVR4 and Linux uses the BSD commands. Normally, if one set of commands do not exist then the other set will be available.
Many systems also use multiple printers. However, if a default printer is not available, a printer name must be specified. pr - Format Files
The print commands simply send a stream of characters to the printer much as the cat command sends a file to the screen. A better approach is to make use of the command pr to format a file for printing.
For example, compare the following:
cat linux
pr linux
By default the output from pr is 66 lines long and has both a page header and a page number.
To format and print a file, use the pr command and pipe the output into the appropriate print command.
LADM (12/02) 88 © Flying Pigs Training lpstat, lp, cancel - Printing Files
Print Queue - lpstat
In SVR4 the command lpstat may be used to view the print queue.
For example:
lpstat or lpstat -t
Print File – lp
In SVR4 the command lp may be used to print a text file, to the default or to a named printer.
For example, to print the text file linux to the default printer:
pr linux | lp ; lpstat
To print the text file linux to the printer lp:
pr linux | lp -dlp
The command will respond with a request id.
Print Job - cancel
In SVR4 the command cancel may be used to cancel print requests provided that the request is still in the print queue. It takes as a parameter a set of print request ids.
For example, to print the text file linux:
pr linux | lp ; lpstat
Note the request id displayed. Now cancel the request:
cancel request id
To cancel all jobs for a user:
cancel -u train9
LADM (12/02) 89 © Flying Pigs Training lpq, lpr, lprm - Printing Files
Print Queue - lpq
In BSD on Linux servers the command lpq may be used to view the print queue.
For example:
lpq or lpq -Plp0
Print File – lpr
The command on Linux and BSD systems for printing files is lpr. This command may be used to print a text file, to the default or to a named printer.
For example, to print the text file linux to the default printer:
pr linux | lpr ; lpq
To print the text file linux to the printer lp0:
pr linux | lpr -Plp0
Print Job - lprm
In BSD the command lprm may be used to cancel print requests provided that the request is still in the print queue.. It takes as a parameter a set of job numbers.
For example, to print the text file linux:
pr linux | lpr -Plp0
View the job number with the command:
lpq -Plp0
Then cancel the job by specifying the job number:
lprm -Plp0 job number
To cancel all jobs for your user:
lprm -Plp0 -
LADM (12/02) 90 © Flying Pigs Training The vi Editor
Objectives
At the end of this section the delegate will be able to:
describe the text editor vi
use the vi editor to create and alter files
LADM (12/02) 91 © Flying Pigs Training What is a Text Editor?
Text editors are programs used to edit simple text files, files that are limited to letters, numbers and characters such as * and $; they do not include graphics, fonts, or special formatting found in documents created by word processors such as Word.
They can include mail messages, shell scripts and program source code and are the most common type of file in use on the server. For this reason, an editor is needed that will work well with text files.
Outside of its wide availability, the other important feature that vi has is that unlike some other editors it only puts what is typed into the file.
However, it is important to remember what vi is not. It is not a simple word processor or a graphics WYSIWYG editor such as Word or Notepad. Using vi vi commands consist of single keystrokes, such as a and i, which perform simple text editing functions.
However, commands are case sensitive so there are at least 52 commands to begin with.
To edit a file, type vi filename, which will load the named file if the file exists or will open a new file if it does not.
For example, to open the sample file vifile, make sure you are in your home directory and type:
cp /scr/vifile .
vi vifile
The main reason why new users experience problems with vi stems from the fact that vi operates in three different modes:
COMMAND Move about and issue INPUT commands
INPUT Make change, press ESC to leave this mode
ED issue ED commands, such as when saving the file
The first mode is Command mode.
LADM (12/02) 92 © Flying Pigs Training vi Command Mode - Moving the Cursor
A terminal's arrow keys may not always work with vi. If this is the case the following keys are always available:
h=[left] j=[down] k=[up] l=[right]
The arrow keys are perhaps the least efficient way of moving around the file. There are many other ways:
CTRL & b Scroll up a page CTRL & f Scroll down a page
CTRL & u Scroll up half a page CTRL & d Scroll down half a page
1G Go to line 1 G Go to last line
H Move to top line of screen M Move to middle line of screen L Move to bottom line of screen
w Move to next word b Move to previous word
0 Go to start of line $ Go to end of line Exercise
Try out some of the above navigation commands.
LADM (12/02) 93 © Flying Pigs Training vi Input Mode - Inserting and Changing Text
The second mode is Input mode. There are many commands available for inputting new text or changing existing text in a file. But mastery of a few will normally suffice!
However, the ESC key must be pressed after EACH change in order to exit from Input mode and return to Command mode before another command may be issued.
The most common error in vi is to forget to leave input mode and then try to move the cursor around.
Try to get into the habit of always pressing the ESC key before entering another command. Even if the ESC key is pressed in error, the system will only issue a beep.
The following are some of most often used LOWER case commands:
i Insert text before the current character a Append text after the current character o Open a new line below the current line r Replace the character under the cursor cw Change a word
Similar commands in UPPER case include.
I Insert text at the beginning of the current line A Append text at the end of the current line O Open a new line above the current line C Change text to end of the line R Replace continuously starting at the current character
Exercise
Insert some text into the file using some of the above commands.
LADM (12/02) 94 © Flying Pigs Training vi Deleting Text
There are also many commands available for deleting part of a line or complete lines of text.
The following are some of the different methods available.
x Delete current character dd Delete current line
D Delete the rest of the line dw Delete a word
Note that numbers may be entered in front of many vi commands, for example:
6x Delete the next 6 characters 2dd Delete the next 2 lines
The following are some useful commands:
u Undo the last command SHIFT & U Undo all changes to current line . Repeat the last command ~ Change case of current character J Join current and next line together >> Indent by a single tab << Remove indent CTRL & G Show current file name CTRL & L Refresh screen ( some systems use CTRL & R )
Exercise
Alter the file vifile using some of the above commands.
LADM (12/02) 95 © Flying Pigs Training Exiting from vi
To quit from vi if you have made no changes, go into Ed mode by typing:
:q
However, this will give an error if you have made changes to the file.
To override this error message and exit without saving changes, type:
:q!
To save the changed file and remain in vi type:
:w
To save the changed file and exit from vi, type:
:wq
The commands :x and ZZ may also be used in place of :wq.
Exercise
Exit from the file vifile without saving any changes.
Now edit the file vifile again, make some changes to the file and exit, this time saving the changes.
There is a vi Prompt Sheet which you may find useful in the Handouts section at the end of the course notes.
LADM (12/02) 96 © Flying Pigs Training vi Setting Options
If using Linux then vim, vi improved, is available and here a reminder is issued whenever Input mode is entered.
However, one of the irritating features of vi on UNIX servers is that it is not obvious when the editor is in Input mode. This problem can be overcome by using the set command to turn on this reminder feature as follows:
:set showmode
Other useful settings include:
:set number Set line numbers on :set nonumber Set line numbers off
:set list Show visible tabs and end of line :set nolist Set list off
:set all Show all settings
Mapping Keys In vi both character keys and control keys can be mapped. To map the characters q and z to commonly used commands:
map q :q! map z :wq
Control keys such as CTRL & N may be mapped as follows.
map ^N :set number^M map ^O :set nonumber^M
Note that the following key combination is used to insert a value such as ^N, or Control & N:
i to enter insert mode, Control & V, Control & N, Esc
Unfortunately, when the file is saved and the user leaves vi, any settings are lost.
However, if the settings are saved in a file called .exrc in the home directory, they will be available whenever vi is loaded.
Exercise
Wth the help of your tutor, create a file called .exrc and enter some of the above commands.
Now open the file vifile and check that any options in the .exrc file are set.
LADM (12/02) 97 © Flying Pigs Training vi Regular Expressions
Vi has the ability to do complex search and replace actions. It does this using the same regular expressions as grep and sed; strings made up of special characters and symbols that allow pattern matching within utilities.
Two of the special characters available are:
^ - the Caret Symbol the caret symbol matches the START of a line
$ - the Dollar Sign the dollar sign matches the END of the line vi Global Search Commands
The following are some examples of how to search for text in vi.
Note that these also work with the commands more, less and man.
/log search forwards for text
n find next occurrence
N find previous occurrence
/^log find lines beginning with text
/log$ find lines ending with text Exercise
Edit the file logfile and find the lines containing the string error.
LADM (12/02) 98 © Flying Pigs Training vi Cut and Paste Text - delete and put
To cut and paste text in vi, use a combination of the delete and put commands.
The delete command deletes and copies text into a buffer. The put command then copies the text from the buffer to the current position in the file.
6dd Delete 6 lines and copy to a buffer
p Put lines from buffer below the current line P Put lines from buffer above the current line
xp Transpose current and next characters vi Copying Text - yank and put
To copy text in vi, use a combination of the yank and put commands.
The yank command marks the specified text and copies it into a buffer. The put command then copies the text from the buffer to the current position in the file.
6yy Copy 6 lines to a buffer
p Put lines from buffer below the current line P Put lines from buffer above the current line Exercise
Copy the /etc/passwd file into your home directory.
Edit this new passwd file and carry out the following operations.
Copy the first line for root so that it appears twice.
Delete several characters from lines 4 and 8.
Find the entry for your own user and move this line to end of the file.
Try out some other copy, cut and paste options.
Save the file and exit.
LADM (12/02) 99 © Flying Pigs Training Comparing Files – cmp, diff, sdiff
The commands cmp, diff and sdiff may be used to compare two files as follows:
cmp passwd /etc/passwd
diff passwd /etc/passwd
sdiff passwd /etc/passwd | more
Exercise
Use the above commands to compare the two passwd files.
LADM (12/02) 100 © Flying Pigs Training vi Line Addresses
Line addresses are made up of numbers and special characters and allow changes to be made to a set of lines.
The special characters that may be used to address lines include:
. - the Full Stop the full stop matches the CURRENT line
$ - the Dollar Sign the dollar sign matches the LAST line vi Global Substitution
The following substitution commands are all run in Ed mode and start with a :, a colon.
:s/error/FAULT/ change the first occurrence on the current line
:s/error/FAULT/g change all occurrences on the current line
:1,$s/error/FAULT/g change all occurrences on all lines
:1,$s/error/FAULT/gc change all occurrences on all lines with a prompt Exercise
1. Edit the file logfile and try out the above Global Search commands.
Global Printing and Deleting
The following commands may be used to delete many lines.
:g/state/p Get all lines containing text and print
:g/state/d delete all lines containing text
:1,$d delete ALL lines from the file
:.,$d delete from the current to last line
:1,.d delete from first to the current line
LADM (12/02) 101 © Flying Pigs Training Why use vi?
So why use vi. Some of the reasons are because it:
is available on all systems on a wide variety of platforms
is considered the standard, and is sometimes the only editor on UNIX
only puts the text typed into the file without any formatting characters
has the ability to run shell commands
has the ability to do complex search and replace
An excellent free alternative to vi is nano, itself a replacement for the Pico text editor that is part of the Pine email suite. It can be downloaded at www.nano- editor.org. Linux systems also have the GUI editor gedit.
LADM (12/02) 102 © Flying Pigs Training vi Advanced Commands
Running Shell Commands Shell commands can be executed from within vi by going into Ed mode and typing an !, an exclamation mark, followed by the command. For example:
:!date
:!ls
Reading in Files The contents of another file may be copied into the current file by using the r command.
For example:
:r people
The results of a shell command can be copied into the file you are editing as follows:
:r !date
:r !ls -l
Opening Files The default in vi is to open the file at the first line but files can be opened at the last line, at a given line number or at the line that matches a pattern.
For example, to open a file at the last line:
vi + logfile
To open a file at a given line number:
vi +800 logfile
To open a file at a pattern:
vi +/Reboot logfile
LADM (12/02) 103 © Flying Pigs Training EXINIT System Variable This system variable may be used as an alternative to the .exrc file. It is useful if all users require the same settings as it can be set up as the default for the environment.
export EXINIT=':set showmode number'
Command Line Edit The vi editor may be used to edit the command line in the Korn shell as follows. First, set the editor vi to be the default editor for the command line:
set -o vi
Then press ESC and the use the following letters to navigate through the history file:
h=[left] j=[down] k=[up] l=[right]
Previous commands may be edited using vi commands BUT note that the terminal arrow keys may NOT be used with command line editing.
LADM (12/02) 104 © Flying Pigs Training Finding Text and Sorting
Objectives
At the end of this section the delegate will be able to:
find text within files using grep
describe and use regular expressions
use the sort command
use the commands unique, cut and translate
make simple use of the awk command
LADM (12/02) 105 © Flying Pigs Training grep - Finding Text in Files
The command grep, Get Regular Expression and Print, in its simple form may be used to find text within a particular file or files. Before trying the examples, first copy and view the sample file:
cp /scr/expfile .
cat expfile
Now to find text within the file:
grep abc expfile
Common options used with this command are:
-i find all lines irrespective of case
-c count how many lines contain the text
-n display line numbers of matching lines
-l display only file names that match
-v find all lines NOT containing the text
For example:
grep -c abc expfile
grep -in abc expfile
grep -v abc expfile
grep -ic this *
grep -il this *
Exercise
Find all occurrences of the text failed in the file logfile.
Now find how many lines in the file contain the text error.
Find how many lines contain the text error irrespective of case.
Find the entry for your login in the /etc/passwd file.
Now find all entries for the delegate logins on the course.
LADM (12/02) 106 © Flying Pigs Training grep - Using Regular Expressions
The command grep may also be used with certain special characters or symbols to refine the search. A regular expression is a character string that includes these special characters to allow pattern matching within utilities such as grep, vi and sed. Note that they should be enclosed in single quotes.
The special characters available include:
^ Start of a string $ End of a string . Any character (except \n newline) * 0 or more of previous expression
Note that ? and *, which may be used at the command line to match a single character and any characters, are not used in the same way here.
To find all lines starting with text using the ^ characters:
grep ‘^abc’ expfile
To find all lines ending with text using the $ characters:
grep ‘abc$’ expfile
To find lines containing a string using both ^ and $ characters:
grep ‘^abc$’ expfile
To find the number of each blank line use:
grep -n ‘^$’ expfile
grep -n ‘^$’ linux
To find lines using the . to match any character:
grep ‘^a.c$’ expfile
To find lines using the * to match 0 or more of the previous expression:
grep ‘^ab*c’ expfile
LADM (12/02) 107 © Flying Pigs Training Exercise
Display the file people and examine its contents.
Find all lines containing the string Smith in the file people.
Hint: use the command grep but remember that by default, it is case sensitive.
Create a new file, npeople, containing all lines beginning with the string Personal in the people file.
Hint: use the command grep and >.
Confirm the contents of the file npeople by listing the file.
Now append all lines ending with the string 500 in the file people to the file npeople.
Hint: use the command grep and >>.
Again, confirm the contents of the file npeople by listing the file.
Find the IP Address of the server which is stored in the file /etc/hosts.
Hint: use the command grep with $(hostname)
LADM (12/02) 108 © Flying Pigs Training egrep - Expression grep
The command egrep allows searching for more than one regular expression simultaneously. The expressions must be enclosed within a pair of single quotes.
For example:
egrep '^root ^sys ^traina' /etc/passwd
This extracts the lines which begin with root, sys or traina from the file.
Note that the complete egrep command spans three separate lines of input but that this format may not work in the bash shell. fgrep - Fast grep
The command fgrep is similar to egrep except that it searches for a character string only, NOT a regular expression.
For example:
fgrep 'root sys traina' /etc/passwd
This extracts the lines which contain root, sys or traina from the file.
Again, the complete fgrep command spans three separate lines of input. Some versions of grep incorporate the functions of both egrep and fgrep. Exercise
1. Use egrep or fgrep to extract from the /etc/passwd file account lines containing lp or your own user id
LADM (12/02) 109 © Flying Pigs Training Using Filters
Piping is the process of sending the output of one command as input into another command and is one of the most powerful tools available.
Commands that appear in a pipeline are often referred to as filters, since in many cases they sift through or modify the input passed to them, before sending the modified stream to STDOUT.
In the following example, standard output from ls -l is passed as standard input to the grep command. Output from the grep command is then passed as input to the more command.
This will display only directories in /etc:
ls -l /etc | grep ‘^d’ | more
The following commands are examples of using filters.
who | grep traina
ps -ef | grep cron Exercise
Use the who and grep commands and a pipeline to display a count of the number of delegate users logged on.
LADM (12/02) 110 © Flying Pigs Training sort - Sorting Files
The command sort may be used to sort a file, by default on the first word as delimited by the tab character.
Copy the file /scr/soup into your home directory and try:
sort soup
sort -r soup
To remove duplicate lines use:
sort -u soup
The command sort may be combined with a pipeline to sort the output of a command, for example:
who | grep '^train' | sort sort - Redirection
The output of the sort command may be redirected to a file as follows:
sort soup > soup
cat soup
What will be the outcome of this command?
When the Shell sees a > sign on the command line it either creates a new file if one does not exist or overwrites the contents of the file if it does!
The -o option may be used to overcome this problem.
Copy the file /scr/soup again and try the command as:
sort soup -o soup
cat soup
Exercise
Try out the above commands.
LADM (12/02) 111 © Flying Pigs Training Sort - Using Fields
The command sort may also be used to sort using fields, where a field is delimited by a space character.
Here are some examples:
ls -l | sort
To sort the output on the 5th key field:
ls -l | sort -k 5
To sort in reverse and on numeric values use:
ls -l | sort -nrk 5
Here is an example of combining sort with find to display the 5 largest files, excluding directories:
find /scr -type f -exec ls -s {} \; 2>/dev/null | sort –nr | head -5
As an alternative, the following option may be used to sort on a particular field:
ls -l | sort -nr +4
Exercise - optional
1. Find the 5 largest files in your home directory.
LADM (12/02) 112 © Flying Pigs Training Sort – Field Separator
A file may also be sorted using a field separator other than a space. The following example attempts to sort the password file on the 4th field but does not work properly:
sort -nk 4 /etc/passwd
To correctly sort the file, specify the field separator as follows:
sort -nk 4 -t: /etc/passwd
Exercise
1. Sort the password file into descending order on the 3rd field, the user id.
2. Now display only the line with the highest delegate user id.
Hint: Use the sort, grep and head commands with a pipeline. uniq - Removing Duplicates
The commands sort -u or uniq may be used to remove duplicate lines from the input, for example:
sort -u soup
sort soup | uniq
sort soup | uniq -c
LADM (12/02) 113 © Flying Pigs Training cut - Cropping Text
The command cut may be used to copy fields or columns from a file. The original file remains unchanged.
For example, to copy columns 1 to 8 inclusive from the file people:
cut -c1-8 people
To copy field 1 and 2 from the file people, where the tab character is used as a delimiter:
cut -f1,2 people cut - Using Delimiters
The command cut uses the tab character as a delimiter.
This means that the following does NOT work as it assumes that the field separator is a tab:
grep '^train' /etc/passwd | cut -f1
In the case of the passwd file this is not true.
However, the following syntax may be used to set the field separator to a colon:
grep '^train' /etc/passwd | cut -f1 -d:
Exercise
1. Sort the password file into descending order on the 3rd field, the user id and this time display only the highest delegate user id.
Hint: Use the sort, grep, head and cut commands with a pipeline.
2. List the login names ONLY of the delegates currently logged on.
Hint: Use the who, grep and cut commands with a pipeline and note that the delimiter here is the space character.
3. Rerun the above command, but this time sort the output and remove any duplicate user logins.
LADM (12/02) 114 © Flying Pigs Training tr - Translate File
The command tr is a good example of the use of piping as it will not accept its input from a file. It may be used to translate text in a file and is often used to convert from lower to upper case and vice versa.
For example, to convert all text in the above sorted file from lower to upper case use: sort soup | uniq | tr [:lower:] [:upper:]
sort soup | uniq | tr [a-z] [A-Z]
Sort - Using Characters
When sorting by field does not give the correct result, the command may be used to sort on certain characters within a field.
For example, copy and sort the following file:
cp /scr/users .
sort -n users
To sort the output correctly via the 5th and 6th characters of the first key field use the -k option as follows:
sort -nk 1.5,1.6 users awk – Displaying Columns
The awk command, and its’ associated commands gawk and nawk, are text processors with a complete and powerful programming language. They can perform complex text processing and are much more powerful than grep.
The awk command sees the first field in it’s input as $1, the second as $2, the third as $3 etc.
Try the following, which will print only the file name and size of all files:
ls -l | awk '{ print $9, $5 }'
Here awk prints a unique, sorted list of delegates logged in:
who | awk '/^train/{ print $1 }' | sort –u
LADM (12/02) 115 © Flying Pigs Training grep - Searching Sub-directories
Some versions allow grep to search sub-directories. But this may not be the case.
For example, try:
grep This *
grep -r This * xargs – Passing Filenames
If the -r option is not available, the xargs command may be used with a pipeline and the find command in order to search files in sub-directories.
To see how this works consider the following. The find command was used to find all occurrences of the file passwd below the directory home and run the wc command on each file found:
find .. -name passwd -exec wc -l {} \;
This command might have been written as follows:
find .. -name passwd -print | wc -l
But here the input to the wc command is treated as the contents of one file:
find .. -name passwd -print
The command xargs is needed in order to pass each INDIVIDUAL file name found to the wc command as a parameter, rather than pass the input as if it were the contents of a file.
This command may now be rewritten using xargs as follows:
find .. -name passwd | xargs wc -l
With grep, the command xargs is needed in order to pass each file name found as a parameter, so that each sub-directory is searched:
find * -print | xargs grep This
LADM (12/02) 116 © Flying Pigs Training More Regular Expressions
A regular expression, regex or regexp for short, is a special text string for describing a search pattern. They can be thought of as wildcards on steroids.
There are eleven characters with special meanings: the opening and closing square brackets [ ], the backslash \, the caret ^, the dollar sign $, the period or dot ., the vertical bar or pipe symbol |, the question mark ?, the asterisk or star *, the plus sign +, the opening and closing round bracket { }. These special characters are often called metacharacters.
Here is the full set of special characters:
^ Start of a string $ End of a string . Any character (except \n newline) * 0 or more of previous expression | Alternation, either one or the other [...] Explicit set of characters to match + 1 or more of previous expression ? 0 or 1 of previous expression \ Preceding a symbol makes it a literal character {...} Explicit quantifier notation (...) Logical grouping of part of an expression
The default version of grep in /usr/bin has only limited regular expression support. In order for all of the following examples to work, first create the following alias:
alias grep=’/usr/xpg4/bin/grep -E’
To find lines using the | to match either expression:
grep ‘aac|acc’ expfile
To find lines using | to match either expression within a string also use ( ):
grep ‘^a(Bc|bc)’ expfile
LADM (12/02) 117 © Flying Pigs Training To find lines using [ ] to match any character:
grep ‘^a[Bb]c’ expfile
To find lines using [ ] to NOT match any character:
grep ‘^a[^Bb]c’ expfile
To find lines using the * to match 0 or more of the previous expression:
grep ‘^ab*c’ expfile
To find lines using the + to match 1 or more of the previous expression:
grep ‘^ab+c’ expfile
To find lines using the ? to match 0 or 1 of the previous expression:
grep ‘^ab?c’ expfile
To find lines containing a special character using the \:
grep ‘a\*c’ expfile
Exercise
Display the file people and examine its contents.
Find all lines containing the names Evans or Maler in the file people.
Find all lines containing the names Smith, Smyth or Smythe in the file people.
Find all lines containing the names Brown, Browen or Bron in the file people.
If you have time:
Find the line containing the string (c0d0s1), including the brackets, in the file logfile.
Find the line containing the character * in the file logfile.
Combine 5 and 6 above to find both expressions.
LADM (12/02) 118 © Flying Pigs Training Regular Expression Examples
To find lines using . and * to match any set of characters:
grep ‘^ab.*c’ expfile
To find lines using { } to match N number of characters:
grep ‘^ab{3}c’ expfile
grep ‘^ab{4}c’ expfile
To find lines using { } to match N or more times:
grep ‘^ab{3,}c’ expfile
To find lines using { } to match N times but not more than M times:
grep ‘^ab{2,3}c’ expfile
Exercise Solution
2. grep 'Evans|Maler' people
3. grep 'Sm(i|y)the?' people
4. grep 'Brow?e?n' people
5. grep '\(c0d0s1\)' logfile
6. grep '\*' logfile
7. grep '\(c0d0s1\)|\*' logfile
LADM (12/02) 119 © Flying Pigs Training Backup Commands
Objectives
At the end of this section the delegate will be able to:
describe reasons for taking backups
use the backup utilities such as tar and cpio
schedule jobs to run at a given time
create and run a simple backu script
use the compress utilities such as gzip
LADM (12/02) 120 © Flying Pigs Training tar - Backing Up Files
The backing up of files means making copies of them, usually on removable media, as a safeguard in case the originals get lost or damaged. Backup tapes are convenient for restoring accidentally deleted files, but they are essent*ial in case of serious hardware failures or other disasters.
Backups are the only practical way of restoring corrupted or deleted files on the system. Unlike Windows, there is no recycle bin!
The command tar, tape archive, may be used to backup all files and sub- directories. Although it is often used to backup to tape, backups can be made to a disc file. The simple format of the command is:
tar options output filename input filenames
The command may take the following arguments:
c create a tar archive
v run in verbose mode
f use the file name specified
For example, to create a tar archive one could use:
tar cvf /tmp/train1.tar .
Note that the command should always be used with a relative path name, since if an absolute path name is used, files may only be restored back to where they were archived from and not to another directory.
The file and file type may then be viewed as follows:
ls -l /tmp/train1.tar
file /tmp/train1.tar
Exercise
Backup all of your files to a tar archive file and examine the new file using the ls and file commands.
If the backup is to be run on a regular basis and create several new backup files, the date may be appended to the filename as follows: tar cvf /tmp/train1$(date +%m%d%H%M).tar .
LADM (12/02) 121 © Flying Pigs Training tar - Viewing the Archive
The contents of the tar archive can be viewed using the arguments:
t list table of contents
v run in verbose mode
f use the file name specified
For example:
tar tvf /tmp/train1.tar | more
To find a particular file use:
tar tvf /tmp/train1.tar | grep oldfile tar - Restoring a File
A file or files may be restored from a tar archive using the arguments:
x extract files from a tar archive
v run in verbose mode
f use the file name specified
The following will restore the selected file from the archive.
tar xvf /tmp/train1.tar ./oldfile
Note that the name of the file to be restored should be as it appears in the output from the table of contents.
Exercise
1. View the contents of the archive file.
2. Delete one of your files, for example the file oldfile.
3. Restore the deleted file from the tar archive.
4. Display the file to confirm that it has been restored.
LADM (12/02) 122 © Flying Pigs Training tar - Restoring a File using Wildcards
If the version of tar in use supports wildcards (on Linux), the following will restore a set of selected files from the archive:
tar xvf /tmp/train1.tar ./t*
If the version DOES NOT support wildcards (on Solaris) however, the following work around may be used.
First display the list of files that need to be restored:
tar tf /tmp/train1.tar | grep '\./t'
Note that the \ character is used as an escape character to state that the full stop following it is not to be treated as a special character.
Now, use this list of files as input to the tar command:
tar xvf /tmp/train1.tar $(tar tf /tmp/train1.tar | grep '\./t')
Exercise
Delete some of your files, for example files beginning with the characters t.
Restore the deleted files from the tar archive.
Display the files to confirm that they have been restored.
LADM (12/02) 123 © Flying Pigs Training crontab - Schedule a Job
The cron daemon may be used by users to schedule a task to run at a specified time. The cron comes alive every minute to read the system time and execute any commands that have been scheduled for that time. Scheduled commands for each user are stored in the directory /var/spool/cron.
The following crontab entries schedule the execution of several jobs to run at different times and on different days:
#m h d/m m w command 30 * * * * banner tea break > /dev/pts/1 0,30 * * * 1-5 tar cvf /tmp/train1.tar . 0 4 * * 1,5 /home/train1/dailybackup 30 20 10 6 * /home/train1/monthupdate
Each cron entry consists of 6 fields, minute, hour, day of the month, month, weekday and the command to be run. The time field uses 24 hour format so for 8am use 8 and for 8pm use 20.
The first five fields are numeric and can hold an integer value, a range of values, a comma separated list or an asterisk. The final field contains a command or script file that is to be executed at the specified time.
The first job above displays a message every 30 minutes of every hour of every day.
The second takes a backup on the hour and half hour, Monday through Friday. Sunday is day 0.
The third runs a backup script at 4.00am on Monday and Friday only.
The last will execute the script once, on 10th June at 8:30pm:
The cron daemon is often used to schedule regular system backups, which may be run at night when the system is lightly loaded.
LADM (12/02) 124 © Flying Pigs Training crontab Options
The crontab entries may be created by submitting a file:
crontab /scr/cronfile
If needed, the default editor for the crontab may be changed by setting the EDITOR variable:
export EDITOR=nano
Then if access to the cron is allowed for the user a job may be scheduled interactively using the editor by entering:
crontab -e
Exercise
1. Check the current time with the date command and use the tty command to identity your terminal.
2. Create dummy crontab entries from the sample file:
crontab /scr/cronfile
3. Edit the crontab file as follows:
crontab -e
4. Schedule a job to display a message on your terminal every hour.
5. Schedule a second job to backup your files every 30 minutes.
6. After the job has run, check for new mail entries using the command mailx.
After saving the file you may view the crontab entry using:
crontab -l
LADM (12/02) 125 © Flying Pigs Training If the crontab entries have been saved in a file, they may be re-scheduled as follows. First save the contents of the crontab:
crontab -l > cronfile
Next remove all scheduled jobs:
crontab -r
At a later date, re-schedule the jobs:
crontab cronfile
LADM (12/02) 126 © Flying Pigs Training gzip - Compress an Archive
In order to save space or if a file is to be mailed or transferred to another user, the command gzip can be used to compress the archive file. Associated commands include bzip2 and compress.
The following example shows the simple format of the command:
ls -l /tmp/train1*
gzip /tmp/train1.tar
ls -l /tmp/train1*
Exercise
List the size of the archive file and then compress the file and compare the new file size. gunzip - Uncompress an Archive
The command gunzip can be used to restore the compressed archive file. Associated commands include bunzip2 and uncompress.
For example:
gunzip /tmp/train1.tar
Exercise
Now uncompress the archive file.
LADM (12/02) 127 © Flying Pigs Training Backup Script
A simple backup script can be created and run as follows. Create a file containing the backup command (don’t forget the quotes): echo ‘tar cvf /tmp/train1$(date +%m%d%H%M).tar .’ > backup
Make the script file executable:
chmod +x backup
Test the script:
./backup
Create a crontab entry of the following form using the full path name of the script:
#m h d/m m w command 0,30 * * * * /home/train1/backup
On Solaris use the form /export/home/train1/backup.
Check for new mail entries after the job has run.
LADM (12/02) 128 © Flying Pigs Training cpio - Backing Up Files
The command cpio, copy input to output, can also be used to backup files and directories. The simple format of the command is:
list of input filenames | cpio options > output filename
Unlike tar it acts as a filter, taking a list of files as its input and copying them to the standard output.
The command may take the following arguments:
o create a cpio archive
v run in verbose mode
B block the output records
c create the archive in ASCII character format
For example, to create a cpio archive:
find . –name ‘*’ | cpio -ovBc > /tmp/train1.cpio
file /tmp/train1.cpio
Because of the way cpio writes headers to the output file, a cpio archive takes up less space on disc than a tar archive. However, tar is much easier to use:
ls -l /tmp/train1*
Exercise
Backup all of your files to a cpio archive file and examine the new file using the ls and file commands.
LADM (12/02) 129 © Flying Pigs Training cpio - Viewing the Archive
The contents of the tar archive can be viewed using the arguments:
i read the cpio archive
t list a table of contents
v run in verbose mode
c read the archive in ASCII character format
For example:
cpio -itvc < /tmp/train1.cpio | more cpio - Restoring a File
A file or files may be restored from a tar archive using the arguments:
i read the cpio archive
v run in verbose mode
d create any sub-directories required
B block the output records
c create the archive in ASCII character format
The following example restores files from the archive. Note that again the name of the file to be restored should be as it appears in the output from the above view command.
cpio -ivBcd t* < /tmp/train1.cpio
Exercise
1. Delete all of your files that begin with the letter t.
2. Restore the deleted files from the cpio archive.
3. Display the files to confirm that they have been restored.
LADM (12/02) 130 © Flying Pigs Training File Permissions
Objectives
At the end of this section the delegate will be able to:
describe the different file permissions
change file permissions on a file or directory
describe how special file permissions operate
set default file permissions for a user
LADM (12/02) 131 © Flying Pigs Training Permissions - File and Directory
The command ls with the parameter -l shows a long listing of files and directories and includes file permissions. For example, to view the permissions for all files:
ls -l
The output takes the form:
- rw- r-- r-- 1 train1 other 32 Mar 13 20:21 bye d rwx r-x r-x 2 train1 other 132 Mar 13 20:22 temp
In this example, the first character, a hyphen, indicates a file. The character d would indicate a directory.
The next nine characters, usually a combination of r, w, x and - indicate file permissions. They are divided into three groups of three characters indicating permissions for the user who owns the file, the group to which they belong and all other users.
The rest of the line shows the number of links, the owner and group id, the file size, date and time updated and file name.
LADM (12/02) 132 © Flying Pigs Training It is these file permissions that determine how a user may access a file. If a user has permissions on a file then the access is as follows:
r read open and read
w write open and write
x execute run as a program
- none no access
There are differences in the way that permissions work with directories. If a user has permissions on a directory then the access is as follows:
r read list using ls
w write update; user can add, delete and rename files
x execute change into using cd
- none no access
LADM (12/02) 133 © Flying Pigs Training Permissions - Evaluating
When evaluating the permissions that a user has on a file, the Shell first asks is this user the owner of the file?
If the user is the owner, permissions are defined by the first three characters in the file's mode - the user domain.
If the user is not the owner, it asks is this user in the same group as the file?
If this is the case, permissions are defined by the second set of three characters in the file's mode - the group domain.
If a user is neither the owner, nor in the same group as a file, their permissions are defined by the last three characters in the file's mode - the other domain.
Thus the other domain applies to everyone else, users who are neither the owner, nor in the same group as the file.
LADM (12/02) 134 © Flying Pigs Training chmod - Changing Permissions
Read, write and execute permissions may be assigned to a file or directory by the file owner or by root, for the domains u, g and o or for all:
u user who owns the file
g group that the file belongs to
o other users
a all users
The command chmod may be used to add or remove permissions on a file or files for one or more of the above.
The syntax is:
chmod [ugoa] [+-=] [rwx] filenames
The following are some examples:
chmod og-x nice
chmod ugo=rw t*
chmod a=--- BYE
chmod o+w passwd
To set permissions on the directory temp:
chmod g=rwx temp
ls -ld temp
Multiple files may be changed in one command. For example, to remove all permissions from files for the group and other domains:
chmod og-rwx *
ls -l
LADM (12/02) 135 © Flying Pigs Training Exercise
Copy the file /etc/group into your home directory.
Set the permissions on the file group as follows:
No permissions for the user No permissions for the group No permissions for other users
List the file entry to confirm that the permissions are correct.
Try to display the file group. What was the outcome?
Try to append text to the end of the file group. What was the outcome?
Hint: use echo End of file >> group
Try to remove the file group. Note the warning message that appears and enter n so as not to remove the file.
Reset the file permissions as follows:
Read and write for the user Read for the group Read for other users
LADM (12/02) 136 © Flying Pigs Training chmod - More on Changing Permissions
The command chmod may also be used to add or remove permissions on a file or files using a set of three octal numbers. Both methods have the exactly same effect.
The syntax for using numbers is:
chmod nnn filenames
This method works by giving the permissions r, w ,x and - numbers as follows:
r 4 w 2 x 1 - 0
For example, the following sets of permissions are represented by numbers thus:
rwx 7 rw- 6 r-x 5 r-- 4 --- 0
However, when chmod is used in this way, all three domains must be set.
For example, to set permissions for a set of files to rw- r-- r--:
chmod 644 *
ls -l
To set permissions for a directory to rwx r-x r-x:
chmod 755 temp
LADM (12/02) 137 © Flying Pigs Training Exercise
List the files in the directory temp.
Now set the permissions on the directory temp as follows:
No permissions for the user No permissions for the group No permissions for other users
List the directory entry to confirm that the permissions are correct.
Hint: use the command ls -ld temp
List the files in the directory temp again. What was the outcome?
Try to change into the directory temp. What was the outcome?
Try to remove files in the directory temp. What was the outcome?
Reset the directory temp permissions as follows:
Read, write and execute permissions for the user Read and execute permissions for the group Read and execute permissions for other users
LADM (12/02) 138 © Flying Pigs Training Sticky Bit
On most systems the sticky bit, the t bit, is set on the directory /tmp. To display the directory use:
ls -ld /tmp
When the sticky bit is set on a directory then any user may create files in that directory but these files are protected such that they can only be deleted by their owner or by root.
The sticky bit may be set as follows:
chmod +t temp SUID Bit
On most systems the SUID bit, the s bit, is set on the passwd command. To display this file use:
ls -l /bin/passwd or ls -l /usr/bin/passwd
The command passwd is used by ordinary users to change their passwd.
However, users do not have write permissions to the file /etc/passwd:
ls -l /etc/passwd
To enable a user to change their passwd, the system administrator can set the SUID bit on the passwd command.
This setting enables the user to change their password, as they assume the permissions of root when they run the passwd command.
The SUID bit may be set as follows:
chmod u+s nice
LADM (12/02) 139 © Flying Pigs Training umask - Default File Permissions
When a user logs in they are assigned a set of default permissions for new files or directories.
The command umask may be used to view these default permissions:
umask
To determine the permissions mode for new directories the current umask is subtracted from octal 777.
To determine the permissions mode for new files the current umask is subtracted from octal 666.
The following table shows some typical settings of umask and the resulting permissions on directories and files.
Umask Directory File 777 666 Default 022 755 644
Insecure 002 775 664
Secure 077 700 600
For example, for directories a umask of 022 gives a setting of 755:
777 - 022 = 755 = rwx r-x r-x
A umask of 022 for files gives a setting of 644:
666 - 022 = 644 = rw- r-- r--
LADM (12/02) 140 © Flying Pigs Training umask - Changing Default Permissions
The command umask may also be used to set the default permissions.
For example to make your files more secure, use a umask of 077 which gives a default setting for files of 600:
umask 077
touch ufile
ls -l ufile
umask 022
LADM (12/02) 141 © Flying Pigs Training New Group - newgrp
The newgrp command allows a user to temporarily join another group provided that the users id is associated with the group in the file /etc/group.
The default groups for a user are shown by the command id:
id
A list of available groups can be found in the file /etc/group:
more /etc/group
To use a second group, the users id must be added by the root user to the group entry in this file, as in this example entry:
staff:x:600:train1,train2,train3
Here the users train1, train2 and train3 may use the group staff.
To use the new group enter the following command:
newgrp staff
This will fork a new shell. Now, any new files or directories created will be in the new group:
touch afile ; ls –l afile
To return to the original shell and the default group enter:
exit
LADM (12/02) 142 © Flying Pigs Training Change Ownership - chown
If the superuser copies a file into a users’ directory, the file will still be owned by root rather than by the user. The command chown allows the root user to change the ownership of a file or files.
For example, as user root:
cd /home/train1
cp /etc/group .
ls -l group
chown train1 group
ls –l group
Change Group - chgrp
The file /etc/group contains a list of available group names.
cat /etc/group
The chgrp command allows the root user to change the group membership of a file or files.
For example, as user root:
cd /home/train1
ls -l group
chgrp users group
ls –l group
The chown command may be used to change both ownership and group membership:
chown train1:users group
LADM (12/02) 143 © Flying Pigs Training Installation
Objectives
At the end of this section the delegate will be able to:
install the Linux system
describe the role of the systems administrator
log on as root, the super user
LADM (12/02) 144 © Flying Pigs Training Installing Linux
Linux can be installed on either a laptop, on a standard PC or into a VM. The installations are very similar and the following briefly outlines how to install the software on the latter, a VM.
The software used is CentOS, an unbranded version of Redhat Enterprise.
Installation Summary
The installation can be summarized as follows:
Install Linux using VMWare or Oracle VirtualBox
When installing CentOS, choose RedHat Enterprise Linux 5
Choose size of 30GB, split disc
Customise memory 1000, network NAT, no sound
Add 2 new discs of 10GB each
Install and create a custom layout on sda
Use swap 2000, /boot 100, / 5000, free 1000+
Network, DHCP, hostname centos.training
Password root123
Software, server, server GUI, include Development
Install
Final tasks
Firewall on
SELinux enforcing
Add a new user
Reboot
LADM (12/02) 145 © Flying Pigs Training The System Administrator
The typical duties of a system administrator vary depending on the number of systems supported and how the duties are divided up. It is not uncommon for system administrators to be experts in administering one or more areas and be inexperienced in others.
For example, some administrators specialise in network administration; others in managing user accounts.
The following is a list of typical system administration duties:
Administering user and group accounts
Administering file systems
Backing up and restoring files and partitions
Administering network services
Administering mail and print services
Identifying problems with search paths, permissions and ownership
To accomplish these tasks, they need to know when and how to perform the following tasks:
Gain full access to all file systems and resources
Shut down and start up the system
Monitor users, files and processes
Use and understand shell scripts and commands
Grant users permissions for specific tasks
Communicate with users
Install third party software
LADM (12/02) 146 © Flying Pigs Training Understanding Superuser Status
The superuser is a privileged user with unrestricted access to all files and commands on the system. The user name for this account is root and the terms root and superuser have the same meaning and can be used interchangeably. The user has the special UID, user ID, of 0 in the password file. The account may be displayed as follows:
head -1 /etc/passwd
An administrator must be root to perform many system administration tasks, such as mounting and unmounting file systems, changing ownership or permissions for a file or directory, backing up and restoring file systems and shutting down the system.
They can become superuser in one of two ways:
When logged in as another user, by typing the su, switch user, command with no arguments, and then typing the root password
From a login prompt, by typing root and then typing the root password
When they have superuser privileges, the shell provides a special #, hash sign, prompt to remind them that they have extra access to the system.
They should become superuser only when it is required, and avoid doing any routine work as superuser. When a task requires them to log in as root, they should switch user to root, perform the required tasks and exit superuser status when the task is complete. To switch to root use:
su - root or su -
Because unauthorised access to root can be a serious security breach, the user should always have a password, which, for enhanced security, should be changed frequently. Note that the default shell for root is usually the Bourne shell.
Note that if the hyphen is not used with this command, as in: su this is not the same as logging in as the new user. This format of the command does not set the new user environment but retains some of the settings of the original user.
LADM (12/02) 147 © Flying Pigs Training sudo – Commands as Root
Rather than switching into the user root, the sudo command allows a permitted user to execute a command as the superuser provided:
this is specified in the /etc/sudoers file
## Allows people in group wheel to run all commands %wheel ALL=(ALL) ALL
they are a member of the wheel group in /etc/group
wheel:x:10:root,user1
By default, sudo requires that users authenticate themselves with a password (by default this is the user's password, not the root password). Once a user has been authenticated, a timestamp is updated and the user may then use sudo without a password for a short period of time. sudo – Example
The command more may be used to list the contents of file /etc/passwd, a special file which contains an entry for every user login on the system.
more /etc/passwd
However, this file does not contain encrypted passwords. These are held in a restricted file which cannot be viewed by a user other than root: cat /etc/shadow The sudo command, if available, may be used to view the file: sudo cat /etc/shadow sudo – Root Password
When installing CentOS, the user is asked for a root password. This is not the case on Debian based systems. To change the root password on Ubuntu for example, run the following:
sudo passwd
LADM (12/02) 148 © Flying Pigs Training Package Management
Objectives
At the end of this section the delegate will be able to:
update the operating system using packages
LADM (12/02) 149 © Flying Pigs Training Managing Packages
Most modern Linux distributions have their own package managers:
Red Hat based systems such as CentOS use Red Hat Package Manager, rpm and yum
Debian based systems such as Ubuntu use apt-get, dpkg and alien
CentOS Package Management
The rpm utility has become quite popular and is found in CentOS, Red Hat, Mandriva, and even Suse/Novell systems. It can install local and remote files and can also determine the state of installed files.
The yum command is the most sophisticated way to manage packages. One of the primary benefits of yum is that it will automatically discover dependencies and then install them.
After installation from a DVD, to upgrade all installed packages:
yum update
The following are examples of using bpth rpm and yum.
An RPM file is named using the following convention:
name-version-release.architecture.rpm
For example:
figlet-2.2.1-1.2.el4.rf.i386.rpm
To install a file with verbose output and showing hashes to denote installation progress:
rpm –ivh figlet-2.2.1-1.2.el4.rf.i386.rpm
rpm -ivh figlet*
yum install figlet
Test the install with:
figlet hello
LADM (12/02) 150 © Flying Pigs Training To list available packages:
rpm –qa | less
yum list installed | less
To determine if a particular package is installed:
rpm -qa | grep figlet
yum list installed figlet
To remove the same package (note the missing rpm suffix):
rpm -e figlet-2.2.1-1.2.el4.rf
yum remove figlet
To list all files in the package, as well as the package description:
rpm -qpil figlet*
To upgrade an RPM package:
rpm -U figlet-2.2.1-1.2.el4.rf.i386.rpm
To discover what package an installed file comes from:
which figlet
rpm -qf /usr/bin/figlet
To ignore dependency problems and install the file:
rpm - -nodeps figlet-2.2.1-1.2.el4.rf.i386.rpm
To verify the signature on an RPM file you want to install:
rpm - -checksig figlet-2.2.1-1.2.el4.rf.i386.rpm
LADM (12/02) 151 © Flying Pigs Training Oracle VirtualBox Pre-requisites
After installation of CentOS as a VM, the following packges are required in order to install VirtualBOX Guest Additions:
yum install kernel-devel kernel-headers
yum upgrade kernel*
To install the gcc package:
yum install gcc
To install the following optional package for mail:
yum install mailx
To install the package figlet:
rpm -ivH /scr/figlet*
LADM (12/02) 152 © Flying Pigs Training Debian Package Management
The apt-get and apt-cache commands are the most sophisticated way to manage Debian packages. One of the primary benefits of apt-get is that it will automatically discover dependencies and then install them. This feature is different from RPM, which will simply notify a failure and then quit.
The command uses the /etc/apt/sources.list file which contains a default set of repositories. It is possible to edit this file and add various repositories, depending on what is to be installed.
less /etc/apt/sources.list
The distribution can be either the release code name (lenny, etch, squeeze, maverick) or the release class (stable, oldstable, testing, unstable) respectively. If tracking a release class use the class name, if tracking a Debian point release, use the code name.
Once added, the user can issue commands to install applications and daemons.
Run this command after changing the /etc/apt/sources.list file:
apt-get clean
apt-get update
After installation from a DVD, to upgrade all installed packages:
apt-get upgrade
To find a package to install:
apt-cache search vim | grep vim-tiny
To show information about the package:
apt-cache show vim-tiny | less
To check to see if it is already installed:
apt-cache policy vim-tiny
To install the package:
apt-get install vim-tiny
LADM (12/02) 153 © Flying Pigs Training To uninstall everything in the same package, except for the configuration files:
apt-get remove vim-tiny
To uninstall the entire package, including its configuration files:
apt-get - -purge remove vim-tiny
Using dpkg The dpkg command is not as sophisticated as apt-get. It will not install dependencies as easily and it is best for installing local files. However, it must be used when you cannot get a package using apt-get or when you want to list information about installed packages. It is also useful when you want to list the contents of a package before installing it.
An Debian file is named using the following convention:
name-version-release.architecture.deb
For example:
fakeroot_1.12.1ubuntu1_i386.deb
To search the listing of packages:
less /var/lib/dpkg/available
grep Package /var/lib/dpkg/available | less
To list packages related to a package:
dpkg -l “*cups*”
To see the entries in /var/lib/dpkg/available of a package:
dpkg - -print-avail cups vim-tiny | less
To install a package, first find it in an archive or on a CDROM:
cd /media/U*/pool/main/f/fakeroot ; ls
man fakeroot
dpkg -i fakeroot*.deb
LADM (12/02) 154 © Flying Pigs Training To remove an installed package:
dpkg -r fakeroot
To uninstall all elements in the package provided there are no dependencies:
dpkg - -purge vim-tiny
The /var/lib/dpkg/ directory contains various files, including /var/lib/dpkg/status, which dpkg uses to remember what has been installed. If this file is missing, dpkg will not work properly. Recover the file from the /var/lib/dpkg/status-old file.
Using alien The alien command converts packages from other managers to Debian packages. It can convert the following packages:
RPM Slackware (.tgz) Solaris (pkg)
For example, to convert the figlet rpm file issue the following command:
alien figlet-2.2.1-1.2.el4.rf.i386.rpm
Alien will convert the file into a new file with a .deb ending. You can then install it using dpkg. You can also use alien to convert Debian packages to RPM and other formats.
LADM (12/02) 155 © Flying Pigs Training Startup and Shutdown
Objectives
At the end of this section the delegate will be able to:
describe the system files used at start up and shutdown
start up and shutdown the system
LADM (12/02) 156 © Flying Pigs Training Booting the System
Linux machines. like the newer Windows systems, cannot just be switched on and off. Linux is a multi-tasking, multi-user operating system which will always have processes, other than your own, running on the system at any one time.
In order to be able to do this, Linux keeps much of its information in memory, updating its disk only when more memory is required, or when instructed. If the machine is simply switched off, then there is no guarantee that all of the buffers will be written back to disk. This could result in lost files or at worst a corrupted file system.
It is therefore helpful to understand the startup and shutdown procedures. This is also useful when we are required to configure these procedures. For instance, how do we get things like the print spooler, cron, system accounting, etc. to startup automatically.
A Linux system will go through a number of stages before multi-user mode is acquired.
The BIOS
When an x86 computer is booted, the processor looks at the end of system memory for the Basic Input/Output System or BIOS program and runs it. The BIOS controls not only the first step of the boot process, but also provides the lowest level interface to peripheral devices. For this reason it is written into read- only, permanent memory and is always available for use. Other platforms use different programs to perform low-level tasks roughly equivalent to those of the BIOS on an x86 system.
Once loaded, the BIOS tests the system, looks for and checks peripherals and then locates a valid device with which to boot the system. Usually it checks any diskette drives and CD-ROM drives present for bootable media, then, failing that, looks to the system's hard drives. In most cases, the order of the drives searched while booting is controlled with a setting in BIOS, and it looks on the master IDE device on the primary IDE bus.
The BIOS then loads into memory whatever program is residing in the first sector of this device, called the Master Boot Record or MBR. The MBR is only 512 bytes in size and contains machine code instructions for booting the machine, called a boot loader, along with the partition table. Once the BIOS finds and loads the boot loader program into memory, it yields control of the boot process to it.
LADM (12/02) 157 © Flying Pigs Training The Boot Loader
Under Red Hat Linux two boot loaders are available: GRUB or LILO. GRUB is the default boot loader, but LILO is available for those who require or prefer it.
Both boot loaders for the x86 platform are broken into at least two stages. The first stage is a small machine code binary on the MBR. Its sole job is to locate the second stage boot loader and load the first part of it into memory. GRUB is the newer boot loader and has the advantage of being able read ext2 and ext3 partitions and load its configuration file, /boot/grub/grub.conf at boot time.
With LILO, the second stage boot loader uses information on the MBR to determine the boot options available to the user. This means that any time a configuration change is made or the kernel is manually upgraded, the /sbin/lilo command must be executed to write the appropriate information to the MBR.
Once the second stage boot loader is in memory, it presents the user with the Red Hat Linux initial, graphical screen showing the different operating systems or kernels it has been configured to boot. On this screen a user can use the arrow keys to choose which operating system or kernel they wish to boot and press Enter. If no key is pressed, the boot loader will load the default selection after a configurable period of time has passed.
Once the second stage boot loader has determined which kernel to boot, it locates the corresponding kernel binary in the /boot/ directory. The kernel binary is named using the format /boot/vmlinuz-
The boot loader then places the appropriate initial RAM disk image, called an initrd, into memory. The initrd is used by the kernel to load drivers necessary to boot the system. This is particularly important if SCSI hard drives are present or if the system uses the ext3 file system
Once the kernel and the initrd image are loaded into memory, the boot loader hands control of the boot process to the kernel.
LADM (12/02) 158 © Flying Pigs Training Using GRUB
When you power on your computer, the first software that runs is a bootloader that invokes the computer's operating system. GRUB, the GRand Unified Bootloader, is an integral part of many Linux systems. It starts the Linux kernel.
GRUB can boot multiple operating systems on computers that have more than one installed on their hard drives. Among GRUB's features are:
Dynamic configuration. Users can change settings and parameters at boot time
Support for multiple executable formats and hard drive filesystems
Both a graphical and command-line interface to let users choose what operating system to boot
One common task preformed by GRUB in the Linux world, besides booting the kernel, is to allow users to dual boot Linux and Microsoft Windows. The typical pattern for creating a dual boot system is to install Windows, if it is not already installed, then install Linux on another hard drive or on an empty partition. Most Linux distributions will detect the Windows installation and automatically install and configure an open source boot loader such as GRUB or LILO to boot both Linux and Windows, replacing the Windows bootloader.
Most Linux distributions that use GRUB come with it installed and ready to use. Many of the distributions that do not have GRUB installed by default have it available in their package systems; check there first before doing a manual installation.
LADM (12/02) 159 © Flying Pigs Training Configuring the GRUB Menu
Most users will want GRUB to automatically boot a specific operating system, or present them with a way to select what system they want from among all that are installed on the hard drive. That is the function of the GRUB menu; a configuration file called menu.lst that goes into the grub directory on the drive on which GRUB is installed. If GRUB finds this file during boot it will automatically load the menu. Changes to this file are automatically picked up when the boot loader is run.
The GRUB menu file can be viewed as:
less /boot/grub/menu.lst
This file is a plain text file with a set of directives and configuration parameters.
default specifies which entry is the default. An entry comprises, at least, title, root, and kernel directives. Entry numbers start at 0 and increment upward.
timeout specifies how long, in seconds, the menu will be displayed before the default entry is executed.
splashimage is the image that will be displayed.
title is the text that is displayed in the menu for the entry that follows.
root tells GRUB on what device and partition it can find the kernel for this entry.
kernel specifies what kernel will be booted if this entry is selected. Options after this directive are passed to the kernel for processing. Here, ro means read-only (the device is not physically made read-only; it just tells the kernel not to attempt writing), quiet indicates not to display debug information and splash means show a splash screen while booting.
initrd tells GRUB what to run after the kernel has been loaded. When this directive is executed, GRUB passes off control of the system to the operating system.
Changes to the menu can be made in one of two ways:
simply edit the menu.lst file
enter the GUI interface by typing an 'e' when the GRUB menu is invoked
LADM (12/02) 160 © Flying Pigs Training The Kernel
The kernel file can be viewed thus:
ls -l /boot/vm*
When the kernel is loaded, it immediately initializes and configures the computer's memory and then configures the various hardware attached to the system including all processors, I/O subsystems and storage devices.
It then looks for the compressed initrd image in a predetermined location in memory, decompresses it, mounts it and loads all necessary drivers.
Next, it initializes virtual devices related to the file system, such as LVM or software RAID before unmounting the initrd disk image and freeing up all the memory the disk image once occupied.
The kernel then creates a root device, mounts the root partition read-only, and frees any unused memory.
At this point, the kernel is loaded into memory and operational. However, since there are no user applications that allow meaningful input to the system, not much can be done with it. In order to set up the user environment, the kernel then executes the /sbin/init program. The init Process
The /sbin/init program (also called init) coordinates the rest of the boot process and configures the environment for the user.
When the init command starts, it becomes the parent or grandparent of all of the processes that start up automatically on a Red Hat Linux system. First, it runs the /etc/rc.d/rc.sysinit script, which sets the environment path, starts swap, checks the file systems and takes care of everything the system needs to have done at system initialization.
The init command then runs the /etc/inittab script, which describes how the system should be set up for each runlevel. Among other things, the /etc/inittab sets the default runlevel and dictates that /sbin/update should be run whenever it starts a given runlevel.
LADM (12/02) 161 © Flying Pigs Training /etc/inittab File
The init process looks at a file called /etc/inittab to find out which processes to start and also when and how to start them. The init process will also monitor its child processes. When one terminates, it will again consult the inittab file to find out what to do next.
Each line in the inittab file defines a process to run and has four fields separated by a colon.
LABEL : RSTATE : ACTION : PROCESS
View the file as follows:
less /etc/inittab
The Label Field
Each line must have its own unique label. Whenever init starts or stops a process, it uses this label to identify the process.
The Rstate Field
The init command uses run levels or run states to identify when to run processes. Whenever a run state changes, init consults the inittab file and any lines which have a matching run state in this field will then be run.
Run states can be listed as single values, multiple values, or if the field is left blank this means all run levels.
Linux supports seven different run levels, typically used in the following manner:
0 Halt 1 Single user mode 2 Multi-user without NFS 3 Multi-user mode – boot up in text mode 4 Unused 5 Multi-user mode – boot up in X Windows 6 Reboot
The current run state can be found as follows:
who -r
LADM (12/02) 162 © Flying Pigs Training The Action Field
The action field defines how to run the process. The following are some of the actions that can be used: initdefault This is a special flag, typically on the first line of Inittab, which tells Init what run state to set when it first starts. The run state to be set is specified in the run state field. sysinit Used for programs that initialise devices, usually during bootup. wait Runs the specified process then waits for termination before reading next line. powerfail Some machines have a capability to signal a power failure. If this signal is seen, run the specified process. respawn Runs the specified process. If the process ever terminates restart it. Used mostly on getty processes for terminals and modems.
Examine the file inittab to see what steps are taken once init has started.
LADM (12/02) 163 © Flying Pigs Training The Process Field
This field defines the command to be executed. Typically this is a special run control script. The init command uses the same control script for each run level. The script may be viewed as follows:
less /etc/rc
The control script is actually a link located in the /etc directory and is passed a different parameter for each run level.
ls -ld /etc/rc*
The script executes files in a set of directories in the /etc directory. These define the sequence in which scripts are performed within each run level. For example, the /etc/rcN.d directory contains files that start and stop processes for run level N. For example to view run level 5 use:
ls /etc/rc5.d
The scripts here have the following characteristics:
The scripts are always run in ASCII sort order
The names of the scripts have the form [K,S] [0-9] [A-Z] [0-99]
Files beginning with S are run to start a system process
Files beginning with K are run to terminate or kill some system process
LADM (12/02) 164 © Flying Pigs Training Loading New Software
If a user created script is to be loaded at startup, a file of the form SNNname may be added to the default directory, for example /etc/rc5.d. Exercise - optional
To illustrate how to load a script at startup, create a file, S99MESS, in the directory /etc/rc5.d.
On startup the script will display a message of the form:
+++++ system name STARTED SUCCESSFULLY +++++
Hint: use the figlet command and back quotes with the command hostname to display the system name
Whilst still logged on, test the new script. It will be properly tested later.
LADM (12/02) 165 © Flying Pigs Training Shutdown Commands
Shutdown An executable shell script that calls the init process to shut down the system. To see the options available try:
shutdown
This command is recommended for most servers because users are notified of the impending shut down as are the systems that are mounting resources from the server being shut down.
If the following command is executed, it will do an immediate shutdown with a grace period of 3 minutes and display a warning message.
shutdown –t180 now LINUX IS SHUTTING DOWN
This command shuts down at a particular time. However, the –k option only tests the command it does not actually shutdown:
shutdown -k 13:15 LINUX IS SHUTTING DOWN
Reboot An executable that syncs the disks and reboots the server:
reboot
However, it may be better to use the following command which will do a more graceful shutdown and startup at 2.40pm:
shutdown –r 14:40
Halt, Poweroff The command halt is an executable that syncs the disks and stops the processor. It may be used but is not recommended because it doesn’t execute the /etc/rc0 script, which stops all processes, syncs the disks and un-mounts any remaining file systems.
Exercise
Reboot the server and check that the new script is executed successfully on startup.
LADM (12/02) 166 © Flying Pigs Training Service and Printer Management
Objectives
At the end of this section the delegate will be able to:
mangage, start and stop services
install and manage printing services
LADM (12/02) 167 © Flying Pigs Training Overview of Services
Services are programs or daemons that once started run continuously in the background. Some wait for input and some monitor changes in the server and respond to them. For example the Apache server has a daemon called httpd (the d is for daemon) that listens on port 80 on the linux server and when it receives a request for a page it sends the appropriate data back to the client machine.
Many services are required to run all the time however many can be safely turned off for both security reasons, as running unnecessary services opens doors into the server, and for performance reasons. If the user turns off a service and loses some functionality they can just turn it on again without doing any harm. For example, the user may decide to start services such as Apache, MySQL and Samba manually as required.
There are 2 main commands used to control services. chkconfig This controls which services are set to start on boot; by their nature these settings are saved and are applied at next boot. Changing these settings will not start the service immediately it will just flag the service to be started from the next boot. service This controls the starting and stopping of services during a session; these settings are not saved. If Apache is started this way but is not set to start on boot, it will continue to run but on next boot will not start automatically.
GUI Tools
There are two ways of managing services, at the command line and via the graphical tools. On CentOS the command is system-config-services. This can be opened from System > Administration > Services or at the command line:
system-config-services
On Debian systems such as Ubuntu the Synaptic Package Manager is an excellent tool for finding, fetching and installing packages. Press System -> Administration -> Synaptic Package Manager to start Synaptic.
From these screens the user can tell which services are set to start on boot, those denoted with a tick in the checkbox, and which are currently running. They can also start, stop and restart services from here.
Note that if the user starts a service it will not be set to start on next boot unless they and save their settings.
LADM (12/02) 168 © Flying Pigs Training /etc/rc.d/init.d scripts
In order to understand how chkconfig works, it's necessary to review how scripts and run level directories are set up. The scripts to run services are all located in /etc/rc.d/init.d on Red hat and CentOS systems although this may differ on other systems.
There are a set of run level directories under /etc named rc0.d, rc1.d, and so on that map to the various run levels. In order to execute a script (start a service) in a specific run level, a symbolic link to the service's script in /etc/rc.d/init.d is created in the directory of interest. So, for example, if the isdn script is to be run for run levels 4 and 5, but not for 0-3 or 6, a symlink to /etc/rc.d/init.d/isdn is created in both /etc/rc4.d and /etc/rc5.d.
This way, if the script for a service needs to be modified, only the original script in /etc/rc.d/init.d needs to be dealt with. The symlinks for each run level automatically pick up the changes.
In order to have a script execute for a certain run level, all that's needed is to create a symlink in that run level's 'rc' directory to the original script file. chkconfig, by virtue of the arguments passed to it, takes care of this.
LADM (12/02) 169 © Flying Pigs Training chkconfig Command
To get information about services use the –list option to the chkconfig command. To see the status of all services type:
chkconfig - -list
This returns a long list of all the services, each column refers to a different run level. In most cases the server is booted into level 5 so this is the column of most interest. The on and off refer to whether the service is set to start on boot, it does not state whether the service is currently running.
To query the status of just one service, use grep to filter the returned data. Here we use chkconfig but only want to see the Apache service.
chkconfig - -list | grep httpd
This shows that Apache is not set to start on boot in all run levels.
Use grep again to see only those services not set to start on boot for a particular run level. Here we see services not set to start in run level 5.
chkconfig - -list | grep 5:off | less
To see services that are set to start in run level 5:
chkconfig - -list | grep 5:on | less
To alter which services start at boot time run chkconfig with different arguments. For example to set Apache to start on boot in run level 5:
chkconfig - -level 5 httpd on
To set Apache to start on boot in run levels 3, 4 and 5:
chkconfig - -level 345 httpd on
chkconfig - -list | grep httpd
To stop Apache starting on boot, replace 'on' with 'off':
chkconfig - -level 345 httpd off
LADM (12/02) 170 © Flying Pigs Training To remove services from chkconfig control use:
chkconfig - -del sshd
chkconfig - -list | grep sshd
This will remove the symlinks in all of the run level directories, which effectively turns the service off at startup. To add the script back in for its’ default run levels:
chkconfig - -add sshd
chkconfig - -list | grep sshd
So how does chkconfig know what to do? If you pass the - -add argument, how does chkconfig know which run levels are the default for the particular script?
The chkconfig program looks inside the original script (inside /etc/rc.d/init.d) for the comment line that begins #chkconfig:
grep chkconfig /etc/init.d/sshd
This returns: # chkconfig: 2345 55 25
The first argument in this comment identifies which run levels are the defaults, for example 2, 3, 4, and 5.
The second argument is the order of the script during startup, for example 55.
The third argument is the order that the service should be run when killed, for example 25
These arguments, S55 and K25, are used for naming the symlinks for this script for startup and kill:
find /etc/r*/rc* -name “*sshd” 2>/dev/null
LADM (12/02) 171 © Flying Pigs Training Service Command
The user can manually start a service when required. This way they can turn off many services not actually required for the normal operation of the server and just start them when the need arises.
To check to see if a service is running use:
service httpd status
To start a service simply use the service command, this example uses Apache (httpd service) as an example but the command is the same for any service:
service httpd start
service httpd status
Stopping a service is just as easy:
service httpd stop
Restarting just uses restart in place of start or stop:
service httpd restart
Note that this will start a service for this session but after rebooting this service may not automatically restart.
To view a list of all the services use:
service - -status-all
To view a list of all the currently running services:
service - -status-all | grep running
To see a list of all stopped services:
service - -status-all | grep stopped
LADM (12/02) 172 © Flying Pigs Training /etc/init.d The service command with start runs the startup script for the service. The script for httpd may be viewd thus:
less /etc/init.d/httpd
If the service command is not available, the service may be started and stopped by directly running the script:
/etc/init.d/httpd start
/etc/init.d/httpd stop Debian Linux
Traditionally, Debian systems such as Ubuntu provided various tools to manage services:
/etc/init.d/service rcconf update-rc.d
Under Red hat and Centos the chkconfig command can be used to configure Sys V style init script links and the service command can stop, start and restart services.
To use the chkconfig and service commands on Debian distributions, first install the following packages:
apt-get install chkconfig sysvinit-utils
This package also installs a simple GUI for managing run levels:
sysv-rc-conf
LADM (12/02) 173 © Flying Pigs Training CUPS Printing System
CUPS is the Common UNIX Printing System. It is aimed at providing a common printing interface across a local network masking differences among the printing systems on each computer. It also provides interactivity with SMB and Windows printers and allows dynamic printer detection and grouping.
CUPS is licensed under the GPL by a company called Easy Software Products. Besides providing CUPS itself as free software the company also provides commercial add-ons and support.
CUPS can be seen as a replacement for the LPD printing system. It replaces the lpr command with its own and the LPD printer drivers with its own versions. However, CUPS is similar to LPD in that it uses PostScript as its underlying language for page descriptions. Linux (and UNIX) programs don't know the difference between CUPS and LPD. Installation
Install the CUPS package as follows:
yum install cups
service status cups
After a short delay the CUPS software will be installed and the scheduler will be started automatically.
If a real printer is not available a device may be used. To enable this facilty first update the file /etc/cups/cupsd.conf as follows:
echo "FileDevice Yes" >> /etc/cups/cupsd.conf
Then restart the service:
service cups restart
LADM (12/02) 174 © Flying Pigs Training Managing Printers
Each printer queue has a name associated with it. Case is not significant. Printer queues also have a device associated with them. The device can be a parallel port, a network interface and so forth. Devices within CUPS use Uniform Resource Identifiers (URIs) which are a more general form of Uniform Resource Locators (URLs) that are used in a web browser.
For example, the first parallel port in Linux usually uses a device URI of:
parallel:/dev/lp1
For a complete list of supported devices run:
lpinfo –v
The first word in each line is the type of device, direct, file, network, or serial. This is followed by the device URI or method name for that device.
Finally, printer queues usually have a PostScript Printer Description ("PPD") file associated with them. PPD files describe the capabilities of each printer, the page sizes supported, etc., and are used for PostScript and non-PostScript printers.
Adding a Printer
CUPS provides two methods for adding printers: a command-line program called lpadmin and a Web interface. The lpadmin command allows the user to perform most printer administration tasks from the command-line and is located in /usr/sbin. The Web interface is located at: http://localhost:631/admin
To add a printer run the lpadmin command with the -p option:
lpadmin -p lp1 -v parallel:/dev/lp1
To view the print queue run:
lpstat -t
Note that the new printer is both disabled and is not accepting print requests.
LADM (12/02) 175 © Flying Pigs Training Accept and Reject
The accept and reject commands accept and reject print jobs for the named printer. For example:
accept lp1 ; lpstat -t
A printer can be stopped but can still be accepting new print jobs. A printer can also be rejecting new print jobs while it finishes those that have been queued. This is useful when maintenance is needed on the printer and it will not be available to users for a long period of time.
Set Default Printer
Run the lpadmin command with the -d option to set a default printer:
lpadmin -d lp1
To print a file and view the print queue:
lp /etc/passwd
lpstat -t Enable and Disable
The cupsenable and cupsdisable commands start and stop printer queues. For example:
cupsenable lp1 ; lpstat -p lp1
Printers that are disabled may still accept jobs for printing, but won't actually print any files until they are restarted.
Deleting a Printer
Run the lpadmin command with the -x option to delete a printer:
lpadmin -x lp1
LADM (12/02) 176 © Flying Pigs Training User Management
Objectives
At the end of this section the delegate will be able to:
describe relevant system files for managing users
add, modify and remove users and groups
describe how to control access to the system
LADM (12/02) 177 © Flying Pigs Training /etc/profile File
This file is run for every user who logs onto the system. View the file using:
less /etc/profile
/etc/passwd File
Each line in the /etc/passwd file defines a user account. In order to add a user to the system this file must be modified. View the file using:
less /etc/passwd
Each user entry is split into seven fields as follows, each separated by a colon:
Login : Password : Userid : Groupid : Description : Home Directory : Program
Every user on the system must have read permission on the /etc/passwd file but no one should have write permission including root, since this will help prevent accidental corruption or deletion. If the file becomes corrupted, then no one will be allowed to log onto the system. Unless a root login is already running, the system will have to be booted from floppy disc or tape to cure the problem.
Login This is the account name for the user and can be up to eight characters long. Any lower case letter or digit may be used in the name, but it is worth avoiding punctuation and non-printing characters. Login names must be unique.
The following command may be used to check whether a particular login exists:
grep '^root' /etc/passwd
LADM (12/02) 178 © Flying Pigs Training Password This field is no longer used and will contain an x character. Historically it contained the users encrypted password. It was then very easy for any user to look at the password file and find all users who did not have a password set.
This is now impossible as the encrypted password is stored in another file called /etc/shadow. This file is read permission only to the super user.
less /etc/shadow
Users can assign themselves passwords using the passwd command.
Userid This is a unique number assigned to each user to identify them to the system. The system always stores the identity of a user as a number and then looks up the user login when necessary. When adding a new user the next available userid should be used but users should not have the same userid.
Note that all users with a userid of zero are super users so as well as root there can be other super user logins.
Groupid This field contains the number of the group to which the users belongs. Valid groups are defined in the /etc/group file.
less /etc/group
Description This field is available for comments about a particular account.
Home Directory This field specifies the full path name of the users home directory. Note that if the directory does not exist or a mistake is made in the pathname, then a user will not be able to login.
Program This is the full path to the shell program which the user will run upon entry to the system. When this program terminates, the user will be logged off.
This field, although typically a shell of one kind or another, can be an application or any other program required to start at login. Checking the passwd File
The command pwck may be used to check the password file for errors.
pwck
LADM (12/02) 179 © Flying Pigs Training Adding a User
The process of adding a new user can be carried out in one of several ways:
create the user manually by entering commands
use the command useradd
use the GUI interface
Creating a User Manually
Although it is not normally done this way, the steps involved in manually adding a new user to the system are as follows.
Log in as root and if required, edit the /etc/group file and add a new entry
Edit the /etc/passwd file and add the new user
Create a new home directory for the user
Change ownership and group of this directory for the new user using the commands chown and chgrp
Create a new password for the user
Log out and log in as the new user
LADM (12/02) 180 © Flying Pigs Training Creating a User - useradd
A new user may be added to the system using the command useradd. This command has many options including the following:
-c "comment" Use open and close double quotes for the comment
-d directory Full pathname of the new user's home directory
-m Create the new home directory if it does not already exist
-g group The group the user is placed in; the default is a new group with the name of the user
-s shell Full pathname of the shell on login, the default is /bin/bash username The new username
The following example adds a new user, here specifying only a comment and the home directory:
useradd -c "User userb" -d /home/userb -m userb
This example adds a new user with a group of users and a default shell of csh:
useradd -d /home/userc -m -g users -s /bin/csh userc
To check the existence of a new entry in the passwd file:
tail /etc/passwd
To check that the users home directory exists:
ls -l /home
Before the account can be used, a new password must be added for the user:
passwd userb
LADM (12/02) 181 © Flying Pigs Training Exercise
Create the following users on the new server, each with a home directory in /home and with a default group and shell:
mary, jane
Also create a new user for your own use.
Log in as your new user and create a new password. Customising the User Environment
Depending on the shell to be used, the following files are executed for each user when they log in and these files can be customised for each user.
/etc/profile All .bash_profile bash .profile sh,ksh,bash .login csh
These files are executed for each user when they fork a new shell:
.bashrc bash .csh csh
The directory /etc/skel contains files including hidden files which are automatically copied into any new user's home directory. A copy of any standard user start up scripts or flies should be stored here. Exercise
Edit the following file in the directory /etc/skel.
vi .bashrc
Customise this file, for example, add the prompt PS1 and create an alias as follows:
PS1=’$PWD: ‘ alias c=’clear’
Exercise
Create the following users on the new server, each with a home directory in /home and with a default bash shell:
fred, alan
Log in as fred and check that the .bashrc file is executed.
LADM (12/02) 182 © Flying Pigs Training /etc/shadow File
Each line in the /etc/shadow file defines a password for a user account. In order to add a user to the system this file must be modified. View the file as:
less /etc/shadow
Each entry is split into nine fields, each separated by a colon. For example:
user1 : erf130hj kd2Z : 14159 : 0: 99999 : 7 : : :
The fields are defined as follows:
1. userid
2. a 13 character encrypted password, or no characters signifying no password
3. date last changed; the number of days between 1 Jan 1970 and the date the password was last modified
4. minimum number of days between password changes
5. maximum number of days the password is valid
6. warn users number of days before the current password is due to expire
7. inactivity, number of days of inactivity allowed for that user
8. expire, an absolute data string when the login may no longer be used
9. flag set to zero for future use
Password Ageing
Users should change their passwords on a regular basis, perhaps every two or three months and the root user at least once a month. This can be enforced by using the password ageing facility using information stored in the /etc/shadow file.
LADM (12/02) 183 © Flying Pigs Training /etc/group File
Users on the system can be split into different groups to allow some users to share files while excluding other users. When a new user is created, the user is assigned a groupid, which is a number that identifies a specific group located in the /etc/group file.
less /etc/group
Each line in this file defines a group and has the following structure:
Name : Password : Groupid : Userlist
Name The name of the group, up to eight characters long.
Password The group password is rarely used since there is no easy way of applying a password to a group.
Groupid A unique number used to identify the group.
Userlist A list of users who are allowed to change into this group. This is not the same as those already belonging to the group as defined by the password file. On some occasions a user may need to access more than one group. By including their names in a comma-separated list, users who can change into this group may be defined. Even if there is no password on the group, a user cannot change to this group unless their name appears in this list. Checking the group File
The command grpck may be used to check the password file for errors.
grpck
LADM (12/02) 184 © Flying Pigs Training Modifying a User - usermod
The details for a user may be modified using the command useradd. For example, the following alters the login shell for the user userc:
usermod -s /bin/bash userc Exercise
Amend the users mary and jane to have a default group of users.
Removing Users
Removing users is simple. However it is probably better to lock an account rather than remove it entirely. The first step is to lock the login. This can be done by using the password command:
passwd -l userb
The username should then be deleted from the /etc/group file and the user's files backed up before they are deleted. However, it may be best just to leave the account locked in case any of the files are needed. Removing Users - userdel
The command to delete a user is userdel. This command deletes a user's login name from the system and makes changes to the system files and to the file system.
For example:
userdel userb
The option –r may be used to remove the user's home directory from the system. The directory must exist but this format should not be used unless there is a certainty that the files owned by the user are no longer required:
userdel -r userb
Exercise
Remove the users jane and fred.
LADM (12/02) 185 © Flying Pigs Training Creating a Group - groupadd
A new group may be added to the system using the command groupadd. The following steps add a new group to the system:
Add the new group, specifying the group name
groupadd staff
Check the existence of a new entry in the group file
tail /etc/group Removing a Group - groupdel
A group may be deleted from the system using the command groupdel as follows:
groupdel staff Modifying a Group - groupmod
The details for a group may be altered using the command groupmod.
Exercise
Add a new group dev1 and check that it exists in the group file.
Modify the users fred and jane to use this new group.
Check that the users are now using the new group.
LADM (12/02) 186 © Flying Pigs Training Using Groups - newgrp
The command newgrp may be used to change a user’s group identification. When a user runs the command, the system places them in a new shell and changes the name of their group to the group specified.
The changes only last for the current session and a user can only change their real group name to a group they are already a member of. However, the root user can change their real group to any group regardless of whether they are a member of it or not.
First, add the user to a secondary group:
groupadd staff
usermod -G staff userc
tail /etc/group
Now that user1 is a member of the group staff, they can change groups as follows:
newgrp staff
Any files created will now be in the staff group.
This command forks a new shell. To leave the shell enter:
exit
Note that the newgrp command does not take input from standard input and cannot be run from within a script. Adding New Commands
If shell scripts or new commands for users are needed, they should be placed in a bin directory, such as /usr/local/bin rather than altering system directories.
Access to this directory is controlled by the PATH variable.
LADM (12/02) 187 © Flying Pigs Training Network Management
Objectives
At the end of this section the delegate will be able to:
configure a Linux system for networking
describe and find out an IP Address
describe how routing works
use network troubleshouting tools
start and stop networking
configure and use the Network File System (NFS)
LADM (12/02) 188 © Flying Pigs Training What is an TCP/IP?
The most significant network protocol standard of today is the Internet Protocol Suite, known as TCP/IP. Developed over the last 20 years, this is a useful and robust protocol that fulfils the basic requirements for computer interoperation, such as remote login, file transfer, mail transfer and process to process communication.
These facilities are available on any system which supports TCP/IP, and provide internetworking between both similar and dissimilar systems. For example, a Windows PC, a Linux server and a DEC VAX system can all communicate using TCP/IP.
The TCP/IP protocols have evolved from the research funded by the United States Defence Advanced Research Projects Agency, DARPA. DARPA funded the integration of TCP/IP into BSD, the Berkeley Software Distribution, version of the UNIX operating system. This version of UNIX was popular with universities and the use of TCP/IP spread rapidly. What is an IP Address?
On a TCP/IP network, each machine is assigned an IP address, which is a 32-bit number uniquely identifying the machine. The IP address is usually represented as a dotted quad; four numbers in decimal notation, separated by dots.
As an example, the following is an IP address: 10.1.1.9.
LADM (12/02) 189 © Flying Pigs Training Finding the IP Address
The name and IP Address of the current server may be held in a special file which can be displayed as follows:
cat /etc/hosts
This will display lines of the form:
127.0.0.1 localhost.localdomain localhost 10.1.1.9 centos.train centos 10.1.1.12 solaris.train solaris
The line 127.0.0.1 is a loop back address.
Here, the current servers are called centos and solaris. The user may use the command hostname to confirm the name of the host server:
hostname ping - Contacting a Server
The command ping may be issued by a user to confirm that another server is on the network. For example, to send a message to a server with an IP Address of 192.168.0.155:
ping 10.1.1.12
If the server is alive and well, the command will return a message otherwise it will hang.
Rather than using the IP Address of the server, the name of the server from the file may be used. For example:
ping centos
ping -c4 $(hostname)
This second example will send a message to the server and stop after 4 iterations.
LADM (12/02) 190 © Flying Pigs Training Network Configuration
The network interface card (NIC) must be supported by the kernel. Determine which card is being used with one of the following commands:
dmesg | grep NIC
cat /proc/interrupts
/sbin/lsmod
From the output can be determined the ethernet card’s chipset, the i/o address and the IRQ.
Host Information
The file /etc/resolv.conf contains a list of DNS servers:
cat /etc/resolv.conf
The following command gives the name of the host:
hostname
The file /etc/sysconfig/network can define if networking must be started and can also contain the HOSTNAME variable:
cat /etc/sysconfig/network
The configuration parameters for a card, for example eth0, are contained in the following file:
cat /etc/sysconfig/network-scripts/ifcfg-eth0
LADM (12/02) 191 © Flying Pigs Training Stop and Start Networking
The interface configuration command ifconfig can be used to view or change the configuration of an IP interface on the system. This command is used to initialise IP interfaces at startup time and example commands can be found in the TCP startup script for the system, in /etc/rc2.
Once initialised the kernel module aliased to the network interface (eth0) in /etc/modules.conf is loaded and assigned an IP address and a netmask value. To list all interfaces use:
ifconfig
To reset the IP Address and netmask:
ifconfig eth0 inet 10.1.1.9 netmask 255.255.255.0
This command may also be used to bring up the network interface. The interface can be switched on and off without loosing this information as long as the kernel module is inserted:
ifconfig eth0 down
ifconfig eth0 up
Two other commands are ifup and ifdown. The ifup utility reads the system’s configuration files in /etc/sysconfig and assigns the stored values for a given interface. The script for eth0 is called ifcfg-eth0 and has to be configured. If a boot protocol such as DHCP is defined then ifup will start the interface with that protocol: ifdown eth0
ifup eth0
At boot time the ethernet card is initialised with the /etc/rc.d/init.d/network script. All the relevant networking files are sourced in the /etc/sysconfig/ directory. In addition the script also reads the sysctl options in /etc/sysctl.conf, this is where one can configure the system as a router.
LADM (12/02) 192 © Flying Pigs Training Common Network Tools
Anyone who is the least bit Internet savvy will be aware that to move data from one point say A to another point B across the Internet, it has to pass through a number of intermediary points say C, D, E etc. What many won't know is that the data is not transferred in one piece when it is sent over the net, rather, it is split into chunks of about 1500 bytes each, then each chunk is enclosed in what is known as a packet. These contain additional data such as the destination IP address and the port number plus other details which provide the unique identity to the packet.
While the packets travel the path from point A to point B, each packet may take a different path depending upon diverse factors and eventually they are merged together in the same order at the receiving end to provide the document sent in the first place.
The intermediate gateways through which the packets pass before they reach the final destination are known as hops. So for data to travel from point A to point B on the net it has to go through a number of hops.
Linux, being a network operating system, has a number of powerful tools which help the network administrator find out a wealth of data about the network and the Internet. The full list of options to any command may be found in the man pages. route This command is used to show or manipulate the IP routing table. For example to list the kernel routing table:
route -n tcpdump This is a command used to analyse network traffic by capturing network packets. The following commands illustrate some options:
Let tcpdump autodetect network interface:
tcpdump or specify a network interface to capture packets from:
tcpdump -i eth0
LADM (12/02) 193 © Flying Pigs Training netstat This gives information on current network connections, the routing table or interface statistics depending on the options used:
netstat | less arp (Address Resolution Protocol) This tool resolves IP addresses to actual MAC addresses. It may be useful to view or alter the contents of the kernel's ARP tables, for example when a suspected duplicate Internet address is the cause of some intermittent network problem. The arp tool was made for situations like this:
arp -a traceroute This tool is used to find out the potential bottlenecks in between the host computer and a remote computer across the net. For example, to run a trace on a domain:
traceroute www.google.co.uk
LADM (12/02) 194 © Flying Pigs Training Network File System (NFS)
The Network File System is one of the most widely used network services. NFS is based on a remote procedure call which allows a client to automatically mount remote file systems from a server and transparently provide access to them as if the file system was local.
Scenario In this section we are going to export a directory from our NFS server (10.1.1.9) and mount it on an NFS client (10.1.1.12).
Server and Client - Prerequisites
First configure NFS on both the server and the client. Start the NFS daemon if it is not already running:
service nfs status
service nfs start
To check whether the system supports NFS run the following:
cat /proc/filesystems | grep nfs
If there is no output it means that NFS is not supported or the NFS module has not been loaded into the kernel. To load NFS module execute:
modprobe nfs
Execute rpcinfo -p to check the correctness of the NFS installation and to confirm that the NFS server is running and accepting calls on a port 2049:
rpcinfo -p | grep nfs
When installed correctly the NFS daemon should be listening on both UDP and TCP 2049 port and portmap should be waiting for instructions on a port 111.
At this point we should have portmap listening on both the NFS server and NFS client:
rpcinfo -p | grep portmap
LADM (12/02) 195 © Flying Pigs Training Server – Edit Export File
The directories that are to be shared over the network using NFS must be defined in the file /etc/exports. Common export options include read write (rw), read only (ro) and synchronized mode (sync).
To export files from the server first create a new directory:
mkdir /home/nfs then copy some files to that new directory:
cd /home
cp -r train1/* nfs
ls nfs
Example Entries To export the /home/nfs directory for access from a client with an IP address of 10.1.1.12 with read and write permissions in synchronized mode:
/home/nfs/ 10.1.1.12 (rw,sync)
Note that server names may be used in place of IP addresses:
/home/nfs/ myserver (rw,sync)
To export the /home/nfs directory for access from a client with an IP address of 10.1.1.12 with read only permissions:
/home/nfs/ 10.1.1.12 (ro)
To export the /home/nfs directory for access from any client with read only permissions in synchronized mode:
/home/nfs/ * (ro,sync)
LADM (12/02) 196 © Flying Pigs Training Edit /etc/exports File As an example, add the following entry to the file /etc/exports:
/home/nfs/ *(ro,sync)
Export the File System Once the /etc/exports file has been edited run the command exportfs:
exportfs -rva Client – Mount Remote Files
To access the exported files on the client first create a new mount point:
mkdir /home/nfs_local
Run the mount command to mount the exported NFS remote file system from the server with an IP address of 10.1.1.9:
mount 10.1.1.9:/home/nfs /home/nfs_local
If required a filesystem type may be specified:
mount -t nfs 10.1.1.9:/home/nfs /home/nfs_local
List the files in the newly mounted directory:
cd /home/nfs_local/
ls
An attempt to create a new file fails as the file system is mounted read only.
touch test_file
LADM (12/02) 197 © Flying Pigs Training Client - Configure Automount
To make this completely transparent to end users, automount the NFS file system every time the client system boots.
To mount the file system automatically, on the client add the following line to the file /etc/fstab:
10.1.1.9:/home/nfs /home/nfs_local/ nfs defaults 0 0
Troubleshooting
Some RedHat systems by default block all incoming traffic to a NFS server via a firewall using iptables rules. For this reason when the firewall is running on the NFS server, the following error might occur: mount.nfs: mount to NFS server '10.1.1.13' failed: System Error: No route to host
To fix this problem either turn off the firewall or add iptables rules to allow traffic on portmap port 111, nfs port 2049 and random ports for other nfs services.
The easiest solution is to just turn off the firewall. This will automatically grant access to the NFS daemon to anyone. Enter the following command to stop the firewall and clean up all iptables rules:
service iptables stop
LADM (12/02) 198 © Flying Pigs Training File System Backup
Objectives
At the end of this section the delegate will be able to:
* describe reasons for taking backups
* use the backup utilities such as tar and dd
* describe how to backup and restore file systems
LADM (12/02) 199 © Flying Pigs Training Why Do We Backup?
Backing up files means making copies of them, usually on removable media, as a safeguard in case the originals get lost or damaged. Backup tapes are convenient for restoring accidentally deleted files, but they are essential in case of serious hardware failures or other disasters.
Backing up files is one of the most crucial system administration functions. To do a full backup on a file system, make sure all users are logged out. Then bring the system down to single-user mode before running the backup.
The administrator must plan and carry out a procedure for regularly scheduled backups of file systems for three major reasons.
To ensure file-system integrity against a possible system crash
To protect user files against accidental deletion
To act as an important safeguard before reinstalling or upgrading a system
Backups are the only practical way of restoring corrupted or deleted files on a Linux system. Unlike Windows, there is no recycle bin!
LADM (12/02) 200 © Flying Pigs Training What is a Backup?
A backup, or archive, is simply a copy of a file, files or entire file system stored on another medium. Backups can be made to any type of storage media including:
floppy discs
magnetic tapes
DAT tapes
other disc files
Types of Backup
There are three types of backup in common use:
FULL The entire system is backed up, typically performed after major system changes.
PARTIAL All files on part of the disk are backed up, i.e. anything less than a full backup.
INCREMENTAL Only files that have been changed since the last backup.
These types of backups are often used in combination. A typical scenario would be:
Full backup once a month
Partial backup weekly
Incremental daily
However this will depend on a variety of factors, such as:
The type of backup media available
The size of file systems to be backed up
The availability and usage level of the system
It is recommended, that, if possible, a full backup is taken every day.
LADM (12/02) 201 © Flying Pigs Training Logging Backups
It is very useful to log all backups, particularly if an incremental system is in use. This should include when and how the backup was taken and the format used. This may save confusion if a file needs to be restored.
At least two sets of media should be used and rotated, backups being logged in a book.
It is also very important to store backups in a safe, clean environment, preferably a fire-proof safe, off site.
Backup Media
Some systems will have more than one device on to which backups may be stored.
Medium Capacity
1/2-inch reel tape 140 Mbytes
2.5-Gbyte 1/4-inch cartridge, QIC 2.5 Gbytes
DDS3 4-mm cartridge tape, DAT 12-24 Gbytes
14-Gbyte 8-mm cartridge tape 14 Gbytes
DLT 7000 1/2-inch cartridge tape 35-70 Gbytes
LADM (12/02) 202 © Flying Pigs Training Backing Up Files - Review
The command tar, tape archive, was covered earlier. Here, we review the command. To create a tar archive use:
tar cvf /tmp/user1.tar .
To view the tar archive:
tar tvf /tmp/user1.tar | more
To restore a file use:
tar xvf /tmp/user1.tar ./oldfile
If the backup is to be run on a regular basis and create several new backup files, the date may be appended to the filename as follows: tar cvf /tmp/user1$(date +%m%d%H%M).tar .
In order to save space or if a file is to be mailed or transferred to another user, the command gzip can be used to compress the file.
gzip /tmp/user1.tar
The command gunzip can be used to restore the compressed archive file.
gunzip /tmp/user1.tar
Remember that the cron daemon may be used by users to schedule a task to run at a specified time.
LADM (12/02) 203 © Flying Pigs Training Backing Up Files - dd
The Disk to Disk copy program, dd, is a byte-for-byte copier which means that the target file is an exact copy of the source file. It is not a true archiving program.
This command will only copy one file at a time, however that file could be a partition device. It is commonly used to copy from one device to another, tape to tape or disk to tape.
The simple format of the command is:
dd options input filename output filename
For example:
dd if=/etc/passwd of=password would copy the file /etc/passwd to a file called password in the current working directory.
Changing the Block Size
The dd program maintains two block sizes, the input block and the output block size. These have a default size of 512 bytes. The speed of the copy can be significantly improved by altering the block sizes as follows:
ibs=NX Alters the input block size
obs=NX Alters the output block size
bs=NX Alters both blocks overriding any previous setting where N is an integer and X can be B for multiplies of 512 or K for multiplies of 1024.
LADM (12/02) 204 © Flying Pigs Training The Conversion Options
The dd program also has the ability to change the file format during transfer. This is particularly useful when moving files between different makes of machine. The following options are available:
ucase Convert file to uppercase
lcase Convert file to lowercase
ascii Convert from ebcdic to ascii format
ebcdic Convert from ascii to ebcdic format
For example, the following would convert the password file to upper case:
dd if=/etc/passwd of=password conv=ucase
LADM (12/02) 205 © Flying Pigs Training Backing Up File Systems - dump
The command dump can back up complete or partial file systems to local or remote tape drives The tape device can be on any system in the network to which the user has access. This command works quickly because it is aware of the structure of the UFS file system type, and it works directly through the raw device interface.
It can also back up incremental file system changes, only those files that were changed since a previous backup. Although it is usually used to backup to tape, backups can also be made to a disc file.
The command may take the following arguments:
* 0 full dump
* 1-9 incremental, since the last lower increment
* u add entry to /etc/dumpdates
* f output device or file
For example, the following takes a full dump of the file system on which /boot is mounted.
Here the file system is un-mounted and checked before it is backed up.
umount /boot
fsck /boot
dump 0uf /dev/st0 /boot
mount /boot
Now view the dumpdates file as follows:
cat /etc/dumpdates
LADM (12/02) 206 © Flying Pigs Training Backing Up To Tape
The tape drive is known to as /dev/st0.
However, if a tape drive is not available, the disc file created may be viewed as follows:
file /dev/st0
ls -l /dev/st0
The administrator should automate backups of file systems using the crontab utility to run a backup script that starts the dump command.
Exercise
Ensure that you have files in /home and then take a full backup to tape of the file system on which /home.
if a tape drive is not available, view the output file.
LADM (12/02) 207 © Flying Pigs Training Restoring File Systems - restore
The command restore can restore individual or complete file systems from a local or remote tape drive or disk file.
The command may take the following arguments:
* r recursive, restore all files
* i interactive restore
* v verbose
* t table of contents
* f output device or file
For example, to view the contents of a dump file:
restore tvf /dev/st0
LADM (12/02) 208 © Flying Pigs Training Interactive Restore - restore
This is the easiest way to restore individual files and directories. The following example restores the files aliases and group and the file magic in the directory apache into the directory /tmp.
cd /tmp
To initiate an interactive restore enter:
restore ivf /dev/st0
Then at the prompt, enter a ? for help:
restore > ?
List the files available:
restore > ls
Add files to the list to be extracted:
restore > add message restore > add grub
Extract the files:
restore > extract Specify next volume #: 1 set owner/mode for ‘.’? n restore > quit
Now view the restored files:
ls -l
Exercise
Use restore to restore several files as in the above example.
LADM (12/02) 209 © Flying Pigs Training Full restore - restore
In the event that the a file system is corrupted or has to be re-sized, the following example restores the complete file system on which /home was mounted:
cd /home
restore rvf
Exercise
Ensure that the directory /home has been backed up successfully.
Un-mount the directory /home.
Use the format command to re-size the file system on which /home is mounted.
Create a new file system on the re-sized disk slice.
Re-mount the directory /home.
Use restore to restore all files for that file system.
LADM (12/02) 210 © Flying Pigs Training File System Management
Objectives
At the end of this section the delegate will be able to:
mount and unmount file systems
check file systems for consistency
create and format new file systems
increase the available swap space
LADM (12/02) 211 © Flying Pigs Training File Systems
This section looks at creating, formatting and mounting file systems. Before it can be used any disk on the server must be formatted into partitons or file systems and each file system must be mounted on a directory. To view the current disk layout use:
df -h
Disks On a running Linux system, disks are represented by entries in the /dev directory. Hard disk descriptors in /dev begin with hd (IDE) or sd (SCSI); a SCSI tape would be st. Since a system can have more than one block device, an additional letter is added to the descriptor to indicate which device is considered.
A set of disks might be known as /dev/hda, /dev/hdb or /dev/sda, /dev/sdb etc.
Partitions A disk such as /dev/sda can be formatted or partitioned into file systems for example /dev/sda1, /dev/sda2 etc. Each will have a size in MBs and be made up of sectors/tracks and cylinders.
Disks normally allow only 4 primary partitions, one of which can be extended. The extended partition can be further divided into logical partitions. There can be a maximum of 64 partitions on an IDE disk and 16 on a SCSI disk.
To list available disks and partitions use:
fdisk –l
Notice the difference between primary, extended and logical partitions.
Mount Before it can be used a file system must first be mounted on a directory. A list of the mounted file systems and the directories they are mounted on can be displayed as follows:
mount
LADM (12/02) 212 © Flying Pigs Training sysfs Sysfs is a virtual filesystem exported by the kernel, similar to /proc. The files in Sysfs contain information about devices and drivers. Some files in Sysfs are even writable, for configuration and control of devices attached to the system. Sysfs is always mounted on /sys. proc The proc file system is a pseudo-file system which is used as an interface to kernel data structures. It is commonly mounted at /proc. Most of it is read-only, but some files allow kernel variables to be changed. tmpfs Reading from RAM is a lot of faster than reading from a hard drive and it reduces disk I/O. The file system tmpfs can be used to store files in memory which is ideal for file caches and other temporary data. Access is fast and the data is lost on power down or reboot. It is normally mounted at /dev/shm. Disc Usage - du
The command du shows the disk usage of each file in each subdirectory of a file system.
For example, to obtain a list of the size of each directory under root use:
du -s * inodes
Inodes store information about files on the disk. The number of inodes on the file system is specified when the file system is created. To view inode numbers use: ls -i
LADM (12/02) 213 © Flying Pigs Training Mount and Umount Commands
A mounted file system is attached to the system directory tree at the specified mount point and becomes available to the system. The root file system is always mounted. Any other file system can be connected or disconnected from the root file system.
The system tracks the mounted file systems in the /etc/mnttab, mount table, file. A list of the mounted file systems may be displayed as:
mount or saved as:
mount > dfile
less dfile
Disk Slices
The file system on which a directory is mounted is known as a disk slice. The following file contains details of all disk slices that are automatically mounted at startup:
cat /etc/fstab
LADM (12/02) 214 © Flying Pigs Training umount
The umount command does not unmount a file system that is busy. A file system is considered busy if a user is in a directory in the file system or if a program has a file open in that file system.
The following example unmounts a file system:
umount /dev/sda1 or umount /boot
To force an unmount if the file system is busy use:
umount -f /boot
To unmount all file systems use:
umount -a mount
The mount command does not mount a read/write file system that has inconsistencies. If an error message is received from the mount command, the file system will need to be checked.
This example mounts the file system:
mount /dev/sda1 /boot or mount /boot
df -h
To mount all file systems use:
mount -a
LADM (12/02) 215 © Flying Pigs Training Checking File Systems
A system program is responsible for checking the file systems during startup. To do this, it invokes another program called fsck.
The command fsck can be run on an unmounted file system as follows:
fsck /boot
The command fsck can be used to check each of the file systems for any errors.
When checking a file system, fsck may encounter errors and may ask various questions during this process. As far as this course is concerned, the answer should always be yes to these questions.
LADM (12/02) 216 © Flying Pigs Training Creating Disk Slices
The interactive menu command fdisk may be used to create new disk slices. In a real world situation this command should only be used in single user mode. With the help of your tutor, work through the following example which will create a new disk slice.
To list available disks use:
fdisk -l
Note: If using a VM, three new disc slices should be added at this point.
To invoke the command for a particular disk use:
fdisk /dev/sdb
To list the commands available enter:
m
To display the current partition table:
p
To create a partition 1 enter:
n
p (for primary)
1
Enter the start cylinder as 1 and the size as +1G.
LADM (12/02) 217 © Flying Pigs Training To display the new partition table:
p
Again list available commands:
m
Now either write the new partition table to disc:
w or quit without saving any changes:
q
LADM (12/02) 218 © Flying Pigs Training Creating a File System
The disk is now labelled but the new slice must have a file system built onto it before it can be used.
The command mkfs will create a new file system on the new slice. The file type may be specified as follows:
mkfs -t ext3 /dev/sdb1
Mounting the File System
The mount command sets up a connection from the new slice to an actual directory.
For example:
df
mkdir /extra
mount /dev/sdb1 /extra
df
The /etc/fstab File
The file /etc/fstab contains details of all disk slices which are to be mounted automatically on boot up. To ensure that the new disk slice is mounted when the server is next rebooted, an entry of the following type must be added to this file:
/dev/sdb1 /extra ext3 defaults 1 2
Test the new file as follows:
umount –a
mount -a
LADM (12/02) 219 © Flying Pigs Training Adding Swap Space
The available swap space can be increased in one of three ways:
Increase the current swap slice size by re-slicing the disk
Use an extra hard slice of the disk
Use a special file as outlined below
First show the current swap space available:
swapon -s
To add extra swap space first create a file in the appropriate place and of the desired size using the command dd as follows:
dd if=/dev/zero of=/var/swap bs=1024 count=102400
This creates a new 100Mb swap file:
ls -l /var/swap
Use the mkswap command to initialise the file as swap area:
mkswap /var/swap
Now use swapon to bring it into use
swapon /var/swap
swapon -s
To remove the file from use:
swapoff /var/swap
Add an entry of the following form in /etc/fstab if the swap area is to be brought into use at each boot:
/var/swap swap swap defaults 0 0
Now reboot to test all of the changes made.
LADM (12/02) 220 © Flying Pigs Training Logical Volume Management
Objectives
At the end of this section the delegate will be able to:
* manage and use logical volumes
LADM (12/02) 221 © Flying Pigs Training Logical Volume Manager (LVM)
With LVM, the user can create logical partitions that can span across one or more physical hard drives. First, the hard drives are divided into physical volumes, then those physical volumes are combined together to create the volume group and finally the logical volumes are created from volume group.
To create a logical volume, run through the following steps:
select the physical storage devices
create suitable disc slices for use
create the Physical Volumes (PV)
create the Volume Group (VG) from the Physical Volumes
create Logical Volumes (LV) from Volume Group
Create Device
First choose the physical volumes that will be used to create the LVM and create suitable disc slices for use:
fdisk /dev/sdc
Create two 1GB slices, sdc1 and sdc2.
fdisk /dev/sdd
Create two slices sdd1 and sdd2, using the same options.
List the new partition table:
fdisk -l Physical Volumes (PV)
Create the physical volumes using the pvcreate command as shown below.
pvcreate /dev/sdc1 /dev/sdc2 /dev/sdd1 /dev/sdd2
When the physical volumes are created, they can be viewed using the pvscan command: pvscan
The list of physical volumes with attributes like size, physical extent size, total physical extent size and the free space etc. can be viewed using pvdisplay:
pvdisplay
LADM (12/02) 222 © Flying Pigs Training Volume Groups (VG)
Volume groups are nothing but a pool of storage that consist of one or more physical volumes. Once the physical volumes are created, the user can create the volume group (VG) from these physical volumes (PV).
In this example, the command vgcreate is used to create the volume group fptvg from the two physical volumes:
vgcreate fptvg /dev/sdc1 /dev/sdd1
The commands vgs and vgdisplay can be used to list the created volume groups: vgs
vgdisplay Logical Volumes (LV)
Now the logical volumes, with the names share and mine, can be created from the volume group using the lvcreate command:
lvcreate - -name share - -size 500M fptvg
lvcreate - -name mine - -size 300M fptvg
Use the lvs, lvscan and lvdisplay commands to view the available logical volumes with their attributes:
lvs
lvscan
lvdisplay
LADM (12/02) 223 © Flying Pigs Training Make File System & Mount
Before the LV can be used its’ filesystem must be created and it must then be mounted on a directory. For example, for the LV mine:
mkfs -t ext3 /dev/fptvg/mine
mkdir /mine
mount /dev/fptvg/share /mine
And for share: mkfs -t ext3 /dev/fptvg/share
mkdir /share
mount /dev/fptvg/share /share
List them with: df -h Extend the LV
Extend the size of the logical volume after creating it by using the command lvextend:
lvextend –L +100M /dev/fptvg/mine Reduce the LV
Reduce the size of the logical volume as follows:
lvreduce –L -200M /dev/fptvg/mive
Remove the LV
To remove the logical volume, first unmount it and the use the command pvremove: umount /dev/fptvg/share
pvremove –ff /dev/sdc1 /dev/sdc2 /dev/sdd1 /dev/sdd2
LADM (12/02) 224 © Flying Pigs Training Handouts
Command Sheet – by Description
Description Command Examples backup files cpio find . -name "file*" -print | cpio -ovBc backup files tar tar cvf backup.tar . Calendar cal cal 2 2000 change directory cd cd /bin change password passwd passwd change permissions chmod chmod +x file* copy files cp cp file1 file2 count lines, characters and words wc wc -l file1 create or time stamp a file touch touch file1 file2 display date and time date date display file type file file * display files cat cat file1 display files more more file* display first few lines of a file head head file1 display terminal device file tty tty display terminal settings stty stty -a display text banner banner "Hello there" display text or variables echo echo "Hello there" display text or variables print print hello there display type of command type type ls display user information id id display who is logged on who who |wc -l edit file vi vi file1 find files find find . -name core -print find text in files grep grep 'text' file* format files (for printing) pr pr file1 | lpr initiate a conversation talk talk user1 kill processes kill kill -9 4252 list files ls ls -F mail system mail mailx make directory mkdir mkdir newdir move (rename) files mv mv file1 file2 path of working directory pwd pwd pauses output sleep sleep 600 print files lpr, lp lpr /etc/passwd print status information ps ps -aef remove (empty) directories rmdir rmdir newdir remove files rm rm -i file* remove print request lprm, cancel lprm 123 run background job at at 13:45 run scheduled background jobs crontab crontab -e save and display output tee ls -l | tee file1 set default permissions umask umask 022 set terminal write permissions mesg mesg n show help pages man man ls show print system information lpq, lpstat lpq show system variables set set -x sort files sort sort file1 -o file1 switch user su su - user9 write to other users write write user1
LADM (12/02) 225 © Flying Pigs Training Command Sheet – by Command
Command Description Examples at run background job at 13:45 banner display text banner "Hello there" cal Calendar cal 2 2000 cat display files cat file1 cd change directory cd /bin chmod change permissions chmod +x file* cp copy files cp file1 file2 cpio backup files find . -name "file*" -print | cpio -ovBc crontab run scheduled background jobs crontab -e date display date and time date echo display text or variables echo "Hello there" file display file type file * find find files find . -name core -print grep find text in files grep 'text' file* head display first few lines of a file head file1 id display user information id kill kill processes kill -9 4252 lpr, lp print files lpr /etc/passwd lprm, cancel remove print request lprm 123 lpq, lpstat show print system information lpq ls list files ls -F mail mail system mailx man show help pages man ls mesg set terminal write permissions mesg n mkdir make directory mkdir newdir more display files more file* mv move (rename) files mv file1 file2 passwd change password passwd pr format files (for printing) pr file1 | lpr print display text or variables print hello there ps print status information ps -aef pwd path of working directory pwd rm remove files rm -i file* rmdir remove (empty) directories rmdir newdir set show system variables set -x sleep pauses output sleep 600 sort sort files sort file1 -o file1 stty display terminal settings stty -a su switch user su - user9 talk initiate a conversation talk user1 tar backup files tar cvf backup.tar . tee save and display output ls -l | tee file1 touch create or time stamp a file touch file1 file2 type display type of command type ls tty display terminal device file tty umask set default permissions umask 022 vi edit file vi file1 wc count lines, characters and words wc -l file1 who display who is logged on who |wc -l write write to other users write user1
LADM (12/02) 226 © Flying Pigs Training vi Prompt Sheet
Edit a file using ‘vi’ by entering:
vi vifile vi has 3 modes:
* Command mode moving around the file
* Input mode amending the file
* Ed mode saving changes to the file
In the first, COMMAND mode, move around the file.
Move around the file with: 'w', 'b', 'return' and '-' or with the arrow keys
Move screens of data with: 'Control & d' 'Control & u' 'Control & f' 'Control & b'
The second mode is INPUT mode, entered by typing a command:
i to insert a to append R to replace text o to insert a line
Press ESC to return to COMMAND mode.
Other commands include: u to undo the last change r to replace 1 character x to delete 1 character dd to delete 1 line
The third mode is ED mode, the mode used to save the file and quit the edit. Enter this mode by typing a colon.
Save the file with :w Save and quit with :wq or: :x
Quit without saving with :q!
LADM (12/02) 227 © Flying Pigs Training LADM (12/02) 228 © Flying Pigs Training Handouts
Command Sheet – by Description
Description Command Examples backup files cpio find . -name "file*" -print | cpio -ovbc backup files tar tar cvf backup.tar . calendar cal cal 2 2000 change directory cd cd /bin change password passwd passwd change permissions chmod chmod +x file* copy files cp cp file1 file2 count lines, characters and words wc wc -l file1 create or time stamp a file touch touch file1 file2 display date and time date date display file type file file * display files cat cat file1 display files more more file* display first few lines of a file head head file1 display terminal device file tty tty display terminal settings stty stty -a display text banner banner "hello there" display text or variables echo echo "hello there" display text or variables print print hello there display type of command type type ls display user information id id display who is logged on who who |wc -l edit file vi vi file1 find files find find . -name core -print find text in files grep grep 'text' file* format files (for printing) pr pr file1 | lpr initiate a conversation talk talk user1 kill processes kill kill -9 4252 list files ls ls -f mail system mail mailx make directory mkdir mkdir newdir move (rename) files mv mv file1 file2 number file nl nl file1 path of working directory pwd pwd pauses output sleep sleep 600 print files lpr, lp lpr /etc/passwd print status information ps ps -aef remove (empty) directories rmdir rmdir newdir remove files rm rm -i file* remove print request lprm, cancel lprm 123 run background job at at 13:45 run scheduled background jobs crontab crontab -e save and display output tee ls -l | tee file1 set default permissions umask umask 022 set terminal write permissions mesg mesg n show help pages man man ls show print system information lpq, lpstat lpq show system variables set set -x sort files sort sort file1 -o file1 switch user su su - user9 write to other users write write user1
LADM (12/02) 229 © Flying Pigs Training Command Sheet – by Command
Command Description Examples at run background job at 13:45 banner display text banner "Hello there" cal Calendar cal 2 2000 cat display files cat file1 cd change directory cd /bin chmod change permissions chmod +x file* cp copy files cp file1 file2 cpio backup files find . -name "file*" -print | cpio -ovBc crontab run scheduled background jobs crontab -e date display date and time date echo display text or variables echo "Hello there" file display file type file * find find files find . -name core -print grep find text in files grep 'text' file* head display first few lines of a file head file1 id display user information id kill kill processes kill -9 4252 lpr, lp print files lpr /etc/passwd lprm, cancel remove print request lprm 123 lpq, lpstat show print system information lpq ls list files ls -F mail mail system mailx man show help pages man ls mesg set terminal write permissions mesg n mkdir make directory mkdir newdir more display files more file* mv move (rename) files mv file1 file2 nl number file nl file1 passwd change password passwd pr format files (for printing) pr file1 | lpr print display text or variables print hello there ps print status information ps -aef pwd path of working directory pwd rm remove files rm -i file* rmdir remove (empty) directories rmdir newdir set show system variables set -x sleep pauses output sleep 600 sort sort files sort file1 -o file1 stty display terminal settings stty -a su switch user su - user9 talk initiate a conversation talk user1 tar backup files tar cvf backup.tar . tee save and display output ls -l | tee file1 touch create or time stamp a file touch file1 file2 type display type of command type ls tty display terminal device file tty umask set default permissions umask 022 vi edit file vi file1 wc count lines, characters and words wc -l file1 who display who is logged on who |wc -l write write to other users write user1
LADM (12/02) 230 © Flying Pigs Training vi Prompt Sheet
Edit a file using ‘vi’ by entering:
vi vifile vi has 3 modes:
* Command mode moving around the file
* Input mode amending the file
* Ed mode saving changes to the file
In the first, COMMAND mode, move around the file.
Move around the file with: 'w', 'b', 'return' and '-' or with the arrow keys
Move screens of data with: 'Control & d' 'Control & u' 'Control & f' 'Control & b'
The second mode is INPUT mode, entered by typing a command:
i to insert a to append R to replace text o to insert a line
Press ESC to return to COMMAND mode.
Other commands include: u to undo the last change r to replace 1 character x to delete 1 character dd to delete 1 line
The third mode is ED mode, the mode used to save the file and quit the edit. Enter this mode by typing a colon.
Save the file with :w Save and quit with :wq or: :x
Quit without saving with :q!
LADM (12/02) 231 © Flying Pigs Training