Remote Access and SSH
1 Remote Access
● Modern computing requires the use of a variety of resources located on different machines in widely separated locations.
● You are familiar with the world wide web, which allows us to obtain information from other machines.
● However, mathematicians frequently need to create and delete files on other machines, or even to use their processing power.
2 SSH
● Secure Shell (SSH) is a cryptographic network protocol to run a command line on a remote machine. ● Client-server architecture: the "ssh client" runs on your local machine, while the remote machine must run an "ssh daemon" - a server.
● There are two ssh protocols: ssh1, which uses 56-bit encryption (this means that the public keys can be expressed in 56 binary digits) and is susceptible to some attacks, and ssh2, which uses 128-bit encryption and is much safer.
3 Command-line Interface
● A command-line user interface (CLI), also known as a console user interface is a means of interacting with a computer program where the user (or client) issues commands to the program in the form of successive lines of text (command lines). A program which handles the interface is called a command language interpreter or shell.
● Implemented by cmd.exe in Windows, Terminal in Linux/MacOS.
● SSH provides access to CLI on a remote machine.
● Graphical User Interfaces (GUI - pronounced "gooey") are the familiar ways that we interact with computers. Before Xerox invented the GUI in the 1970s, command lines provided the only interfaces to computers. Since the mid-eighties, GUIs have become the only interfaces to computers that most people know. 4 SSH Clients
● There is a bunch of them. Here are the most popular. ● PuTTY (Windows). Free and open-source. Has a handy GUI. ● OpenSSH (Linux). ● MacOS has a built-in ssh client. ● On Linux/MacOS typically the command looks like ssh username@remote.host.name[:port].
5 Processes
● Processes are the tasks that a computer handles. Every time you run any command, it appears on the computer as a process, with attributes and identifiers. If a process runs amok, you need to do something about it. Thus, commands for managing processes are integral to any good operating system.
● In most modern operating systems there may be lots of processes running at the same time. This is referred as multitasking.
● Process has its owner associated with it (a user which started this process). The process has the same permissions as its owner does.
● Each process has it’s own dedicated piece of memory. You may think of it as a sandbox where the code of your particular program/task is executed.
6 Managing Processes
● Commands for managing processes are integral to any good operating system.
● In Windows it’s Task Manager (taskmgr.exe). Could be reached by pressing Ctrl+Esc or Ctrl+Alt+Del and mostly used to kill application that is hanging.
● Starting a process from the command line is the easiest thing to do on any computer. Just type the name of the process you want to run.
● For example, on a Unix computer the command /usr/bin/firefox should run the Firefox Browser.
● On windows, the corresponding command would be "c:\program files\firefox\firefox.exe". ● In addition you can specify command options/arguments/parameters which define the way command should behave. Each command has it’s own predefined list of parameters which make sense specifically for this command. To see the exhaustive list all options for standard commands use man
● By default, every process that you start runs in the foreground. It gets its input from the keyboard and sends its output to the screen.
● If the ls command wants any input (which it does not), it waits for it from the keyboard. While a program is running in the foreground and is time-consuming, no other commands can be run (start any other processes) because the prompt would not be available until the program finishes processing and comes out.
● A background process runs without being connected to your terminal. If the background process requires any keyboard input, it waits but you couldn’t see this.
● The advantage of running a process in the background is that you can run other commands; you do not have to wait until it completes to start another.
● The simplest way to start a background process is to add an ampersand (&) at the end of the command.
● So typing e.g. /usr/bin/firefox will start Firefox Browser in foreground, whereas /usr/bin/firefox & will run Firefox Browser in background. 8 Managing Processes: ps command.
● In Linux the basic command for monitoring processes is called ps (process status). Typing ps by itself lists the processes that you own from your current session on the computer.
● Here is the most important information you can get from ps command output.
● User – owner of the process
● PID – unique ID of the process
● TTY – terminal associated with the process
● STIME – process start time
● TIME – CPU time taken by the process
● CMD – The command that started this process
9 Managing Processes: ps command.
● ps -f gives you some more details
● ps -x – shows information about processes without terminals
● ps -a shows processes for all users
● ps -u shows the process's user/owner ● You can get more by running man ps
10 Interrupting Process
● The easiest way to kill a process that has taken over your command line is to press
● To kill process which is running in background use kill command. You will need PID for that which you can find with ps command.
● kill -9 1245 will kill process with PID 1245.
11 Filtering Output of ps command
● Use ps [your options] | grep
● Use wildcards for specifying
● There are a few "wildcard" characters that can match many things. The question mark matches any single character. Thus, ls image?.jpg would list all of the files image1.jpg, image9.jpg, and imageA.jpg on a directory, but would not list image23.jpg The asterisk matches any number of characters, so that ls image*.jpg would list image1.jpg, image23.jpg and image49027.jpg.
12 More on Processes
● Parent and Child Processes: each unix process has two ID numbers assigned to it: The Process ID (pid) and the Parent process ID (ppid). Each user process in the system has a parent process. Most of the commands that you run have the shell as their parent. Check the ps -f example where this command listed both the process ID and the parent process ID.
● Zombie and Orphan Processes: normally, when a child process is killed, the parent process is updated via a SIGCHLD signal. Then the parent can do some other task or restart a new child as needed. However, sometimes the parent process is killed before its child is killed. In this case, the "parent of all processes," the init process, becomes the new PPID (parent process ID). In some cases, these processes are called orphan processes. ● When a process is killed, a ps listing may still show the process with a Z state. This is a zombie or defunct process. The process is dead and not being used. These processes are different from the orphan processes. They have completed execution but still find an entry in the process table. ● Daemon Processes: daemons are system-related background processes that often run with the permissions of root and services requests from other processes. A daemon has no controlling terminal. If you do a "ps -ef" and look at the tty field, all daemons will have a ? for the tty.
● To be precise, a daemon is a process that runs in the background, usually waiting for something to happen that it is capable of working with. For example, a printer daemon waiting for print commands or SSH daemon you are actually interacting with to connect to this machine through SSH.
● If you have a program that calls for lengthy processing, then it’s worth to make it a daemon and run it in the background.
13 Navigation and File Management
● Another fundamental skills you need to master are moving around the filesystem and getting an idea of what is around you. We will discuss the tools that allow you to do this. ● You also probably will need to change what’s around you. Here is where file management commands come in handy.
14 Finding Where You Are with the pwd Command
● When you log into your server, you are typically dropped into your user account's home directory. A home directory is a directory set aside for your user to store files and create directories. It is the location in the filesystem where you have full dominion. ● To find out where your home directory is in relationship to the rest of the filesystem, you can use the pwd command. This command displays the directory that we are currently in. ● You should get back some information that looks like this: /home/user1 ● The home directory is named after the user account, so the above example is what the value would be if you were logged into the server with an account called user1. This directory is within a directory called /home, which is itself within the top-level directory, which is called "root" but represented by a single slash "/".
15 Looking at the Contents of Directories with ls command
● ls will give you the list of all files and folders (except hidden) in your current directory sorted alphabetically.
● ls -l can give you more details on each file/folder such as permissions, size, last modification date, owner, etc.
● ls -a will list all files and folders including the hidden ones.
● ls -t will sort by modification time. ls -S will sort by file size.
● Ls -R will list subfolders recursively.
● Use ls [options]
● If you use ls -a you can see ‘.’ and ‘..’ folders. ‘.’ stands for current directory and ‘..’ stands for parent directory.
16 Moving Around the Filesystem with cd
● Use cd [path] to change your working directory. ● You can use either absolute path, e.g. cd /home/user1/dir1/dir2, or relative path cd dir1/dir2. The first option will work regardless your current working directory. The second one will succeed only if you are currently in your home folder, i.e. /home/user1 ● cd .. brings you to parent directory. Cd ../../ brings to directories up. ● cd – brings you to previous location. ● cd (no options) or cd ~ brings to home directory (~ is an alias/shortcut for your home directory)
17 Standard Directories
18 Create, View, Edit File
● touch
● less
● nano [filename] will start a basic and easy-to- use text editor called GNU nano. [filename] is optional here – you can first start the editor with empty file and then open the some using nano functionality.
19 Copy, Delete, Move Files and Folders
● mkdir
● cp
● rm
● rmdir
● rm -r
● mv
● For cp, rm, rmdir, mv you can use some filter expression for source name to fetch multiple files/folder at a time, e.g. cp /home/user1/dir1/*.txt /home/ser1/dir2 will copy all files with .txt extension from dir1 to dir2.
20 File Permissions
● Part of the security provided by every operating system involves the rights to read, copy, change, delete, and execute the files stored on a machine's file system. The way those rights are handled and modified varies from OS to OS.
● The basic rights to all files are similar across both Unix and Windows operating systems. The permissions on those files fall into three basic categories: ● Read: This includes permission to view the contents of a file or directory, and to make a copy of it.
● Write: This includes permission to change the contents of a file, to make a new file on a directory, and even to delete the file or directory. ● Execute: This is the permission to run the program contained in the file. In case of directory execute permission means that user is allowed to enter the directory, and access files and directories inside. ● Each operating system provides other rights. For example, in W2k the ability to change file rights is separate from the ability to change the contents of a file, while in Unix the ability to change permissions on the file is included in the "write" permission.
21 File Permissions
● The command to change file rights in Unix is called chmod. In order to use it, we must understand the listing of rights on files in Unix. Each file has a 10-character string associated with it that describes its nature and rights completely. We can view this string using the ls -l command: ls -l myfile.txt ● The result of this command is a line that begins with e.g. -rw-rw-r--. ● The first symbol tells if this is a file or a folder. The dash means it is a plain file. If it were a directory, the first character would be a "d".
● The next nine characters are broken into three sets of three. The first three characters correspond to the rights of the owner of the file. The next three give the rights for members of the group that owns the file. The third set of three gives the rights of everyone who has an account on the machine. Thus, if we run the command ls -l and look at "WebSite" folder we see: drwxr-xr-x 3 owner grp 1024 Dec 13 14:16 WebSite ● The first ten characters say that WebSite is a directory, the owner has read (r), write (w), and execute (x) permission on the file, while members of the group (called "grp") are treated the same as anyone else: they can read or execute the file, but not write to it. This means that anyone can view, copy or run the files in the directory, but cannot change it or delete its contents. ● Here is another example: the command ls -l book*.pdf yields -rw-rw-r-- 1 sally grp 2384061 Jan 4 10:45 book1.pdf. In this case, book1.pdf is a plain file whose owner ("sally") and group ("grp") has read and write permission, while everyone else has only read permission.
22 Setting File Permissions
● We can change these permissions using the chmod command. The command has the form chmod [ugo][+-][rwx]
● +- options: “+” adds the specified modes to the specified classes, “-” removes the specified modes from the specified classes ● rwx options actually specifies modes: r – read, w – write, x – execute (you can use multiple options at a time, e.g wx means that your command will grant/revoke both write and execute permissions).
● Example: chmod o+w file1 will add write permission for all users on current machine, chmod g-wx file1 will revoke write and execute permissions form owner’s group.
23 File Types
● Question: "since all files on computers are composed effectively of ones and zeros, how can they be different?" ● The answer lies in the programs that are meant to interpret the files to the user. Yes - all files are the same, but they are interpreted differently to you by the computer. ● There are many ways to distinguish among files. The simplest and most basic concerns whether they are considered to be text or binary. Text files (in English) are usually coded in ASCII (the American Standard Code for Information Interchange) - a seven bit scheme for representing letters as binary numbers.
● All non-ASCII files on a computer (except for a few proprietary machines such as IBM mainframes) are considered to be "binary" files. Such files may be programs - sequences of instructions that the computer can run directly - or they may require other programs to interpret them. For example, a MS Word document cannot be viewed well using a simple text editor because it contains non-ASCII character codes.
24 More Commands
● man
● find -iname
●
● exit terminates current session
● echo [sometext] display some text
● clear cleans screen
25 Shells
● A shell is a user interface for access to an operating system's services. Think of a shell is another name for command-line interface. It is named a shell because it is a layer around the operating system kernel. The aspect we want to emphasize here is that each shell defines a set of commands and syntax (programming language) it exposes to user.
● There are only two standard version of a shell on Windows: cmd and Powershell. Unix has many: sh (the Bourne Shell), bash (the Bourne Again Shell), csh (the C Shell), tcsh (the Trendy C Shell, or something like that...), and ksh (the Korn Shell) are but a few. We will use bash. bash is widely considered to be technically superior to most other shells, but for what most users do, it doesn't matter.
● When you start an ssh session a specific shell is initialized (assigned to your session), so every command you run is interpreted by the rules defined for this specific shell and not any other. To see what shell are you using type echo $0.
26 Environment Variables
● There are only a few places where you must interact directly with your shell in a way that depends on which shell you use. Environment variables are one such area. Every shell has certain variables that are used by the shell in its operations.
● The most critical is the PATH variable. This gives a list of the directories in which the shell will look for commands that are not built-in.
● To see current variable value use echo $
● You can also try echo $HOST, echo $TERM, echo $USER. The names are self-explanatory.
27 Scripts
● Script is an executable file that simply contains set of some commands. We can say that script is a command in it’s own rights (like ls, kill, touch) defined by user to do some custom tasks the one needs. ● Script is a key feature to automate tasks one needs to run repeatedly.
28 Example
● For example, consider a situation for the instructor of a course. He records grades for the course in a spreadsheet, but then saves the file in both the native format for the spreadsheet program, and also as a comma separated value file. He then runs a program that converts the comma separated file into an HTML table, and then uses SCP to send the file to a web server where his students can view it. The instructor might try to automate this process to some extent using the following script:
libreoffice --calc /home/me/M300/grades.xls read cd "/home/me/M300" txt2html grades.txt scp [email protected] grades.txt
● The first command runs LibreOffice Calc (Excel analogous) on the proper file for the instructor. The second holds the program until the instructor indicates that he is ready to continue by hitting any key (this gives him time to enter the grades and save the files). The third switches to the directory where the files are. The next command runs the program that does the conversion (which might be a script itself), and the last runs ftp for him. Finally, the last line uploads file to www.myserver.wsu.edu. Thus, to carry out the entire process the instructor only needs to type the name of the script (or double-click it in the file manager), and hit any key after the grades are saved.
● This is the simplest kind of script one could write. Ultimately, all it does is save a few keystrokes for the instructor. Most scripts are much more powerful. They store and retrieve values, and make decisions based on conditions on the computer.
29 Scripts
● In Linux a script may have any name you want to give it. The only critical property of the file is that it must be executable. You can make the file called myscript executable by typing chmod +x myscript. Many people like to make their bash- shell scripts end with .sh, e.g. myscript.sh, but this is not mandatory. ● The first line of a bash-shell script should be #!/bin/bash. This tells the operating system that the commands in the script should be interpreted using the bash shell. ● To run script simply type path to the file and its name of the file, e.g. /home/user1/scripts/myscript.sh. Note that to run script which is in the current folder type ./myscript (add ./ in the beginning).
● So to create and run script called myscript you need: nano myscrpt (type #!/bin/bash and some commands you want to run and save file and close nano) chmod +x myscript ./myscript
30 Using Variables
● After we get the script started, we typically assign some variables containing directory names or names of the files we will work with.
● Recall an example with grades. we might be working with the grades from Math 300, so we look for all of our files in the M300 directory. That way if we are also teaching Math 171, we can use the same script for both courses by entering different variable values. ● Let’s say the name the script is upload_grades.sh. We modify it in the following way: libreoffice --calc /home/me/$COURSE/grades.xls read cd "/home/me/$COURSE" txt2html grades.txt scp [email protected] grades.txt
● To run it for M300 we do: COURSE=M300 export COURSE ./upload_grades.sh ● To run it for M171 we do: COURSE=M171 export COURSE ./upload_grades.sh ● The first line sets value for COURSE variable (note: there is no $ sign in this case). The second line makes variable visible to the script (using export
● The real power of scripts lies in their ability to make decisions based on the existence of files and directories, the values of variables, and other factors relating to the machine they run on. In particular, it is important to be able to choose which of several actions to take in response to input or environment. Thus, every scripting language includes some sort of "if" construct. ● Example: if [ $VAR == "yes" ]; then echo "The answer was yes." else echo "The answer was no." fi ● The above example prints “The answer was yes.”, if VAR has values “yes”; otherwise, the output “The answer is no.” ● Spaces are important! To make it more accurate, the red underscore here denotes a whitespace: if_[_$VAR_==_"yes"_]; ● You can specify as many commands as you want to be executed between then and else, and between else and fi.
32 “for” loop
● The Bourne shell provides a number of other powerful constructs that have no parallel in Windows cmd. For example, there is a "for" loop command in the Bourne shell. It looks like this (a red underscore here denotes a whitespace):
for_VAR_in_"Linux"_"Redhat"_"Unix";_do echo_We_like_$VAR done
● In this case, the first line defines a variable called VAR that takes on the values from a list, which here comprises the strings "Linux", "Redhat", and "Unix". This block of code executes three times. The first time, the variable VAR contains the string "Linux", the second time it contains "Redhat", and, obviously, the third time it contains "Unix". Thus, the output from the command is
We like Linux We like Redhat We like Unix
33 Redirecting Script Output
● By default output of any script executed from some terminal session is displayed in the terminal windows of this session. This is true for both foreground and background execution.
● Recall that if you want to run some time consuming script and meanwhile work on something else using the same session, you may start process in background.
● To prevent a mess created by mixing output of background process and your current commands you can redirect script output to some file.
● Use somescript > output_file [2> error_file]. This will send script output to output_file. You can optionally specify separate error_file to send all error messages to.
34 Moving Files Across Network
● So far we dealt only with files on a single machine we are connected to.
● There is a bunch of ways to receive/send files from/to other machines on the network.
35 Moving Files Across Network: SFTP
● Secure FTP (SFTP) performs file transfers through an encrypted tunnel, so that the information in them cannot be understood by anyone intercepting it. ● To use it, you again need client software on your local machine, and an SSH server that supports SFTP on the remote machine. While there are many client packages, in the simplest setting the interface is a command line.
● A typical session follows (with comments on the tasks performed in blue): open the ftp session > open ftphost.com list the contents of the remote directory > ls change the remote directory > cd newdirectory change the local directory > lcd mydirectory put a file on the remote machine > put myfile remotefile get a file from the remote machine > get remotefile close the session >exit
36 Moving Files Across Network: SCP
● There is a second way to transfer files using SSH, called SCP (Secure CoPy). We used in example scrip to upload grades to remote machine. ● There is really little difference in the background details between this and sftp, but the appearance to the user is significantly different. In particular, the scp command works very like the cp (copy) command of Unix. ● For example, to copy a file called remote.txt that resides on a computer named mycomputer.math.wsu.edu to the current directory, one could type a command of the form scp [email protected]:remote.txt local.txt. ● To copy a file called local.txt that resides in current directory to a remote computer named mycomputer.math.wsu.edu of the form scp local.txt [email protected]:remote.txt.
37 GUI for SFTP
● For Windows there exist a nice graphical client on called WinSCP. It supports both SFTP and SCP. Here is a quick manual how to connect to the server http://www.math.wsu.edu/kcooper/M300/remote_access/ using WinSCP.
● On Linux distributions providing GUI a client is usually built into standard file explorer. Look for something like “Connect to a Server” and type address in the following form sftp://[email protected]. You will see contents of remote folder in the same way you see local folders on your machine.
38 In-Class Exercise (ICE)
● The server we are all using: adams.math.wsu.edu ● The port for SSH is default (22) so can be omitted. ● You can login into Windows using “windows” account and start PuTTY to connect to the server via SSH.
● You can login into Linux using your personal account, start Terminal and run ssh [email protected] to connect to the server via SSH. ● For the last exercise we will need SFTP. The remote machine is the same, the port is also the same (since SFTP runs as a subsystem of SSH). Your options for the client are WinSCP for Windows or File Explorer for Linux.
● The distribution of Linux we have here is CentOS. See https://en.wikipedia.org/wiki/CentOS for details.
39 ICE #1
1) Start ssh session 2) Run command to display your current working directory. What directory is that? 3) Go two levels up in directories hierarchy. List contents of the current directory in the way that you can see permissions for each folder. What special directory are you currently in? 4) Go to your home directory with one command. 5) Create the following folders structure inside your home directory: ~/dir1 ~/dir1/dir2 ~/dir1/dir3 ~/dir1/dir3/dir4 6) Run command to show nested folders structure to make sure you did everything correctly. 7) Go to ~/dir1/dir3 and create three empty files there: balck_dog.txt, white_dog.txt, red_cat.txt 8) What user permissions do these newly created files have? Is any user on this machine can modify it? Since we all are working on the same machine you can ask your neighbor to go to your home directory and try to modify them. If it didn’t work, make it possible. Ask your neighbor to try again. 9) Open editor for red_cat.txt and add text saying something, e.g. “It’s pretty strange!”. Close editor 10) Display the contents of red_cat.txt 11) Copy all the files containing “dog” in it’s name to ~/dir1/dir2 12) Move red_cat.txt to ~/dir1. 13) Remove ~/dir1/dir3. 14) Go to your home directory to show and run command to show nested folders structure. 15) Remove dir1. 40 ICE #2
1) Create a directory ice2 and go to that directory. 2) Create an empty file named mycommand. 3) mycommand is going to be a script and it’s supposed to do the following: if the value of environment variable called NEED_A_FILE_HERE equals “yes”, then it creates a file in current directory with name file_for_{username} and notifies user about that (where {username} is name of the current user); otherwise, it deletes file_for_{username} if any exits in current folder and notifies user about that. So, you need to edit mycommand and add corresponding commands. 4) Try to run mycommand (type ./mycommand). Why do you get an error message? Fix it and try again. 5) Set NEED_A_FILE_HERE to “yes” and make it visible to your script. Try to run script the again. Check if a new file was created. 6) Go to your home folder and run your script again. Check if a new file was created. 7) Now set NEED_A_FILE_HERE to “no” and try to run it again. Check if file was deleted. 8) How can you make your command accessible from any location by just typing mycommand without specifying a path to the script (like you did it in home folder). Make it possible.
41 ICE #3
1) Create a directory ice3 and go to that directory. 2) Create a script with name dosomething.sh with following commands: echo Doing something... for VAR in {0..1200}; do echo $((VAR*3)) secnonds elasped. sleep 3 done This script imitates some long-running task which reports some progress periodically. It will take one hour for this script to be completed. 3) Run the script as a foreground process. How can you interrupt it? Wait for some time and do it. 4) Run the script as a background process. Where does output of your script go? How can you interrupt background process? Wait for some time and do it. 5) Now run the script as a background process and redirect its output to a file called some_output.txt. Wait for 15 seconds and check what’s in that file. Wait for 15 more seconds and do it again. Has anything changed? Kill the background process you’ve created.
42 ICE #4
1) Your SSH should be still open and you are connected to a remote machine. 2) Create a directory ice4 and go to that directory. Create an empty file called remote.txt. Close SSH session. 3) Start SFTP client. 4) Copy remote.txt to your local machine with name local.txt. 5) Using any graphical editor (e.g. Notepad on Windows) add the following text to local.txt: “This file was modified on local machine.”. 6) Copy local.txt to remote machine with name remote.txt (so you will overwrite the existing one). 7) Close SFTP client. 8) Open SSH session and and see what’s inside remote.txt. 9) Close SSH session.
43 Commands Cheat Sheet
44