Principles of System Administration Jan Schaumann [email protected] April 14, 2021 Contents Preface to the preface i Preface ii I Introduction and General Concepts 1 1 Introduction 2 1.1 What exactly does a System Administrator do? . .3 1.2 The Profession of System Administration . .6 1.3 System Administration Education . .9 1.3.1 Challenges in System Administration Education . 11 1.4 The Three Pillars of Exceptional System Design . 13 1.4.1 Scalability . 14 1.4.2 Security . 16 1.4.3 Simplicity . 19 1.5 The Future of System Administration . 20 Problems and Exercises 23 Problems . 23 Exercises . 24 2 Unix 28 2.1 Unix History . 28 2.1.1 The Operating System . 28 2.1.2 Networking . 35 2.1.3 Open Source . 37 2.2 Basic Unix Concepts and Features . 38 1 CONTENTS 2 2.2.1 The shell . 39 2.2.2 Manual pages and documentation . 42 2.2.3 A portable, multitasking, multiuser system . 44 2.2.4 The Unix Philosophy . 46 Problems and Exercises 50 Exercises . 51 3 Documentation Techniques 55 3.1 System Documentation Writing 101 . 56 3.1.1 Know Your Audience . 56 3.2 Online Documentation . 58 3.3 Different Document Types . 59 3.3.1 Processes and Procedures . 60 3.3.2 Policies . 61 3.3.3 Online Help and Reference . 62 3.3.4 Infrastructure Architecture and Design . 63 3.3.5 Program Specification and Software Documentation . 64 3.4 Collaboration . 65 3.5 Formats . 66 Problems and Exercises 68 II Fundamental Technologies and Concepts 70 4 Of File Systems and Storage Models 74 4.1 Introduction . 74 4.2 Storage Models . 76 4.2.1 Direct Attached Storage . 76 4.2.2 Network Attached Storage . 78 4.2.3 Storage Area Networks . 80 4.2.4 Cloud Storage . 83 4.2.5 Storage Model Considerations . 86 4.3 Disk Devices and Interfaces . 87 4.3.1 Physical Disk Structure . 90 4.4 Dividing and Combining Disks . 92 4.4.1 Partitions . 93 CONTENTS 3 4.4.2 Logical Volumes . 97 4.4.3 RAID . 99 4.5 File Systems . 106 4.5.1 File System Types . 106 4.6 File System Layout . 110 4.7 The Unix File System . 112 4.8 Conclusions . 118 Problems and Exercises 122 Problems . 122 5 Software Installation and Package Management 126 5.1 Introduction . 126 5.2 Types of Software . 128 5.2.1 Up and down the software stack . 128 5.2.2 Operating System vs. System Software vs. Third Party Software . 135 5.3 File System Layout . 137 5.4 OS Installation . 141 5.4.1 Identifying server requirements . 142 5.4.2 OS Installation Overview . 143 5.4.3 OS Installation Details . 147 5.5 Package Management Systems . 151 5.5.1 \Manual" Software Installation . 151 5.5.2 Software Installation by Package Manager . 153 5.5.3 Inherent Security Risks . 158 5.6 Managing Software Updates and Patches . 160 5.7 Conclusions . 163 Problems and Exercises 166 Problems . 166 6 Of Users and Groups 170 6.1 Introduction . 170 6.2 Types of Users . 171 6.3 Groups of Users . 175 6.4 User Authentication . 177 6.4.1 Authentication Examples . 179 CONTENTS 4 6.4.2 The Problem with Passwords . 181 6.4.3 Sharing root ....................... 183 6.5 Summary . 184 Problems and Exercises 188 Problems . 188 7 Configuration Management 191 7.1 Introduction . 191 7.2 Services, not Single Points of Failure . 194 7.3 Defining Services and Requirements . 195 7.3.1 Example Service: Syslog . 196 7.3.2 Example Requirements: LDAP Client Configuration . 197 7.3.3 CM Requirements . 199 7.4 Of States and Sets . 204 7.4.1 States . 204 7.4.2 Sets . 207 7.5 Fighting entropy . 210 7.5.1 Deployment roles . 210 7.5.2 Idempotence and Convergence . 214 7.5.3 Quis custodiet ipsos custodes? . 217 7.6 Even more formal process definitions . 217 7.7 Summary . 218 Problems and Exercises 222 Problems . 222 8 Automation 226 8.1 Introduction . 226 8.2 Of Laziness And Other Virtues . 227 8.3 Benefits of Automation . 229 8.3.1 Repeatability . 229 8.3.2 Reliability . 230 8.3.3 Flexibility . 232 8.4 Who benefits from automation? . 233 8.4.1 Ourselves . 234 8.4.2 Our Peers . 234 8.4.3 All Users . 235 CONTENTS 5 8.5 Levels of Automation . 236 8.6 Automation Pitfalls . 238 8.6.1 Increased Complexity and Impact . 238 8.6.2 Loss of Audit Trail . 240 8.6.3 Loss of Accountability . 240 8.6.4 Safeguards . 241 8.7 Summary . 243 Problems and Exercises 246 Problems . 246 9 Building Scalable Tools 249 9.1 Introduction . 249 9.2 How Software evolves . 251 9.2.1 Scripts . 252 9.2.2 Programs . 253 9.2.3 Software Products . 254 9.3 Principles of Developing Robust System Tools . 256 9.3.1 Unix Philosophy and User Interface . 257 9.3.2 Principle of Least Astonishment . 263 9.3.3 Explicit and predictable failure . 264 9.3.4 There's no such thing as a temporary solution. 265 9.3.5 Readability counts . 266 9.3.6 Of Buses and Windows . 268 9.3.7 Code Reviews . 270 9.4 Additional Guidelines . 270 9.5 Summary . 272 Problems and Exercises 274 Problems . 274 10 Networking 278 11 Security 279 11.1 Authentication and Authorization . 279 CONTENTS 6 III Managing Complex Services 280 12 Managing System Services 282 13 Backup, Failover, and Disaster Recovery 283 13.1 File System Backups and Snapshots . 283 14 Monitoring and Visibility 284 IV Meta Matter 285 15 Everything We Haven't Covered 287 15.1 Containers . 287 15.2 Service Orchestration . 287 16 Ethics and Legal Issues 288 17 Future Direction of System Administration 289 Glossary of Terms 290 A Image Attributions 291 Index 294 Listings 1 Sample command-line invocations . vi 2.1 Simple I/O redirection in the shell . 39 2.2 Simple job control in the shell . 42 2.3 2-clause BSD license . 49 4.1 fdisk(8) sample invocation and output on a Linux system . 94 4.2 disklabel(8) invocation and output on a NetBSD system . ..