Software Best Practices Marco Mambelli – [email protected] Engineering Week 17 February 2020 Software • Set of instructions and its associated documentations that tells a computer what to do or how to perform a task • Any manuscript/artifact/product written by you with the scope to be used by machine and humans 2 2/17/20 Marco Mambelli | Software best practices 3 2/17/20 Marco Mambelli | Software best practices Outline • General applicability, more in detail – Version control and Git – Documentation • More specific to coding – Requirements – Design • Technology selection • OS Requirements • Software inputs • Software logs, metrics and accounting – Code development – Validation and testing – Releases – Deployment – Bug tracking – Change management – Critical services operation 4 2/17/20 Marco Mambelli | Software best practices “Piled Higher and Deeper” by Jorge Cham, http://www.phdcomics.com 5 2/17/20 Marco Mambelli | Software best practices Version Control System • Preserves different version of a document • Helps merging different contributions • Answers important questions on the documents – What changed? – Who changed it? – Why? 6 2/17/20 Marco Mambelli | Software best practices Centralized vs distributed VCS 7 2/17/20 Marco Mambelli | Software best practices Common RCS • SVN (Apache Subversion) – Newer system based on CVS – Includes atomic operations – Cheaper branch operations, slower comparative speed – Does not use peer-to-peer model – Still contains bugs relating to renaming files and directories – Insufficient repository management commands • Mercurial – Easier to learn than Git (also more similar to CVS/SVN) – Distributed model – No merging of two parents – Less out of the box power, in Python, extension-based rather than scriptability • Git – Very different from CVS/SVN – Cheap branch operations, dramatic increase in operation speed – Full history tree available offline – Distributed, peer-to-peer model – Not optimal for single developers 8 2/17/20 Marco Mambelli | Software best practices Git concepts – Local repository • Snapshot with GUID (SHA1 hash) • Repository – init • Staging – add • Commit – commit (checkout) • Tag – tag 9 2/17/20 Marco Mambelli | Software best practices Git remotes – Remote repositories • Remote – clone – remote add – push – fetch (pull) Working copy Server • Define a main repository! 10 2/17/20 Marco Mambelli | Software best practices Git resources • Clients for the major platforms – Command line – GUI (Fork, GitHub desktop, GitKraken, Tower,…) • Online hosting – Bitbucket – GitLab – GitHub • Hosting at Fermilab – Git integrated with Redmine – GitLab instance 11 2/17/20 Marco Mambelli | Software best practices Do I need to put everything in GitHub? • Word processors have frequently an integrated VC – Track changes in MS Word – Version history in Google Docs – Recording and Displaying Changes in Libre Office • Not ideal for large binary files – File sharing have versioning systems • Version history in Dropbox • Versions in Box • Version History in OneDrive – Dedicated versioning apps, e.g. ForeverSave – OS X time machine and other backup solutions 12 2/17/20 Marco Mambelli | Software best practices Use Git for … • Code or text like documents* • When you need to know – What changed? – Who changed it? – Why? * Git provides LFS (Large File Storage) https://git-lfs.github.com/ 13 2/17/20 Marco Mambelli | Software best practices Centralized workflow • Single (remote) repo • Single ordered flow • Conflicts solved one at the time by the developer 14 2/17/20 Marco Mambelli | Software best practices Feature branching workflow • Single (remote) repo • Leverage branches: • master (releases) • Development • Features • Hotfixes • Easier to enforce https://nvie.com/posts/a-successful-git-branching-model/ policies 15 2/17/20 Marco Mambelli | Software best practices Fork and branch workflow R1 R2 • Multiple forked repos • Leverage branches • Feature branches in forked repos • Squash R1 • Rebase • Pull requests R2 • Even easier to enforce policies • Restricted access 16 2/17/20 Marco Mambelli | Software best practices Final Git recommendations • Write meaningful commit messages – First line is the summary – Enough detail to understand the changes • Access to the repository based on software purpose – Least Privilege approach – Consider signing commits https://help.github.com/en/articles/signing-commits • Public software should have a license – LICENSE (text file in the root of the repository) – BSD 3-clause, Apache 2.0 – At Fermilab you can get help in picking and reviewing a license • Contact Aaron Sauers • https://cdcvs.fnal.gov/redmine/projects/scd-cst/wiki/Software_licensing • A DOI, Digital Object Identifier, can facilitate citations – https://about.zenodo.org/ 17 2/17/20 Marco Mambelli | Software best practices What should never go in Git? • PASSWORDS! • Any credential: SSH keys, certificates, … • Privale or PII – IP addresses – Names, birth dates, SSN … 18 2/17/20 Marco Mambelli | Software best practices Documentation • Target audience – Developers (including your future self) – Users (operators/end users) • Just a README (README.md) • Common documents in a complex project – Requirement documentation – Design documentation – SQA/Test documentation – Installation and operation documentation – User manual – API specs 19 2/17/20 Marco Mambelli | Software best practices Documentation resources • For software or to share information • Fermilab supports – Microsoft SharePoint – WordPress – Custom website on Apache (HTML, CSS, includes) – DocDb • GitHub Pages – https://pages.github.com/ – Markdown 20 2/17/20 Marco Mambelli | Software best practices Outline • Version control and Git • Documentation • Requirements • Design – Technology selection – OS Requirements – Software inputs – Software logs, metrics and accounting • Code development • Validation and testing • Releases • Deployment • Bug tracking • Change management • Critical services operation 21 2/17/20 Marco Mambelli | Software best practices Planning and Requirements • Introduction – Purpose – Scope – Rationale – Terminology • Overview • Plan in advance • Requirements – Actors • Write requirements – The Major Inputs and Output – Behavioral requirements (use cases) – Constraints – Actors • Architectural Overview – Roles – The Major Inputs and Output – Functional unit or Component block diagram – Physical unit block diagram – Behavioral requirements (use – Deployment scenario • Component Interfaces cases) • Protocols • Discussion – Decisions and Choices – Constraints – Rationale – Implications resulting from Choices • Example document – Resulting rules – Constraints imposed on other systems – Outline • Testing considerations – Link: https://drive.google.com/file/d/1Lk7ku0GcqZ5IJFO3pTEIM_R0DmG0aHvG/view?usp=sharing 22 2/17/20 Marco Mambelli | Software best practices Technology selection • Is it there a product providing a similar functionality? • Or a component needed? • If yes – How well is adopted? – Open Source? – With active community support? – Accepts feature requests/contributions? 23 2/17/20 Marco Mambelli | Software best practices Technology selection: more questions • Performance and scalability requirements • Current and future platform requirements • Long term costs – Licensing and support fees – In-house support • Industry trends (be aware but independent) • Strength and weaknesses of the developers in the project • Team consensus on the technology 24 2/17/20 Marco Mambelli | Software best practices OS requirements • These are good practices in general and requirements at Fermilab • Recent and supported versions of Operating System – Lab configurations management practices https://cd-docdb.fnal.gov/cgi-bin/sso/RetrieveFile?docid=4264&filename=ConfigMgmt.pdf&version=3 – Listed baseline configurations https://fermipoint.fnal.gov/org/cs/pages/computer-security-documents---general-computing-environment.aspx • Rare and well documented exceptions – ServiceNow request https://cd-docdb.fnal.gov/cgi-bin/sso/RetrieveFile?docid=4141&filename=Variance-Request-process-rc.docx&version=2 https://fermi.service-now.com/navpage.do 25 2/17/20 Marco Mambelli | Software best practices Software inputs (configuration) • Avoid hardcoded inputs! • Evaluate all inputs – How frequently does it change (install, all invocations, …)? – Who is providing it (expert, trusted, person…)? • Possible input mechanisms – Configuration files – Command line (options and arguments) – Environment variables – Interactive inputs (GUI, prompt, …) 26 2/17/20 Marco Mambelli | Software best practices Software logs and metrics Monitoring, accounting, troubleshooting and debugging • Have a consistent structure • Readable for humans • Structured for machine parsing • Configurable severity levels • Distinguish between debug/info/warning/error/fatal messages • Use an existing library (log4j, python logging, …) • Well structured tags • Independent control for logging of different components 27 2/17/20 Marco Mambelli | Software best practices Log levels TRACE • finer-grained informational events than the DEBUG • fine-grained informational events that are most useful to debug an application • informational messages that highlight the progress of the application at coarse-grained level • potentially harmful situations • error events that might still allow the application to continue running • very severe error events that will presumably lead the application to abort. 28 2/17/20 Marco Mambelli | Software best practices Software development (coding) • Provide clear coding guidelines – PEP8,