Measuring Software Performance on Linux Technical Report
Total Page:16
File Type:pdf, Size:1020Kb
Load more
Recommended publications
-
Overcoming Traditional Problems with OS Huge Page Management
MEGA: Overcoming Traditional Problems with OS Huge Page Management Theodore Michailidis Alex Delis Mema Roussopoulos University of Athens University of Athens University of Athens Athens, Greece Athens, Greece Athens, Greece [email protected] [email protected] [email protected] ABSTRACT KEYWORDS Modern computer systems now feature memory banks whose Huge pages, Address Translation, Memory Compaction aggregate size ranges from tens to hundreds of GBs. In this context, contemporary workloads can and do often consume ACM Reference Format: Theodore Michailidis, Alex Delis, and Mema Roussopoulos. 2019. vast amounts of main memory. This upsurge in memory con- MEGA: Overcoming Traditional Problems with OS Huge Page Man- sumption routinely results in increased virtual-to-physical agement. In The 12th ACM International Systems and Storage Con- address translations, and consequently and more importantly, ference (SYSTOR ’19), June 3–5, 2019, Haifa, Israel. ACM, New York, more translation misses. Both of these aspects collectively NY, USA, 11 pages. https://doi.org/10.1145/3319647.3325839 do hamper the performance of workload execution. A solu- tion aimed at dramatically reducing the number of address translation misses has been to provide hardware support for 1 INTRODUCTION pages with bigger sizes, termed huge pages. In this paper, we Computer memory capacities have been increasing sig- empirically demonstrate the benefits and drawbacks of using nificantly over the past decades, leading to the development such huge pages. In particular, we show that it is essential of memory hungry workloads that consume vast amounts for modern OS to refine their software mechanisms to more of main memory. This upsurge in memory consumption rou- effectively manage huge pages. -
Asynchronous Page Faults Aix Did It Red Hat Author Gleb Natapov August 10, 2010
Asynchronous page faults Aix did it Red Hat Author Gleb Natapov August 10, 2010 Abstract Host memory overcommit may cause guest memory to be swapped. When guest vcpu access memory swapped out by a host its execution is suspended until memory is swapped back. Asynchronous page fault is a way to try and use guest vcpu more efficiently by allowing it to execute other tasks while page is brought back into memory. Part I How KVM Handles Guest Memory and What Inefficiency it Has With Regards to Host Swapping Mapping guest memory into host memory But we do it on demand Page fault happens on first guest access What happens on a page fault? 1 VMEXIT 2 kvm mmu page fault() 3 gfn to pfn() 4 get user pages fast() no previously mapped page and no swap entry found empty page is allocated 5 page is added into shadow/nested page table What happens on a page fault? 1 VMEXIT 2 kvm mmu page fault() 3 gfn to pfn() 4 get user pages fast() no previously mapped page and no swap entry found empty page is allocated 5 page is added into shadow/nested page table What happens on a page fault? 1 VMEXIT 2 kvm mmu page fault() 3 gfn to pfn() 4 get user pages fast() no previously mapped page and no swap entry found empty page is allocated 5 page is added into shadow/nested page table What happens on a page fault? 1 VMEXIT 2 kvm mmu page fault() 3 gfn to pfn() 4 get user pages fast() no previously mapped page and no swap entry found empty page is allocated 5 page is added into shadow/nested page table What happens on a page fault? 1 VMEXIT 2 kvm mmu page fault() 3 gfn to pfn() -
What Is an Operating System III 2.1 Compnents II an Operating System
Page 1 of 6 What is an Operating System III 2.1 Compnents II An operating system (OS) is software that manages computer hardware and software resources and provides common services for computer programs. The operating system is an essential component of the system software in a computer system. Application programs usually require an operating system to function. Memory management Among other things, a multiprogramming operating system kernel must be responsible for managing all system memory which is currently in use by programs. This ensures that a program does not interfere with memory already in use by another program. Since programs time share, each program must have independent access to memory. Cooperative memory management, used by many early operating systems, assumes that all programs make voluntary use of the kernel's memory manager, and do not exceed their allocated memory. This system of memory management is almost never seen any more, since programs often contain bugs which can cause them to exceed their allocated memory. If a program fails, it may cause memory used by one or more other programs to be affected or overwritten. Malicious programs or viruses may purposefully alter another program's memory, or may affect the operation of the operating system itself. With cooperative memory management, it takes only one misbehaved program to crash the system. Memory protection enables the kernel to limit a process' access to the computer's memory. Various methods of memory protection exist, including memory segmentation and paging. All methods require some level of hardware support (such as the 80286 MMU), which doesn't exist in all computers. -
Linux Kernel and Driver Development Training Slides
Linux Kernel and Driver Development Training Linux Kernel and Driver Development Training © Copyright 2004-2021, Bootlin. Creative Commons BY-SA 3.0 license. Latest update: October 9, 2021. Document updates and sources: https://bootlin.com/doc/training/linux-kernel Corrections, suggestions, contributions and translations are welcome! embedded Linux and kernel engineering Send them to [email protected] - Kernel, drivers and embedded Linux - Development, consulting, training and support - https://bootlin.com 1/470 Rights to copy © Copyright 2004-2021, Bootlin License: Creative Commons Attribution - Share Alike 3.0 https://creativecommons.org/licenses/by-sa/3.0/legalcode You are free: I to copy, distribute, display, and perform the work I to make derivative works I to make commercial use of the work Under the following conditions: I Attribution. You must give the original author credit. I Share Alike. If you alter, transform, or build upon this work, you may distribute the resulting work only under a license identical to this one. I For any reuse or distribution, you must make clear to others the license terms of this work. I Any of these conditions can be waived if you get permission from the copyright holder. Your fair use and other rights are in no way affected by the above. Document sources: https://github.com/bootlin/training-materials/ - Kernel, drivers and embedded Linux - Development, consulting, training and support - https://bootlin.com 2/470 Hyperlinks in the document There are many hyperlinks in the document I Regular hyperlinks: https://kernel.org/ I Kernel documentation links: dev-tools/kasan I Links to kernel source files and directories: drivers/input/ include/linux/fb.h I Links to the declarations, definitions and instances of kernel symbols (functions, types, data, structures): platform_get_irq() GFP_KERNEL struct file_operations - Kernel, drivers and embedded Linux - Development, consulting, training and support - https://bootlin.com 3/470 Company at a glance I Engineering company created in 2004, named ”Free Electrons” until Feb. -
Guide to Openvms Performance Management
Guide to OpenVMS Performance Management Order Number: AA–PV5XA–TE May 1993 This manual is a conceptual and tutorial guide for experienced users responsible for optimizing performance on OpenVMS systems. Revision/Update Information: This manual supersedes the Guide to VMS Performance Management, Version 5.2. Software Version: OpenVMS VAX Version 6.0 Digital Equipment Corporation Maynard, Massachusetts May 1993 Digital Equipment Corporation makes no representations that the use of its products in the manner described in this publication will not infringe on existing or future patent rights, nor do the descriptions contained in this publication imply the granting of licenses to make, use, or sell equipment or software in accordance with the description. Possession, use, or copying of the software described in this publication is authorized only pursuant to a valid written license from Digital or an authorized sublicensor. © Digital Equipment Corporation 1993. All rights reserved. The postpaid Reader’s Comments forms at the end of this document request your critical evaluation to assist in preparing future documentation. The following are trademarks of Digital Equipment Corporation: ACMS, ALL–IN–1, Bookreader, CI, DBMS, DECnet, DECwindows, Digital, MicroVAX, OpenVMS, PDP–11, VAX, VAXcluster, VAX DOCUMENT, VMS, and the DIGITAL logo. All other trademarks and registered trademarks are the property of their respective holders. ZK4521 This document was prepared using VAX DOCUMENT Version 2.1. Contents Preface ............................................................ ix 1 Introduction to Performance Management 1.1 Knowing Your Workload ....................................... 1–2 1.1.1 Workload Management .................................... 1–3 1.1.2 Workload Distribution ..................................... 1–4 1.1.3 Code Sharing ............................................ 1–4 1.2 Evaluating User Complaints . -
Virtual Memory §5.4 Virt Virtual Memory U Al Memo Use Main Memory As a “Cache” For
Chapter 5 Large and Fast: Exppgloiting Memory Hierarchy Part II Virtual Memory §5.4 Virt Virtual Memory u al Memo Use main memory as a “cache” for secondary (disk) storage r y Managed jointly by CPU hardware and the operating system (OS) Programs share main memory Each gets a private virtual address space holdinggqy its frequently used code and data Protected from other programs CPU and OS translate virtual addresses to physical addresses VM “block” is called a page VM translation “miss” is called a page fault Chapter 5 — Large and Fast: Exploiting Memory Hierarchy — 2 Address Translation Fixed-size pages (e.g., 4K) Chapter 5 — Large and Fast: Exploiting Memory Hierarchy — 3 Page Fault Penalty On page fault, the page must be fetched from disk Takes millions of clock cycles Handled by OS code Try to minimize page fault rate Fully associative placement Smart replacement algorithms Chapter 5 — Large and Fast: Exploiting Memory Hierarchy — 4 Page Tables Stores placement information Array of page table entries, indexed by virtual page number Page table register in CPU points to page table in physical memory If page is present in memory PTE stores the physical page number Plus other status bits (referenced, dirty, …) If page is not present PTE can refer to location in swap space on dis k Chapter 5 — Large and Fast: Exploiting Memory Hierarchy — 5 Translation Using a Page Table Chapter 5 — Large and Fast: Exploiting Memory Hierarchy — 6 Mapping Pages to Storage Chapter 5 — Large and Fast: Exploiting Memory Hierarchy — 7 Replacement and Writes -
Operating Systems
UC Santa Barbara Operating Systems Christopher Kruegel Department of Computer Science UC Santa Barbara http://www.cs.ucsb.edu/~chris/ Virtual Memory and Paging UC Santa Barbara • What if a program is too big to be loaded in memory • What if a higher degree of multiprogramming is desirable • Physical memory is split in page frames • Virtual memory is split in pages • OS (with help from the hardware) manages the mapping between pages and page frames 2 Mapping Pages to Page Frames UC Santa Barbara • Virtual memory: 64KB • Physical memory: 32KB • Page size: 4KB • Virtual memory pages: 16 • Physical memory pages: 8 3 Memory Management Unit UC Santa Barbara • Automatically performs the mapping from virtual addresses into physical addresses 4 Memory Management Unit UC Santa Barbara • Addresses are split into a page number and an offset • Page numbers are used to look up a table in the MMU with as many entries as the number of virtual pages • Each entry in the table contains a bit that states if the virtual page is actually mapped to a physical one • If it is so, the entry contains the number of physical page used • If not, a page fault is generated and the OS has to deal with it 5 Page Tables UC Santa Barbara • Page tables contain an entry for each virtual table • If virtual memory is big (e.g., 32 bit and 64 bit addresses) the table can become of unmanageable size • Solution: instead of keeping them in the MMU move them to main memory • Problem: page tables are used each time an access to memory is performed. -
An Evolutionary Study of Linux Memory Management for Fun and Profit Jian Huang, Moinuddin K
An Evolutionary Study of Linux Memory Management for Fun and Profit Jian Huang, Moinuddin K. Qureshi, and Karsten Schwan, Georgia Institute of Technology https://www.usenix.org/conference/atc16/technical-sessions/presentation/huang This paper is included in the Proceedings of the 2016 USENIX Annual Technical Conference (USENIX ATC ’16). June 22–24, 2016 • Denver, CO, USA 978-1-931971-30-0 Open access to the Proceedings of the 2016 USENIX Annual Technical Conference (USENIX ATC ’16) is sponsored by USENIX. An Evolutionary Study of inu emory anagement for Fun and rofit Jian Huang, Moinuddin K. ureshi, Karsten Schwan Georgia Institute of Technology Astract the patches committed over the last five years from 2009 to 2015. The study covers 4587 patches across Linux We present a comprehensive and uantitative study on versions from 2.6.32.1 to 4.0-rc4. We manually label the development of the Linux memory manager. The each patch after carefully checking the patch, its descrip- study examines 4587 committed patches over the last tions, and follow-up discussions posted by developers. five years (2009-2015) since Linux version 2.6.32. In- To further understand patch distribution over memory se- sights derived from this study concern the development mantics, we build a tool called MChecker to identify the process of the virtual memory system, including its patch changes to the key functions in mm. MChecker matches distribution and patterns, and techniues for memory op- the patches with the source code to track the hot func- timizations and semantics. Specifically, we find that tions that have been updated intensively. -
Review Der Linux Kernel Sourcen Von 4.9 Auf 4.10
Review der Linux Kernel Sourcen von 4.9 auf 4.10 Reviewed by: Tested by: stecan stecan Period of Review: Period of Test: From: Thursday, 11 January 2018 07:26:18 o'clock +01: From: Thursday, 11 January 2018 07:26:18 o'clock +01: To: Thursday, 11 January 2018 07:44:27 o'clock +01: To: Thursday, 11 January 2018 07:44:27 o'clock +01: Report automatically generated with: LxrDifferenceTable, V0.9.2.548 Provided by: Certified by: Approved by: Account: stecan Name / Department: Date: Friday, 4 May 2018 13:43:07 o'clock CEST Signature: Review_4.10_0_to_1000.pdf Page 1 of 793 May 04, 2018 Review der Linux Kernel Sourcen von 4.9 auf 4.10 Line Link NR. Descriptions 1 .mailmap#0140 Repo: 9ebf73b275f0 Stephen Tue Jan 10 16:57:57 2017 -0800 Description: mailmap: add codeaurora.org names for nameless email commits ----------- Some codeaurora.org emails have crept in but the names don't exist for them. Add the names for the emails so git can match everyone up. Link: http://lkml.kernel.org/r/[email protected] 2 .mailmap#0154 3 .mailmap#0160 4 CREDITS#2481 Repo: 0c59d28121b9 Arnaldo Mon Feb 13 14:15:44 2017 -0300 Description: MAINTAINERS: Remove old e-mail address ----------- The ghostprotocols.net domain is not working, remove it from CREDITS and MAINTAINERS, and change the status to "Odd fixes", and since I haven't been maintaining those, remove my address from there. CREDITS: Remove outdated address information ----------- This address hasn't been accurate for several years now. -
OMV Crash 25Feb2020 Syslog
Feb 25 00:00:00 prometheus rsyslogd: [origin software="rsyslogd" swVersion="8.1901.0" x-pid="858" x-info="https://www.rsyslog.com"] rsyslogd was HUPed Feb 25 00:00:00 prometheus systemd[1]: logrotate.service: Succeeded. Feb 25 00:00:00 prometheus systemd[1]: Started Rotate log files. Feb 25 00:00:01 prometheus CRON[10420]: (root) CMD (/usr/sbin/omv- mkrrdgraph >/dev/null 2>&1) Feb 25 00:09:00 prometheus systemd[1]: Starting Clean php session files... Feb 25 00:09:00 prometheus sessionclean[10668]: PHP Warning: PHP Startup: Unable to load dynamic library '/usr/lib/php/20151012/pam.so' - /usr/lib/php/20151012/pam.so: cannot open shared object file: No such file or directory in Unknown on line 0 Feb 25 00:09:00 prometheus sessionclean[10668]: PHP Warning: PHP Startup: Unable to load dynamic library '/usr/lib/php/20151012/pam.so' - /usr/lib/php/20151012/pam.so: cannot open shared object file: No such file or directory in Unknown on line 0 Feb 25 00:09:00 prometheus sessionclean[10668]: PHP Warning: PHP Startup: Unable to load dynamic library '/usr/lib/php/20151012/pam.so' - /usr/lib/php/20151012/pam.so: cannot open shared object file: No such file or directory in Unknown on line 0 Feb 25 00:09:00 prometheus systemd[1]: phpsessionclean.service: Succeeded. Feb 25 00:09:00 prometheus systemd[1]: Started Clean php session files. Feb 25 00:09:01 prometheus CRON[10770]: (root) CMD ( [ -x /usr/lib/ php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/ php/sessionclean; fi) Feb 25 00:15:01 prometheus CRON[10858]: (root) CMD (/usr/sbin/omv- mkrrdgraph >/dev/null 2>&1) Feb 25 00:17:01 prometheus CRON[10990]: (root) CMD ( cd / && run- parts --report /etc/cron.hourly) Feb 25 00:17:01 prometheus postfix/postsuper[10993]: fatal: scan_dir_push: open directory hold: No such file or directory Feb 25 00:30:01 prometheus CRON[11200]: (root) CMD (/usr/sbin/omv- mkrrdgraph >/dev/null 2>&1) Feb 25 00:39:00 prometheus systemd[1]: Starting Clean php session files.. -
Open Source Practice of LG Electronics
Open Source Practice of LG Electronics Hyo Jun Im Software Platform Laboratory, LG Electronics Introduction to LG Electronics LG Electronics is a global leader and technology innovator in consumer electronics, mobile communications and home appliances. Overview This presentation covers both the open source compliance and open source engagement Compliance LG Community Engagement Open Source Compliance How LG Electronics built the open source compliance process from scratch. Steps for Building Up Open Source Compliance Process | Kick-starting open source compliance | Raising awareness | Setting up the process | Filling holes in the process | Efficient open source compliance Kick-starting Open Source Compliance | Challenges > Lack of knowledge on open source compliance > Lack of verification tools | What we did > Just rushed into writing the open source software notice in the user manual > Manual open source identification and verification | What we got > Basic knowledge of popular open source licenses > Need for automated verification tools > Knowledge on practical issues with open source compliance Raising Awareness | Introduction to relevant lawsuit case | Online / Offline training | Escalation of the open source compliance issues to management | Help from outside experts Industry Lawsuit Case • 2006 Set the precedent that the sale of a product can be prohibited 2007 • Distributors can be held responsible, and lawsuits filed against them • 2008 Similar to the case with Skype • 2010 Forced to give TVs to charity organizations as well as -
Red Hat Enterprise Linux Roadmap
RED HAT ENTERPRISE LINUX ROADMAP Transforming Application Delivery While Protecting Your Application Investments Ron Pacheco, Director, Product Management Karen Noel, Director of Engineering May 2, 2017 Disclaimer The content set forth herein does not constitute in any way a binding or legal agreement or impose any legal obligation or duty on Red Hat. This information is provided for discussion purposes only and is subject to change for any or no reason. MARKET REQUIREMENTS “Light up my hardware.” “Enable my operations.” Circa 2002 Circa 2017 ● Light up my hardware ● Everything I’m used to and... ● Award winning support ● More software available ● Vast ecosystem ● Less software installed ● Peerless security ● Non-disruptive updates ● Lifecycle options ● More automation BRIEF HISTORY OF RED HAT ENTERPRISE LINUX Not pictured: tens of thousands of incremental improvements Control Groups Mandatory access controls Secure Virtualization IBM Power Automated security Real-time kernel IBM Z Series RHEL is born. audits ‘02 ‘03 ‘05 ‘07 ‘10 ‘11 ‘12 ‘13 ‘15 Virtualization Containers Software-defined Cluster file Software-defined Networking systems storage FLEXIBILITY ACROSS DEPLOYMENT TARGETS via a Trusted, Consistent Host Environment RED HAT ENTERPRISE LINUX IS THE FOUNDATION RED HAT ENTERPRISE LINUX ROADMAP Red Hat CY2016 CY2017 Summit CY2018 CY2019 CY2020 2017 RHEL 7 RHEL ATOMIC HOST RHEL FOR REAL TIME .2 .3 .4 .5 .6.5 .7 RHEL FOR SAP APPLICATIONS RHEL FOR SAP HANA RHEL 6 RHEL FOR SAP APPLICATIONS .8 .9 .10 RHEL FOR SAP HANA RHEL 5 .11 SOFTWARE