DOCSLIB.ORG

Tuning Mysql Operating System Is in a Good Database Performance Depends on a Number of Factors You Position to Take Off

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Tuning Mysql Operating System Is in a Good Database Performance Depends on a Number of Factors You Position to Take Off LINUX MAGAZINE November 2010 NEWS 8 NEWS Find out why the MeeGo mobile Tuning MySQL operating system is in a good Database performance depends on a number of factors you position to take off. must identify, analyze, and fine-tune in a systematic way. 9 DELTACLOUD Learn how to test, measure, and optimize your MySQL Red Hat releases its cloud APIs installation from the bottom up by looking at your into the open source wilderness. hardware, operating system, and database. 10 MOZILLA Mozilla announces the Firefox 4 Beta and Fennec Alpha releases and a new gaming platform. MySQL Tuning 18Take a holistic approach to 11 UTOUCH analyzing and optimizing your MySQL Shuttleworth talks about the new database. gesture suite for touch devices. 12 GALAXY TAB TCP Tuning Tips 28Return to the fundamentals Samsung reveals its Android then apply some simple 2.2-powered tablet. techniques to keep your 14 TECH TOOLS network humming. Useful tools for the tech domain. Miro 32Breaking news 24/7 can leave Speeding Up Python you feeling behind the times before 36Stop waiting on your code to you get up in the morning. The Miro execute. Learn about some cool tools SERVICE media aggregator helps you keep up. for speeding up your Python apps. 3 Comment 15 DVD 16 Letters Community Notebook 96 Featured Events 84 Cache 88 Anna Torvalds Then and now: Rikki looks back 10 Take the Torvalds tour of Helsinki. 98 Preview years to the first issue. 90 Nils Torvalds 85 Doghouse Early influences. Linux Magazine ISSN 1471-5678 Memory doesn't always serve us Linux Magazine is published monthly well, so maddog sets us straight. 92 Projects on the Move by Linux New Media Ltd, Manchester, Democratic decision making; data England. Company registered in England. 86 Kernel News sharing using a Git repository. 4 NOVEMBER 2010 ISSUE 120 LINUX-MAGAZINE.COM | LINUXPROMAGAZINE.COM 004-005_TOC.indd 4 16.09.2010 17:06:07 Uhr LINUX MAGAZINE November 2010 HIGHLIGHTS MYSQL PYTHON iTUNES PHATCH 18 Analyze the individual 36 FInd out why the 48 New HTML5 features 76 Watermark your web hardware and software Python language is so and a little Python let galleries with GIMP, components of your slow and what you can you stream your music ImageMagick, and MySQL database. do to speed it up. over the Internet. Phatch. FEATURES LINUXUSER 68 Command Line A number of tools helps you GNU Parallel schedule automatic backups and other system tasks. 44 Put those CPU cores on your network to good use with this 72 Clementine command-line tool. This media player manages your collection and plays your tunes, too. Accessing iTunes 76 Phatch 48 Use the backup file of your iTunes Fix your photos fast with this library to stream your music over batch-mode image processing tool. the Internet. 80 Workspace: OOo Extensions 41 Charly: GestióIP 48 Accessing Your iTunes Create and package OpenOffice Organize your IP addresses. Library extensions from scratch. Stream music over the Internet. 42 Security Lessons: Disposable Computers 58 Ask Klaus! Create then wipe a bootable USB Knoppix creator Klaus Knopper drive for sensitive jobs online. answers your Linux questions. 44 GNU Parallel 62 Perl: RRDtool Multi-core processing from the An open source library with Holt- command line. Winters forecasting. Searchable Linux Archive: 10,000 Pages of Practical Know-How 10th Anniversary DVD ■ Thousands of articles – PDF and HTML ■ 7.5GB dual-layer DVD SEE P15 FOR FULL DETAILS ■ All the tricks, all the hacks, all the apps LINUX-MAGAZINE.COM | LINUXPROMAGAZINE.COM ISSUE 120 NOVEMBER 2010 5 004-005_TOC.indd 5 16.09.2010 17:06:16 Uhr.
Load more

Recommended publications
  • Parallel Processing.Pptx
    Compute Grid: Parallel Processing RCS Lunch & Learn Training Series Bob Freeman, PhD Director, Research Technology Operations HBS 8 November, 2017 Overview • Q&A • Introduction • Serial vs parallel • Approaches to Parallelization • Submitting parallel jobs on the compute grid • Parallel tasks • Parallel Code Serial vs Parallel work Serial vs Multicore Approaches Traditionally, software has been written for serial computers • To be run on a single computer having a single Central Processing Unit (CPU) • Problem is broken into a discrete set of instructions • Instructions are executed one after the other • One one instruction can be executed at any moment in time Serial vs Multicore Approaches In the simplest sense, parallel computing is the simultaneous use of multiple compute resources to solve a computational problem: • To be run using multiple CPUs • A problem is broken into discrete parts (either by you or the application itself) that can be solved concurrently • Each part is further broken down to a series of instructions • Instructions from each part execute simultaneously on different CPUs or different machines Serial vs Multicore Approaches Many different parallelization approaches, which we won't discuss: Shared memory Distributed memory 6 Hybrid Distributed-Shared memory Parallel Processing… So, we are going to briefly touch on two approaches: • Parallel tasks • Tasks in the background • gnu_parallel • Pleasantly parallelizing • Parallel code • Considerations for parallelizing • Parallel frameworks & examples We will not discuss parallelized frameworks such as Hadoop, Apache Spark, MongoDB, ElasticSearch, etc Parallel Jobs on the Compute Grid… Nota Bene!! • In order to run in parallel, programs (code) must be explicitly programmed to do so. • And you must ask the scheduler to reserve those cores for your program/work to use.
    [Show full text]
  • Pipenightdreams Osgcal-Doc Mumudvb Mpg123-Alsa Tbb
    pipenightdreams osgcal-doc mumudvb mpg123-alsa tbb-examples libgammu4-dbg gcc-4.1-doc snort-rules-default davical cutmp3 libevolution5.0-cil aspell-am python-gobject-doc openoffice.org-l10n-mn libc6-xen xserver-xorg trophy-data t38modem pioneers-console libnb-platform10-java libgtkglext1-ruby libboost-wave1.39-dev drgenius bfbtester libchromexvmcpro1 isdnutils-xtools ubuntuone-client openoffice.org2-math openoffice.org-l10n-lt lsb-cxx-ia32 kdeartwork-emoticons-kde4 wmpuzzle trafshow python-plplot lx-gdb link-monitor-applet libscm-dev liblog-agent-logger-perl libccrtp-doc libclass-throwable-perl kde-i18n-csb jack-jconv hamradio-menus coinor-libvol-doc msx-emulator bitbake nabi language-pack-gnome-zh libpaperg popularity-contest xracer-tools xfont-nexus opendrim-lmp-baseserver libvorbisfile-ruby liblinebreak-doc libgfcui-2.0-0c2a-dbg libblacs-mpi-dev dict-freedict-spa-eng blender-ogrexml aspell-da x11-apps openoffice.org-l10n-lv openoffice.org-l10n-nl pnmtopng libodbcinstq1 libhsqldb-java-doc libmono-addins-gui0.2-cil sg3-utils linux-backports-modules-alsa-2.6.31-19-generic yorick-yeti-gsl python-pymssql plasma-widget-cpuload mcpp gpsim-lcd cl-csv libhtml-clean-perl asterisk-dbg apt-dater-dbg libgnome-mag1-dev language-pack-gnome-yo python-crypto svn-autoreleasedeb sugar-terminal-activity mii-diag maria-doc libplexus-component-api-java-doc libhugs-hgl-bundled libchipcard-libgwenhywfar47-plugins libghc6-random-dev freefem3d ezmlm cakephp-scripts aspell-ar ara-byte not+sparc openoffice.org-l10n-nn linux-backports-modules-karmic-generic-pae
    [Show full text]
  • Pash: Light-Touch Data-Parallel Shell Processing
    PaSh: Light-touch Data-Parallel Shell Processing Nikos Vasilakis∗ Konstantinos Kallas∗ Konstantinos Mamouras MIT University of Pennsylvania Rice University [email protected] [email protected] [email protected] Achilles Benetopoulos Lazar Cvetković Unaffiliated University of Belgrade [email protected] [email protected] Abstract Parallelizability Parallelizing Runtime Classes §3 Transformations §4.3 Primitives §5 Dataflow This paper presents PaSh, a system for parallelizing POSIX POSIX, GNU §3.1 Regions shell scripts. Given a script, PaSh converts it to a dataflow Annotations §3.2 § 4.1 DFG § 4.4 graph, performs a series of semantics-preserving program §4.2 transformations that expose parallelism, and then converts Seq. Script Par. Script the dataflow graph back into a script—one that adds POSIX constructs to explicitly guide parallelism coupled with PaSh- Fig. 1. PaSh overview. PaSh identifies dataflow regions (§4.1), converts provided Unix-aware runtime primitives for addressing per- them to dataflow graphs (§4.2), applies transformations (§4.3) based onthe parallelizability properties of the commands in these regions (§3.1, §3.2), formance- and correctness-related issues. A lightweight an- and emits a parallel script (§4.4) that uses custom primitives (§5). notation language allows command developers to express key parallelizability properties about their commands. An accompanying parallelizability study of POSIX and GNU • Command developers, responsible for implementing indi- commands—two large and commonly used groups—guides vidual commands such as sort, uniq, and jq. These de- the annotation language and optimized aggregator library velopers usually work in a single programming language, that PaSh uses. PaSh’s extensive evaluation over 44 unmod- leveraging its abstractions to provide parallelism when- ified Unix scripts shows significant speedups (0.89–61.1×, ever possible.
    [Show full text]
  • GNU Astronomy Utilities
    GNU Astronomy Utilities Astronomical data manipulation and analysis programs and libraries for version 0.7, 8 August 2018 Mohammad Akhlaghi Gnuastro (source code, book and webpage) authors (sorted by number of commits): Mohammad Akhlaghi ([email protected], 1101) Mos`eGiordano ([email protected], 29) Vladimir Markelov ([email protected], 18) Boud Roukema ([email protected], 7) Leindert Boogaard ([email protected], 1) Lucas MacQuarrie ([email protected], 1) Th´er`eseGodefroy ([email protected], 1) This book documents version 0.7 of the GNU Astronomy Utilities (Gnuastro). Gnuastro provides various programs and libraries for astronomical data manipulation and analysis. Copyright c 2015-2018 Free Software Foundation, Inc. Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.3 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license is included in the section entitled \GNU Free Documentation License". For myself, I am interested in science and in philosophy only because I want to learn something about the riddle of the world in which we live, and the riddle of man's knowledge of that world. And I believe that only a revival of interest in these riddles can save the sciences and philosophy from narrow specialization and from an obscurantist faith in the expert's special skill, and in his personal knowledge and authority; a faith that so well fits our `post-rationalist' and `post- critical' age, proudly dedicated to the destruction of the tradition of rational philosophy, and of rational thought itself.
    [Show full text]
  • OSS Alphabetical List and Software Identification
    Annex: OSS Alphabetical list and Software identification Software Short description Page A2ps a2ps formats files for printing on a PostScript printer. 149 AbiWord Open source word processor. 122 AIDE Advanced Intrusion Detection Environment. Free replacement for Tripwire(tm). It does the same 53 things are Tripwire(tm) and more. Alliance Complete set of CAD tools for the specification, design and validation of digital VLSI circuits. 114 Amanda Backup utility. 134 Apache Free HTTP (Web) server which is used by over 50% of all web servers worldwide. 106 Balsa Balsa is the official GNOME mail client. 96 Bash The Bourne Again Shell. It's compatible with the Unix `sh' and offers many extensions found in 147 `csh' and `ksh'. Bayonne Multi-line voice telephony server. 58 Bind BIND "Berkeley Internet Name Daemon", and is the Internet de-facto standard program for 95 turning host names into IP addresses. Bison General-purpose parser generator. 77 BSD operating FreeBSD is an advanced BSD UNIX operating system. 144 systems C Library The GNU C library is used as the C library in the GNU system and most newer systems with the 68 Linux kernel. CAPA Computer Aided Personal Approach. Network system for learning, teaching, assessment and 131 administration. CVS A version control system keeps a history of the changes made to a set of files. 78 DDD DDD is a graphical front-end for GDB and other command-line debuggers. 79 Diald Diald is an intelligent link management tool originally named for its ability to control dial-on- 50 demand network connections. Dosemu DOSEMU stands for DOS Emulation, and is a linux application that enables the Linux OS to run 138 many DOS programs - including some Electric Sophisticated electrical CAD system that can handle many forms of circuit design.
    [Show full text]
  • Openstackでnecが実現する 「OSSクラウド」の世界
    ＯｐｅｎＳｔａｃｋでＮＥＣが実現する 「ＯＳＳクラウド」の世界 ２０１３年３月１２日 日本電気株式会社 プラットフォームマーケティング戦略本部 ＯＳＳ推進室 技術主幹 高橋 千恵子 目次 ▐ ＯＳＳの動向 ▐ ＮＥＣのＯＳＳ/Ｌｉｎｕｘ事業 ▐ ＯＳＳプラットフォームへの取り組み 高可用Ｌｉｎｕｘプラットフォーム ＯＳＳミドルウェアの活用 ▐ ＯＳＳクラウド ＯｐｅｎＳｔａｃｋへの取り組み ＯＳＳクラウドソリューション ～沖縄クラウドサービス基盤～ ＯｐｅｎＦｌｏｗ+ＯｐｅｎＳｔａｃｋ ▐ 最後に ●本資料に掲載された社名、商品名は各社の商標または登録商標です。 Page 1 © NEC Corporation 2013 ＯＳＳの動向 Page 2 © NEC Corporation 2013 ＯＳＳコミュニティによるプロジェクト ▐ 全般 ・・・・・SaaS型アプリケーションが増加/著名OSSのベンダー配布、有償サポートや買収が進む OSS定点観測は、freecode.com にて実施。 ▐ ｲﾝﾌﾗ系・・・仮想化やクラウド基盤関係のOSSが注目される/システム、ネット管理OSSの伸びがある サイトでの人気度（300以上)で順位付け。 OSSプロジェクトは2012.4時点で32.4万件。 ▐ ﾃﾞﾌﾟﾛｲﾒﾝﾄ系・・PostgreSQL、mySQLが高人気 これらのDB管理、DBクラスタリングOSSも伸びている ビジネス用途を中心に表示。 ▐ ｱﾌﾟﾘ系・・・・SaaS型グループウェアOSS、ソフトウェア分類を越えた統合的な著名OSSの伸びがある コンシューマ・プライベート系は除く。 コンテンツアプリケーション •PHProject [GW] エンジニ ｺﾗﾎﾞﾚｰﾃｨﾌﾞ •EGroupware CRM ERM SCM • Dokuwiki [Wiki] • jGnash アリング • WebGUI [CMF] • phpBB [GW] •OBM [GW] • Enterprise CRM and ア • Tiki Tiki CMS Groupware • ProcessMaker • CorneliOS [CMS] • Ariadne Groupware System • Task Juggler [PM] • Blender オペレーション • TinyMCE [DCM] • XODA • Simple Groupware [GW] プ • OpenWebMail[webmail]• Teamwork [GW] • Dolibarr • Achievo [PM] • mxGraph 製造管理 • Drupal [CMS] • mnoGoSerch • LedgerSMB • white_dune • eZpublish [publish FW]• Managing • ZIm [blog] • Plans [GW] • The Apache Open リ • Tine2.0 [CRM&GW] • GnuCash • Jgraph • OTRS • XWiki [Wiki] • Midgard • Zimbra [GW] for Business • Elastix • GroupOffice [GW] • Twiki [KB] • TUTOS[ERP&PM] • graphviz • FUDForum • OpenSearch Project ｺﾝｼｭｰﾏ 系 • SquirrelMail [Webmail] • Time Trex • BRL-CAD • Asterisk • Plone CMS Server • Citadel [bbs] •
    [Show full text]
  • Research Document Number Plate Recognition
    Research Document Number Plate Recognition Francois Ribemont Supervisor: Nigel Whyte December 16, 2011 Contents 1 1 Introduction Number Plate Recognition (NPR) is a software that reads Irish car plate num- bers from pictures. The input of the program is an image, and the output of the program is a text string containing the car plate number. By executing a series of algorithms, the program will be able to take an image as an input and gener- ate the number plate in a text format. For the Figure ??, the output would be: 03MH1847. Figure 1: Irish car plate This software is usually embedded with video cameras. It is mainly used by the police to recognize stolen vehicles or speeding. If, for instance, a car goes too fast on a motorway, some sensors send a signal to the camera, and then a picture is taken. Then it gives the taken pictures to the NPR system, and NPR output the number plate in a text format. Irish car plates respect this format: YY-CC-SSSSSS where YY is a double digit for the year of registration, CC represents the two letters from which county the car has been registered in, and finally SSSSSS is the serial number. This serial number starts from 0 every year, and is specific to a county, meaning that 157 in Dublin is different than in Carlow. All the specifications about Irish car plate 2 regulations, such as the size of characters, thickness of the edges, distances between characters, can be found on the Irish Status Book's website [?].
    [Show full text]
  • Use of Linux Command Line Not Only for Metacentrum of CESNET
    Introduction Linux UN*X Command line Text Scripting Software MetaCentrum Administration The end Linux, command line & MetaCentrum Use of Linux command line not only for MetaCentrum of CESNET Vojtěch Zeisek Department of Botany, Faculty of Science, Charles University in Prague Institute of Botany, Czech Academy of Sciences, Průhonice https://trapa.cz/, [email protected] January 28 and 29, 2016 . Vojtěch Zeisek (https://trapa.cz/) Linux, command line & MetaCentrum January 28 and 29, 2016 1 / 146 Introduction Linux UN*X Command line Text Scripting Software MetaCentrum Administration The end Outline I 1 Introduction Licenses and money 2 Linux Choose one Differences 3 UN*X Basic theory of operating system Permissions Text FISH 4 Command line Chaining Information and management Directories . Archives . Vojtěch Zeisek (https://trapa.cz/) Linux, command line & MetaCentrum January 28 and 29, 2016 2 / 146 Introduction Linux UN*X Command line Text Scripting Software MetaCentrum Administration The end Outline II Searching Network Parallelisation Other 5 Text Reading Extractions Manipulations Editors Regular expressions 6 Scripting Basic skeleton Reading variables Branching the code Loops . Vojtěch Zeisek (https://trapa.cz/) Linux, command line & MetaCentrum January 28 and 29, 2016 3 / 146 Introduction Linux UN*X Command line Text Scripting Software MetaCentrum Administration The end Outline III 7 Software 8 MetaCentrum Information Usage Tasks Graphical connection 9 Administration File systems System services 10 The end . Vojtěch Zeisek (https://trapa.cz/)
    [Show full text]
  • Can Free Software Replace Proprietary Software for Graphic Production? Investigating to Which Extent Free Software Can Be Used for Book and Magazine Production
    Can free software replace proprietary software for graphic production? Investigating to which extent free software can be used for book and magazine production. 2013-04-25 Staffan Melin, D89, [email protected] Master´s thesis in Media Technology, School of Computer Science and Communication at the Royal Institute of Technology, Stockholm, Sweden This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported License. Abstract Free and open source software is widely used. At the same time there are several areas where it is not. One of these is graphic production where the applications from Adobe – Indesign, Photoshop and Illustrator – dominates. In this thesis I start by describing a workflow for graphic production. Next I research the field of free software and put together a set of tools that fit into this workflow. The choices are made on the basis of functionality and how they work together. I then go on to apply these tools to two real world scenarios: production of a book and a magazine. The results show that free software can be used for graphic production without any loss of quality and only minor problems compared to the proprietary tools. A look into future development shows that the bulk of these problems are being taken care of by the open source community. Keywords: free software, open source, proprietary software, graphic production, graphic design, FLOSS, FOSS, layout, libre, Scribus, Inkscape, GIMP, GNOME Color Manager. Sammanfattning Fri och open source programvara används på många områden. Samtidigt finns det många områden där det inte används. Ett av dessa är grafisk produktion där programmen från Adobe – Indesign, Photoshop och Illustrator – dominerar.
    [Show full text]
  • Application of Open-Source Enterprise Information System Modules: an Empirical Study
    University of Nebraska - Lincoln DigitalCommons@University of Nebraska - Lincoln Dissertations, Theses, and Student Research from the College of Business Business, College of Summer 7-20-2010 APPLICATION OF OPEN-SOURCE ENTERPRISE INFORMATION SYSTEM MODULES: AN EMPIRICAL STUDY Sang-Heui Lee University of Nebraska-Lincoln Follow this and additional works at: https://digitalcommons.unl.edu/businessdiss Part of the Management Information Systems Commons, Other Business Commons, and the Technology and Innovation Commons Lee, Sang-Heui, "APPLICATION OF OPEN-SOURCE ENTERPRISE INFORMATION SYSTEM MODULES: AN EMPIRICAL STUDY" (2010). Dissertations, Theses, and Student Research from the College of Business. 13. https://digitalcommons.unl.edu/businessdiss/13 This Article is brought to you for free and open access by the Business, College of at DigitalCommons@University of Nebraska - Lincoln. It has been accepted for inclusion in Dissertations, Theses, and Student Research from the College of Business by an authorized administrator of DigitalCommons@University of Nebraska - Lincoln. APPLICATION OF OPEN-SOURCE ENTERPRISE INFORMATION SYSTEM MODULES: AN EMPIRICAL STUDY by Sang-Heui Lee A DISSERTATION Presented to the Faculty of The Graduate College at the University of Nebraska In Partial Fulfillment of Requirements For the Degree of Doctor of Philosophy Major: Interdepartmental Area of Business (Management) Under the Supervision of Professor Sang M. Lee Lincoln, Nebraska July 2010 APPLICATION OF OPEN-SOURCE ENTERPRISE INFORMATION SYSTEM MODULES: AN EMPIRICAL STUDY Sang-Heui Lee, Ph.D. University of Nebraska, 2010 Advisor: Sang M. Lee Although there have been a number of studies on large scale implementation of proprietary enterprise information systems (EIS), open-source software (OSS) for EIS has received limited attention in spite of its potential as a disruptive innovation.
    [Show full text]
  • W. Augustine Dunn, III Ph.D. –
    132 Nicoll St FL 1 New Haven CT, 06511 H 770-312-9544 B [email protected] W. Augustine Dunn, III Ph.D. Í gusdunn.com Technical Expertise EXPERT Python, Bash, regular expressions, advanced data visualization, IPython notebooks, SGE & PBS HPC schedulers, YAML, HTML, CSS, XML, LATEX, pandoc, markdown, reStructuredText, Git, Python software packaging & templating, software documentation with Sphinx, Gnu Parallel INTERMEDIATE Perl, R, Makefiles/build-systems, JSON, vim, unit testing, Mercurial, Bazaar, Subversion BASIC MySQL, PostgreSQL, SQlite, Lua, Tcl, Apache, javascript PYTHON LIBS pandas, scipy, numpy, statsmodels, pyMC, matplotlib/pyplot, seaborn, ggplot, rpy2, networkx, pybedtools, pysam, cookiecutter MISC OSX, Windows, Linux, MS Word, MS Excel, Photoshop/Gimp, Illustrator/Inkscape WET-LAB RNA-seq, ddRAD-seq, proteomics, broad range of molecular biology & protein biochemistry techniques Authored Software blacktie An object oriented python pipeline that simplifies & streamlines the running of complex tophat/cufflinks- based RNA-seq experiments to a single command plus configuration file: prioritizing repeatability & usability. Downloaded from https://pypi.python.org/pypi/blacktie over 9,000 times. gFunc A python-based integrative analysis framework using network graphs to combine multidimensional data-types from disparate “Omics” sources for creating/exploiting functional-genomic gene sets across multiple species. spartan A bioinformatics package, providing the essentials to get a variety of computational jobs done quickly without flourish when that is all that is needed. Experience 2014–present Postdoctoral Associate, Dept. of Ecology & Evolutionary Biology, Yale University, New Haven, CT. Characterization of gene-flow & genotype/phenotype relationships in tsetse fly populations in Uganda. Highlights: { Supervised month-long field expedition collecting tsetse flies in northern Uganda.
    [Show full text]
  • An Order-Aware Dataflow Model for Parallel Unix Pipelines
    65 An Order-Aware Dataflow Model for Parallel Unix Pipelines SHIVAM HANDA∗, CSAIL, MIT, USA KONSTANTINOS KALLAS∗, University of Pennsylvania, USA NIKOS VASILAKIS∗, CSAIL, MIT, USA MARTIN C. RINARD, CSAIL, MIT, USA We present a dataflow model for modelling parallel Unix shell pipelines. To accurately capture the semantics of complex Unix pipelines, the dataflow model is order-aware, i.e., the order in which a node in the dataflow graph consumes inputs from different edges plays a central role in the semantics of the computation and therefore in the resulting parallelization. We use this model to capture the semantics of transformations that exploit data parallelism available in Unix shell computations and prove their correctness. We additionally formalize the translations from the Unix shell to the dataflow model and from the dataflow model backtoa parallel shell script. We implement our model and transformations as the compiler and optimization passes of a system parallelizing shell pipelines, and use it to evaluate the speedup achieved on 47 pipelines. CCS Concepts: • Software and its engineering ! Compilers; Massively parallel systems; Scripting languages. Additional Key Words and Phrases: Unix, POSIX, Shell, Parallelism, Dataflow, Order-awareness ACM Reference Format: Shivam Handa, Konstantinos Kallas, Nikos Vasilakis, and Martin C. Rinard. 2021. An Order-Aware Dataflow Model for Parallel Unix Pipelines. Proc. ACM Program. Lang. 5, ICFP, Article 65 (August 2021), 28 pages. https://doi.org/10.1145/3473570 1 Introduction Unix pipelines
    [Show full text]