DOCSLIB.ORG

Using Python for Interactive Data Analysis

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Using Python for Interactive Data Analysis Using Python for Interactive Data Analysis Perry Greenfield Robert Jedrzejewski Space Telescope Science Institute June 13, 2005 1 Copyright (c) 2005, Association of Universities for Research in Astronomy, Inc (AURA). All rights reserved. 2 Contents Purpose 7 Prerequisites 7 Practicalities 7 1 Reading and manipulating image data 8 1.1 Example session to read and display an image from a FITS file . 8 1.2 Starting the Python interpreter . 8 1.3 Loading modules . 8 1.4 Reading data from FITS files . 9 1.5 Displaying images . 9 1.6 Array expressions . 9 1.7 FITS headers . 10 1.8 Writing data to FITS files . 10 1.9 Some Python basics . 11 1.9.1 Memory vs. data files . 11 1.9.2 Python variables . 11 1.9.3 How does object oriented programming affect you? . 11 1.9.4 Errors and dealing with them . 12 1.10 Array basics . 13 1.10.1 Creating arrays . 13 1.10.2 Array numeric types . 14 1.10.3 Printing arrays . 14 1.10.4 Indexing 1-D arrays . 15 1.10.5 Indexing multidimensional arrays . 16 1.10.6 Compatibility of dimensions . 16 1.10.7 ufuncs . 17 1.10.8 Array functions . 18 1.10.9 Array methods . 20 1.10.10 Array attributes: . 21 1.11 Example . 21 1.12 Exercises . 21 2 Reading and plotting spectral data 23 2.1 Example session to read spectrum and plot it . 23 2.2 An aside on how Python finds modules . 23 2.3 Reading FITS table data . 24 2.4 Quick introduction to plotting . 25 2.4.1 Simple x-y plots . 25 2.4.2 Labeling plot axes . 26 2.4.3 Overplotting . 27 2.4.4 Legends and annotation . 28 2.4.5 Saving and printing plots . 28 2.5 A little background on Python sequences . 29 2.5.1 Strings . 29 2.5.2 Lists . 31 2.5.3 Tuples . 31 2.5.4 Standard operations on sequences . 32 2.5.5 Dictionaries . 32 2.5.6 A section about nothing . 34 2.6 More on plotting . 34 2.6.1 matplotlib terminology, configuration and modes of usage . 34 3 2.6.2 matplot functions . 35 2.7 Plotting mini-Cookbook . 36 2.7.1 customizing standard plots . 36 2.7.2 implot example . 38 2.7.3 imshow example . 39 2.7.4 figimage . 40 2.7.5 histogram example . 40 2.7.6 contour example . 41 2.7.7 subplot example . 42 2.7.8 readcursor example . 43 2.8 Exercises . 44 3 More advanced topics in PyFITS, numarray and IPython 45 3.1 IPython . 45 3.1.1 Obtaining information about Python objects and functions . 45 3.1.2 Access to the OS shell . 46 3.1.3 Magic commands . 47 3.1.4 Syntax shortcuts . 48 3.1.5 IPython history features . 49 3.2 Python Introspection . 49 3.3 Saving your data . 50 3.4 Python loops and conditionals . 50 3.4.1 The for statement . 50 3.4.2 Blocks of code, and indentation . 51 3.4.3 Python if statements . 51 3.4.4 What’s True and what’s False . 52 3.4.5 The in crowd . 52 3.5 Advanced PyFITS Topics . 52 3.5.1 Header manipulations . 52 3.5.2 PyFITS Object Oriented Interface . 53 3.5.3 Controlling memory usage . 53 3.5.4 Support for common FITS conventions . 54 3.5.5 Noncompliant FITS data . 54 3.6 A quick tour of standard numarray packages . 55 3.6.1 random_array . 55 3.6.2 fft . 56 3.6.3 convolve . 57 3.6.4 linear_algebra . 57 3.6.5 MA (Masked Arrays) . 57 3.6.6 nd_image (Multi-dimensional array processing) . 58 3.6.7 ieespecial . 58 3.7 Intermediate numarray topics . ..
Load more

Recommended publications
  • Management of Large Sets of Image Data Capture, Databases, Image Processing, Storage, Visualization Karol Kozak
    Management of large sets of image data Capture, Databases, Image Processing, Storage, Visualization Karol Kozak Download free books at Karol Kozak Management of large sets of image data Capture, Databases, Image Processing, Storage, Visualization Download free eBooks at bookboon.com 2 Management of large sets of image data: Capture, Databases, Image Processing, Storage, Visualization 1st edition © 2014 Karol Kozak & bookboon.com ISBN 978-87-403-0726-9 Download free eBooks at bookboon.com 3 Management of large sets of image data Contents Contents 1 Digital image 6 2 History of digital imaging 10 3 Amount of produced images – is it danger? 18 4 Digital image and privacy 20 5 Digital cameras 27 5.1 Methods of image capture 31 6 Image formats 33 7 Image Metadata – data about data 39 8 Interactive visualization (IV) 44 9 Basic of image processing 49 Download free eBooks at bookboon.com 4 Click on the ad to read more Management of large sets of image data Contents 10 Image Processing software 62 11 Image management and image databases 79 12 Operating system (os) and images 97 13 Graphics processing unit (GPU) 100 14 Storage and archive 101 15 Images in different disciplines 109 15.1 Microscopy 109 360° 15.2 Medical imaging 114 15.3 Astronomical images 117 15.4 Industrial imaging 360° 118 thinking. 16 Selection of best digital images 120 References: thinking. 124 360° thinking . 360° thinking. Discover the truth at www.deloitte.ca/careers Discover the truth at www.deloitte.ca/careers © Deloitte & Touche LLP and affiliated entities. Discover the truth at www.deloitte.ca/careers © Deloitte & Touche LLP and affiliated entities.
    [Show full text]
  • Ironpython in Action
    IronPytho IN ACTION Michael J. Foord Christian Muirhead FOREWORD BY JIM HUGUNIN MANNING IronPython in Action Download at Boykma.Com Licensed to Deborah Christiansen <[email protected]> Download at Boykma.Com Licensed to Deborah Christiansen <[email protected]> IronPython in Action MICHAEL J. FOORD CHRISTIAN MUIRHEAD MANNING Greenwich (74° w. long.) Download at Boykma.Com Licensed to Deborah Christiansen <[email protected]> For online information and ordering of this and other Manning books, please visit www.manning.com. The publisher offers discounts on this book when ordered in quantity. For more information, please contact Special Sales Department Manning Publications Co. Sound View Court 3B fax: (609) 877-8256 Greenwich, CT 06830 email: [email protected] ©2009 by Manning Publications Co. All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by means electronic, mechanical, photocopying, or otherwise, without prior written permission of the publisher. Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in the book, and Manning Publications was aware of a trademark claim, the designations have been printed in initial caps or all caps. Recognizing the importance of preserving what has been written, it is Manning’s policy to have the books we publish printed on acid-free paper, and we exert our best efforts to that end. Recognizing also our responsibility to conserve the resources of our planet, Manning books are printed on paper that is at least 15% recycled and processed without the use of elemental chlorine.
    [Show full text]
  • Modern Programming Languages CS508 Virtual University of Pakistan
    Modern Programming Languages (CS508) VU Modern Programming Languages CS508 Virtual University of Pakistan Leaders in Education Technology 1 © Copyright Virtual University of Pakistan Modern Programming Languages (CS508) VU TABLE of CONTENTS Course Objectives...........................................................................................................................4 Introduction and Historical Background (Lecture 1-8)..............................................................5 Language Evaluation Criterion.....................................................................................................6 Language Evaluation Criterion...................................................................................................15 An Introduction to SNOBOL (Lecture 9-12).............................................................................32 Ada Programming Language: An Introduction (Lecture 13-17).............................................45 LISP Programming Language: An Introduction (Lecture 18-21)...........................................63 PROLOG - Programming in Logic (Lecture 22-26) .................................................................77 Java Programming Language (Lecture 27-30)..........................................................................92 C# Programming Language (Lecture 31-34) ...........................................................................111 PHP – Personal Home Page PHP: Hypertext Preprocessor (Lecture 35-37)........................129 Modern Programming Languages-JavaScript
    [Show full text]
  • TANGO Device
    School on TANGO Controls system Basics of TANGO Lorenzo Pivetta Claudio Scafuri Graziano Scalamera http://www.tango-controls.org L.Pivetta, C.Scafuri, G.Scalamera School on TANGO Control System - Trieste 4-8th July 2016 2 Prerequisites To better understand the training a background on the following arguments is desirable: ● Programming language ● Object oriented programming ● Linux/UNIX operating system ● Networking ● Control systems L.Pivetta, C.Scafuri, G.Scalamera School on TANGO Control System - Trieste 4-8th July 2016 3 Outline 1 - What is TANGO? 2 - TANGO architecture Language/OS/Compilers Device hierarchy CORBA and ZeroMQ TANGO domains TANGO device and device server TANGO Database Communication models 3 - TANGO configuration/tools Multicast Jive Polling Starter/Astor Events Pogo Alarms TANGO installation Groups Client basics TANGO ACL Logging system Historical DataBase 4 – Examples Test device L.Pivetta, C.Scafuri, G.Scalamera School on TANGO Control System - Trieste 4-8th July 2016 4 What is TANGO? Scientific workspaces Native client applications In short: Industrial SCADA Control system framework TANGO Based on CORBA and ZMQ C++ TANGO Java Archiving Python System Centralized config. database TANGO TANGO TANGO TANGO Clients binding binding binding binding (CLI/GUI) Software bus for distributed TANGO software bus objects Provides unified interface to Device Device Device Device Device Device Device Server Server Server Server Server Server Server all equipments hiding how they are HV ps + Pylon OPC UA Data Motion TANGO SNMP connected/managed
    [Show full text]
  • A Practical Guide for Improving Transparency and Reproducibility in Neuroimaging Research Krzysztof J
    bioRxiv preprint first posted online Feb. 12, 2016; doi: http://dx.doi.org/10.1101/039354. The copyright holder for this preprint (which was not peer-reviewed) is the author/funder. It is made available under a CC-BY 4.0 International license. A practical guide for improving transparency and reproducibility in neuroimaging research Krzysztof J. Gorgolewski and Russell A. Poldrack Department of Psychology, Stanford University Abstract Recent years have seen an increase in alarming signals regarding the lack of replicability in neuroscience, psychology, and other related fields. To avoid a widespread crisis in neuroimaging research and consequent loss of credibility in the public eye, we need to improve how we do science. This article aims to be a practical guide for researchers at any stage of their careers that will help them make their research more reproducible and transparent while minimizing the additional effort that this might require. The guide covers three major topics in open science (data, code, and publications) and offers practical advice as well as highlighting advantages of adopting more open research practices that go beyond improved transparency and reproducibility. Introduction The question of how the brain creates the mind has captivated humankind for thousands of years. With recent advances in human in vivo brain imaging, we how have effective tools to ​ ​ peek into biological underpinnings of mind and behavior. Even though we are no longer constrained just to philosophical thought experiments and behavioral observations (which undoubtedly are extremely useful), the question at hand has not gotten any easier. These powerful new tools have largely demonstrated just how complex the biological bases of behavior actually are.
    [Show full text]
  • Python Programming
    Python Programming Wikibooks.org June 22, 2012 On the 28th of April 2012 the contents of the English as well as German Wikibooks and Wikipedia projects were licensed under Creative Commons Attribution-ShareAlike 3.0 Unported license. An URI to this license is given in the list of figures on page 149. If this document is a derived work from the contents of one of these projects and the content was still licensed by the project under this license at the time of derivation this document has to be licensed under the same, a similar or a compatible license, as stated in section 4b of the license. The list of contributors is included in chapter Contributors on page 143. The licenses GPL, LGPL and GFDL are included in chapter Licenses on page 153, since this book and/or parts of it may or may not be licensed under one or more of these licenses, and thus require inclusion of these licenses. The licenses of the figures are given in the list of figures on page 149. This PDF was generated by the LATEX typesetting software. The LATEX source code is included as an attachment (source.7z.txt) in this PDF file. To extract the source from the PDF file, we recommend the use of http://www.pdflabs.com/tools/pdftk-the-pdf-toolkit/ utility or clicking the paper clip attachment symbol on the lower left of your PDF Viewer, selecting Save Attachment. After extracting it from the PDF file you have to rename it to source.7z. To uncompress the resulting archive we recommend the use of http://www.7-zip.org/.
    [Show full text]
  • 4 Hash Tables and Associative Arrays
    4 FREE Hash Tables and Associative Arrays If you want to get a book from the central library of the University of Karlsruhe, you have to order the book in advance. The library personnel fetch the book from the stacks and deliver it to a room with 100 shelves. You find your book on a shelf numbered with the last two digits of your library card. Why the last digits and not the leading digits? Probably because this distributes the books more evenly among the shelves. The library cards are numbered consecutively as students sign up, and the University of Karlsruhe was founded in 1825. Therefore, the students enrolled at the same time are likely to have the same leading digits in their card number, and only a few shelves would be in use if the leadingCOPY digits were used. The subject of this chapter is the robust and efficient implementation of the above “delivery shelf data structure”. In computer science, this data structure is known as a hash1 table. Hash tables are one implementation of associative arrays, or dictio- naries. The other implementation is the tree data structures which we shall study in Chap. 7. An associative array is an array with a potentially infinite or at least very large index set, out of which only a small number of indices are actually in use. For example, the potential indices may be all strings, and the indices in use may be all identifiers used in a particular C++ program.Or the potential indices may be all ways of placing chess pieces on a chess board, and the indices in use may be the place- ments required in the analysis of a particular game.
    [Show full text]
  • Jupyter Tutorial Release 0.8.0
    Jupyter Tutorial Release 0.8.0 Veit Schiele Oct 01, 2021 CONTENTS 1 Introduction 3 1.1 Status...................................................3 1.2 Target group...............................................3 1.3 Structure of the Jupyter tutorial.....................................3 1.4 Why Jupyter?...............................................4 1.5 Jupyter infrastructure...........................................4 2 First steps 5 2.1 Install Jupyter Notebook.........................................5 2.2 Create notebook.............................................7 2.3 Example................................................. 10 2.4 Installation................................................ 13 2.5 Follow us................................................. 15 2.6 Pull-Requests............................................... 15 3 Workspace 17 3.1 IPython.................................................. 17 3.2 Jupyter.................................................. 50 4 Read, persist and provide data 143 4.1 Open data................................................. 143 4.2 Serialisation formats........................................... 144 4.3 Requests................................................. 154 4.4 BeautifulSoup.............................................. 159 4.5 Intake................................................... 160 4.6 PostgreSQL................................................ 174 4.7 NoSQL databases............................................ 199 4.8 Application Programming Interface (API)..............................
    [Show full text]
  • Automated Regional Behavioral Analysis for Human Brain Images
    METHODS ARTICLE published: 28 August 2012 NEUROINFORMATICS doi: 10.3389/fninf.2012.00023 Automated regional behavioral analysis for human brain images Jack L. Lancaster 1*, Angela R. Laird 1, Simon B. Eickhoff 2,3, Michael J. Martinez 1, P. M i c k l e F o x 1 and Peter T. Fox 1 1 Research Imaging Institute, The University of Texas Health Science Center at San Antonio, San Antonio, TX, USA 2 Institute of Neuroscience and Medicine (INM-1), Research Center, Jülich, Germany 3 Institute for Clinical Neuroscience and Medical Psychology, Heinrich-Heine University, Düsseldorf, Germany Edited by: Behavioral categories of functional imaging experiments along with standardized brain Robert W. Williams, University of coordinates of associated activations were used to develop a method to automate regional Tennessee Health Science Center, behavioral analysis of human brain images. Behavioral and coordinate data were taken USA from the BrainMap database (http://www.brainmap.org/), which documents over 20 years Reviewed by: Glenn D. Rosen, Beth Israel of published functional brain imaging studies. A brain region of interest (ROI) for behavioral Deaconess Medical Center, USA analysis can be defined in functional images, anatomical images or brain atlases, if Khyobeni Mozhui, University of images are spatially normalized to MNI or Talairach standards. Results of behavioral Tennessee Health Science Center, analysis are presented for each of BrainMap’s 51 behavioral sub-domains spanning five USA behavioral domains (Action, Cognition, Emotion, Interoception, and Perception). For each *Correspondence: Jack L. Lancaster, Research Imaging behavioral sub-domain the fraction of coordinates falling within the ROI was computed Institute, The University of Texas and compared with the fraction expected if coordinates for the behavior were not Health Science Center at San clustered, i.e., uniformly distributed.
    [Show full text]
  • JS and the DOM
    Back to the client side: JS and the DOM 1 Introduzione alla programmazione web – Marco Ronchetti 2020 – Università di Trento Q What are the DOM and the BOM? 2 Introduzione alla programmazione web – Marco Ronchetti 2020 – Università di Trento JS and the DOM When a web page is loaded, the browser creates a Document Object Model of the page, which is as a tree of Objects. Every element in a document—the document as a whole, the head, tables within the document, table headers, text within the table cells—is part of its DOM, so they can all be accessed and manipulated using the DOM and a scripting language like JavaScript. § With the object model, JavaScript can: § change all the HTML [elements, attributes, styles] in the page § add or remove existing HTML elements and attributes § react to HTML events in the page § create new HTML events in the page 3 Introduzione alla programmazione web – Marco Ronchetti 2020 – Università di Trento Using languages implementations of the DOM can be built for any language: e.g. Javascript Java Python … But Javascript is the only one that can work client-side. 4 Introduzione alla programmazione web – Marco Ronchetti 2020 – Università di Trento Q What are the fundamental objects in DOM/BOM? 5 Introduzione alla programmazione web – Marco Ronchetti 2020 – Università di Trento Fundamental datatypes - 1 § Node Every object located within a document is a node. In an HTML document, an object can be an element node but also a text node or attribute node. § Document (is-a Node) § the root document object itself.
    [Show full text]
  • Project 0: Implementing a Hash Table
    CS165: DATA SYSTEMS Project 0: Implementing a Hash Table CS 165, Data Systems, Fall 2019 Goal and Motivation. The goal of Project 0 is to help you develop (or refresh) basic skills at designing and implementing data structures and algorithms. These skills will be essential for following the class. Project 0 is meant to be done prior to the semester or during the early weeks. If you are doing it before the semester feel free to contact the staff for any help or questions by coming to the office hours or posting in Piazza. We expect that for most students Project 0 will take anything between a few days to a couple of weeks, depending in the student’s background on the above areas. If you are having serious trouble navigating Project 0 then you should reconsider the decision to take CS165. You can expect the semester project to be multiple orders of magnitude more work and more complex. How much extra work is this? Project 0 is actually designed as a part of the fourth milestone of the semester project. So after finishing Project 0 you will have refreshed some basic skills and you will have a part of your project as well. Basic Project Description. In many computer applications, it is often necessary to store a collection of key-value pairs. For example, consider a digital movie catalog. In this case, keys are movie names and values are their corresponding movie descriptions. Users of the application look up movie names and expect the program to fetch their corresponding descriptions quickly.
    [Show full text]
  • Mango Ripeness Stages Using Three Image Processing Algorithms
    RESEARCH ARTICLE Determining ‘Carabao’ Mango Ripeness Stages Using Three Image Processing Algorithms ٠ Mary Grace C. Galela ٠ Deborah S. Naciongayo ٠ Miguel Carlos S. Guillermo Cinmayii G. Manliguez ٠ Emma Ruth V. Bayogan ٠ Czaryl P. Dioquino University of the Philippines Mindanao, PHILIPPINES Abstract Harvested mangoes are commonly classified or sorted manually. This method is tedious, time consuming, inaccurate and prone to errors. Human inspection is also subjective and factors like visual stress and tiredness may arise that can result in the inconsistencies in judgment. The use of a chroma meter is reliable but the equipment is expensive. This study explored the use of three digital image processing algorithms to determine harvested ‘Carabao’ mango ripeness stages. Canny edge detection, Sobel edge detection, and Laplacian of Gaussian detection algorithms were used to extract a mango image from its original image. The mean red, green, and blue (RGB) values of the detected images were converted to L*a*b* color values which were used to identify the ripeness level of the mango image based on the standard generated from gathered data of ‘Carabao’ mangoes. The standard generated was also based on the mango peel color index scale from University of the Philippines Los Baños Postharvest Horticulture Training and Research Center (PHTRC). The algorithms’ performance had an overall accuracy of 80.5% for canny edge detection algorithm and L*a*b* color extraction using neural networks; 63.88% for Sobel edge detection algorithm and L*a*b* color extraction using rgb2lab function in MATLAB software; and 17.33% Laplacian of Gaussian detection and L*a*b* color extraction using OpenCV.
    [Show full text]