DOCSLIB.ORG

Information Theory for the Analysis of Large Spatio-Temporal Datasets

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Information Theory for the Analysis of Large Spatio-Temporal Datasets Information Theory for the Analysis of Large Spatio-Temporal Datasets Yannick Allard Michel Mayrand Dan Radulescu Prepared by: OODA Technologies Inc. 4580 Circle Rd. Montréal (Qc), H3W 1Y7 PWGSC Contract Number: W7707-145677 Technical Authority: Bruce McArthur, Defence Scientist The scientific or technical validity of this Contract Report is entirely the responsibility of the contractor and the contents do not necessarily have the approval or endorsement of the Department of National Defence of Canada. Contract Report DRDC-RDDC-2017-C051 February 2017 Template in use: (2010) SR Advanced Template_EN (051115).dotm © Her Majesty the Queen in Right of Canada, as represented by the Minister of National Defence, 2017 © Sa Majesté la Reine (en droit du Canada), telle que représentée par le ministre de la Défense nationale, 2017 Information Theory for the Analysis of Large Spatio-Temporal Datasets B316<=:=573A Yannick Allard Michel Mayrand Dan Radulescu Prepared By: OODA Technologies Inc. 4580 Circle Rd. Montr´eal(Qc), H3W 1Y7 514-476-4773 Prepared For: Defence Research & Development Canada, Atlantic Research Centre 9 Grove Street, PO Box 1012 Dartmouth, NS B2Y 3Z7 902-426-3100 ext.359 Scientific Authority: Bruce McArthur, Defence Scientist Contract Number: W7707-145677 Call Up Number: 14 Project: Vessel Traffic Analysis of Large Spatio-Temporal Datasets Report Delivery Date: February 17, 2017 The scientific or technical validity of this Contract Report is entirely the responsibility of the contractor and the contents do not necessarily have the approval or endorsement of the Department of National Defence of Canada. © Her Majesty the Queen in Right of Canada, as represented by the Minister of National Defence, 2017. © Sa Majest´ela Reine (en droit du Canada), telle que repr´esent´eepar le ministre de la D´efensenationale, 2017. This page is intentionally left blank. Executive Summary One important aspect of Maritime Domain Awareness (MDA) is the aggregation of data and information to help formulate an accurate description of vessel activity, with the Automatic Iden- tification System (AIS) being a key data source. However, the introduction of AIS, together with other sources of MDA-related data, has also posed a challenge, in that there is an overabundance of data, making manual analysis prohibitively time-consuming. In response, there has been a growing interest in the application of techniques for automated or semi-automated analysis of vessel traffic from large volume datasets. These techniques draw from research areas such as: data mining { the computational process of discovering patterns in large data sets; spatio-temporal trajectory anal- ysis; information theory techniques and visual analytics, analytical reasoning supported by highly interactive visual interfaces. All of these approaches have been used in support of MDA-related capabilities, which include anomaly detection, traffic route extraction, area of interest analysis, collision risk analysis, vessels of interest analysis, vessel tracking and data fusion. In this study, we focus our investigation on the area of information theory, which has so far seen little overlap with the maritime domain, despite having significant potential. In this report, we present standard measures and techniques in information theory, and its appli- cation in large spatio-temporal datasets, in particular, maritime AIS datasets. By its very nature, information theory should be able to handle problems specific to AIS data, as well as problems that arise when performing spatio-temporal data mining based on the conducted literature survey and current challenges in MDA. A demonstration of a possible application over a one-month dataset has been implemented. The entropy and spatial diversity measures were applied as local measures at multiple scales and on selected attributes within Canada's Exclusive Economic Zone. Its potential was assessed through visual inspection. It was highlighted that transit zones, or shipping routes, exhibit fairly stable characteristics resulting in lower entropic and spatial diversity intensity compared to areas where multiple activities occur. Temporal behaviour was not investigated, however, results suggest that the implementation of spatio-temporal scale selection process should lead to the extraction of meaningful maritime pro- cesses over large datasets. Where processing is concerned, one should consider implementing an information-theoretic measure within a framework designed for large-scale cloud computing to speed up information extraction, even in an exploratory study. In conclusion, it was shown that information-theoretic measures have the potential to be used over large maritime datasets in a data mining context, and that this potential remains largely untapped based on the available literature. i OODA Technologies Inc. The use or disclosure of the information on this sheet is subject to the restrictions on the title page of this document. Study Report RISOMIA Call-up 14 This page is intentionally left blank. ii The use or disclosure of the information on this sheet is subject to the restrictions on the title page of this document. Contents Executive Summary i Contents iii List of Figures vii List of Tables ix 1 Introduction 1 2 Methodology 3 2.1 Summary of the Literature Survey . 3 2.2 Summary of Search Terms . 4 3 AIS Data 7 3.1 AIS Data and Information Product Sources . 7 3.2 AIS Capabilities and MDA Usage . 8 3.3 Problems with AIS Data . 9 3.4 Other Maritime Data Sources . 11 3.4.1 Earth Observation Data and Information Product Sources . 11 3.4.2 Earth Observation Capabilities and MDA Usage . 12 3.4.3 Other Public Information Products . 12 3.5 Spatio-Temporal and AIS Data Mining . 14 iii Study Report RISOMIA Call-up 14 4 Information Theory Basic Concepts 17 4.1 Information-Theoretic Measures . 17 4.1.1 Entropy . 17 4.1.1.1 Shannon Entropy . 17 4.1.1.2 R´enyi Entropy . 18 4.1.2 Conditional Entropy . 18 4.1.3 Mutual Information . 19 4.1.4 Kullback-Leibler Divergence . 19 4.1.5 Self-Information or Surprisal . 19 4.2 Spatio-Temporal Extension of Information Theory . 20 4.2.1 Co-occurrence-based Spatial Entropy . 20 4.2.2 Distance Ratios Spatial Diversity . 21 4.2.3 Markov Random Field-based Spatial Entropy . 21 4.3 Summary . 22 5 Applications of Information Theory 25 5.1 Maritime Domain Applications . 25 5.1.1 Pattern Discovery in Maritime Data . 25 5.1.2 Measures of Diversity Based on Distance and Co-occurrence . 26 5.1.3 Vessel Imaging . 27 5.1.4 Feature Extraction and Categorization . 28 5.1.5 Vessel Report Quality Assessment . 28 5.2 Non-Maritime Domain Applications . 28 5.2.1 Psychology . 28 5.2.2 Image Processing . 29 5.2.3 Information Theory for KDD . 31 5.2.4 Decision Trees . 31 5.2.5 Entropy Based Time Series Analysis in Biomedical Applications . 33 iv The use or disclosure of the information on this sheet is subject to the restrictions on the title page of this document. CONTENTS 5.3 Application in Visualization and Interactive Analysis . 33 5.4 Summary . 39 6 Proof-of-Concept Demonstration 41 6.1 Potential Applications of Information Theory for Maritime Domain Awareness . 42 6.1.1 Global and Local Entropy Computation . 43 6.1.2 Co-occurrence-based Spatial Entropy for Pattern Extraction . 44 6.1.3 Spatial Diversity Applied to Vessel/Data Attributes Using Distance Ratios 45 6.1.4 Other Avenues and Considerations . 45 6.1.5 Space-time Scale Selection for Spatio-Temporal Analysis . 46 6.1.5.1 Multigrid Methods . 46 6.1.5.2 Space-time Permutation Scan Statistic . 47 6.1.5.3 Scale-Space Analysis . 48 6.1.5.4 Number of Empty Grid Cells for Spatial Entropy . 48 6.1.5.5 Entropy-based Scale Saliency . 49 6.1.6 Summary of the Proposed Proof-of-Concept Demonstration . 51 6.2 Application of Information-Theoretic Measures over a Large Maritime Dataset . 51 6.2.1 Description of the Dataset . 52 6.2.2 Global Entropy of the Dataset . 52 6.2.3 Multiscale Analysis and Measures Comparison . 54 7 Conclusion 79 Bibliography 81 Appendix A Conferences and books A-1 A.1 Conferences . A-1 A.2 Books . A-1 Appendix B Administration and User Guide B-3 v OODA Technologies Inc. The use or disclosure of the information on this sheet is subject to the restrictions on the title page of this document. Study Report RISOMIA Call-up 14 B.1 Administration guide . B-4 B.1.1 Requirements . B-4 B.1.2 Quick Installation Guide Using Docker . B-4 B.2 User guide . B-6 B.2.1 Creating a MSARI database from raw AIS compressed files . B-6 B.2.2 Java script . B-8 B.2.3 SQL Scripts . B-10 B.2.4 Bash Script . B-11 vi The use or disclosure of the information on this sheet is subject to the restrictions on the title page of this document. List of Figures 4.1 Venn diagram representing the different entropies and the mutual information. 19 4.2 Two neighborhoods of the same equivalence class (4-2-1-1), as the actual values of the neighbours are not considered, only their numbers. Source: reference [76]. 22 6.1 Multi-level application of information theory for analysis of large spatio-temporal datasets . 42 6.2 A multigrid Example . 47 6.3 Entropy of the course over ground attributes over multiple scales . 55 6.4 Entropy of the speed attribute over multiple scales . 56 6.5 Entropy of the ship type attribute over multiple scales . 57 6.6 Spatial diversity of the course over ground attributes over multiple scales . 58 6.7 Spatial diversity of the speed attribute over multiple scales . 59 6.8 Spatial diversity of the ship type attribute over multiple scales . 60 6.9 Entropy and spatial diversity of the course over ground attributes at 0.5 and 0.1 degrees .
Load more

Recommended publications
  • Package 'Infotheo'
    Package ‘infotheo’ February 20, 2015 Title Information-Theoretic Measures Version 1.2.0 Date 2014-07 Publication 2009-08-14 Author Patrick E. Meyer Description This package implements various measures of information theory based on several en- tropy estimators. Maintainer Patrick E. Meyer <[email protected]> License GPL (>= 3) URL http://homepage.meyerp.com/software Repository CRAN NeedsCompilation yes Date/Publication 2014-07-26 08:08:09 R topics documented: condentropy . .2 condinformation . .3 discretize . .4 entropy . .5 infotheo . .6 interinformation . .7 multiinformation . .8 mutinformation . .9 natstobits . 10 Index 12 1 2 condentropy condentropy conditional entropy computation Description condentropy takes two random vectors, X and Y, as input and returns the conditional entropy, H(X|Y), in nats (base e), according to the entropy estimator method. If Y is not supplied the function returns the entropy of X - see entropy. Usage condentropy(X, Y=NULL, method="emp") Arguments X data.frame denoting a random variable or random vector where columns contain variables/features and rows contain outcomes/samples. Y data.frame denoting a conditioning random variable or random vector where columns contain variables/features and rows contain outcomes/samples. method The name of the entropy estimator. The package implements four estimators : "emp", "mm", "shrink", "sg" (default:"emp") - see details. These estimators require discrete data values - see discretize. Details • "emp" : This estimator computes the entropy of the empirical probability distribution. • "mm" : This is the Miller-Madow asymptotic bias corrected empirical estimator. • "shrink" : This is a shrinkage estimate of the entropy of a Dirichlet probability distribution. • "sg" : This is the Schurmann-Grassberger estimate of the entropy of a Dirichlet probability distribution.
    [Show full text]
  • A Flow-Based Entropy Characterization of a Nated Network and Its Application on Intrusion Detection
    A Flow-based Entropy Characterization of a NATed Network and its Application on Intrusion Detection J. Crichigno1, E. Kfoury1, E. Bou-Harb2, N. Ghani3, Y. Prieto4, C. Vega4, J. Pezoa4, C. Huang1, D. Torres5 1Integrated Information Technology Department, University of South Carolina, Columbia (SC), USA 2Cyber Threat Intelligence Laboratory, Florida Atlantic University, Boca Raton (FL), USA 3Electrical Engineering Department, University of South Florida, Tampa (FL), USA 4Electrical Engineering Department, Universidad de Concepcion, Concepcion, Chile 5Department of Mathematics, Northern New Mexico College, Espanola (NM), USA Abstract—This paper presents a flow-based entropy charac- information is collected by the device and then exported for terization of a small/medium-sized campus network that uses storage and analysis. Thus, the performance impact is minimal network address translation (NAT). Although most networks and no additional capturing devices are needed [3]-[5]. follow this configuration, their entropy characterization has not been previously studied. Measurements from a production Entropy has been used in the past to detect anomalies, network show that the entropies of flow elements (external without requiring payload inspection. Its use is appealing IP address, external port, campus IP address, campus port) because it provides more information on flow elements (ports, and tuples have particular characteristics. Findings include: i) addresses, tuples) than traffic volume analysis. Entropy has entropies may widely vary in the course of a day. For example, also been used for anomaly detection in backbones and large in a typical weekday, the entropies of the campus and external ports may vary from below 0.2 to above 0.8 (in a normalized networks.
    [Show full text]
  • Bits, Bans Y Nats: Unidades De Medida De Cantidad De Información
    Bits, bans y nats: Unidades de medida de cantidad de información Junio de 2017 Apellidos, Nombre: Flores Asenjo, Santiago J. (sfl[email protected]) Departamento: Dep. de Comunicaciones Centro: EPS de Gandia 1 Introducción: bits Resumen Todo el mundo conoce el bit como unidad de medida de can- tidad de información, pero pocos utilizan otras unidades también válidas tanto para esta magnitud como para la entropía en el ám- bito de la Teoría de la Información. En este artículo se presentará el nat y el ban (también deno- minado hartley), y se relacionarán con el bit. Objetivos y conocimientos previos Los objetivos de aprendizaje este artículo docente se presentan en la tabla 1. Aunque se trata de un artículo divulgativo, cualquier conocimiento sobre Teo- ría de la Información que tenga el lector le será útil para asimilar mejor el contenido. 1. Mostrar unidades alternativas al bit para medir la cantidad de información y la entropía 2. Presentar las unidades ban (o hartley) y nat 3. Contextualizar cada unidad con la historia de la Teoría de la Información 4. Utilizar la Wikipedia como herramienta de referencia 5. Animar a la edición de la Wikipedia, completando los artículos en español, una vez aprendidos los conceptos presentados y su contexto histórico Tabla 1: Objetivos de aprendizaje del artículo docente 1 Introducción: bits El bit es utilizado de forma universal como unidad de medida básica de canti- dad de información. Su nombre proviene de la composición de las dos palabras binary digit (dígito binario, en inglés) por ser históricamente utilizado para denominar cada uno de los dos posibles valores binarios utilizados en compu- tación (0 y 1).
    [Show full text]
  • The Indefinite Logarithm, Logarithmic Units, and the Nature of Entropy
    The Indefinite Logarithm, Logarithmic Units, and the Nature of Entropy Michael P. Frank FAMU-FSU College of Engineering Dept. of Electrical & Computer Engineering 2525 Pottsdamer St., Rm. 341 Tallahassee, FL 32310 [email protected] August 20, 2017 Abstract We define the indefinite logarithm [log x] of a real number x> 0 to be a mathematical object representing the abstract concept of the logarithm of x with an indeterminate base (i.e., not specifically e, 10, 2, or any fixed number). The resulting indefinite logarithmic quantities naturally play a mathematical role that is closely analogous to that of dimensional physi- cal quantities (such as length) in that, although these quantities have no definite interpretation as ordinary numbers, nevertheless the ratio of two of these entities is naturally well-defined as a specific, ordinary number, just like the ratio of two lengths. As a result, indefinite logarithm objects can serve as the basis for logarithmic spaces, which are natural systems of logarithmic units suitable for measuring any quantity defined on a log- arithmic scale. We illustrate how logarithmic units provide a convenient language for explaining the complete conceptual unification of the dis- parate systems of units that are presently used for a variety of quantities that are conventionally considered distinct, such as, in particular, physical entropy and information-theoretic entropy. 1 Introduction The goal of this paper is to help clear up what is perceived to be a widespread arXiv:physics/0506128v1 [physics.gen-ph] 15 Jun 2005 confusion that can found in many popular sources (websites, popular books, etc.) regarding the proper mathematical status of a variety of physical quantities that are conventionally defined on logarithmic scales.
    [Show full text]
  • Quantum Information Chapter 10. Quantum Shannon Theory
    Quantum Information Chapter 10. Quantum Shannon Theory John Preskill Institute for Quantum Information and Matter California Institute of Technology Updated June 2016 For further updates and additional chapters, see: http://www.theory.caltech.edu/people/preskill/ph219/ Please send corrections to [email protected] Contents 10 Quantum Shannon Theory 1 10.1 Shannon for Dummies 2 10.1.1 Shannon entropy and data compression 2 10.1.2 Joint typicality, conditional entropy, and mutual infor- mation 6 10.1.3 Distributed source coding 8 10.1.4 The noisy channel coding theorem 9 10.2 Von Neumann Entropy 16 10.2.1 Mathematical properties of H(ρ) 18 10.2.2 Mixing, measurement, and entropy 20 10.2.3 Strong subadditivity 21 10.2.4 Monotonicity of mutual information 23 10.2.5 Entropy and thermodynamics 24 10.2.6 Bekenstein’s entropy bound. 26 10.2.7 Entropic uncertainty relations 27 10.3 Quantum Source Coding 30 10.3.1 Quantum compression: an example 31 10.3.2 Schumacher compression in general 34 10.4 Entanglement Concentration and Dilution 38 10.5 Quantifying Mixed-State Entanglement 45 10.5.1 Asymptotic irreversibility under LOCC 45 10.5.2 Squashed entanglement 47 10.5.3 Entanglement monogamy 48 10.6 Accessible Information 50 10.6.1 How much can we learn from a measurement? 50 10.6.2 Holevo bound 51 10.6.3 Monotonicity of Holevo χ 53 10.6.4 Improved distinguishability through coding: an example 54 10.6.5 Classical capacity of a quantum channel 58 ii Contents iii 10.6.6 Entanglement-breaking channels 62 10.7 Quantum Channel Capacities and Decoupling
    [Show full text]
  • A Passive Photon–Atom Qubit Swap Operation
    LETTERS https://doi.org/10.1038/s41567-018-0241-6 A passive photon–atom qubit swap operation Orel Bechler1,3, Adrien Borne1,3, Serge Rosenblum1,3, Gabriel Guendelman1, Ori Ezrah Mor1, Moran Netser1, Tal Ohana1, Ziv Aqua1, Niv Drucker1, Ran Finkelstein1, Yulia Lovsky1, Rachel Bruch1, Doron Gurovich1, Ehud Shafir1,2 and Barak Dayan 1* Deterministic quantum interactions between single photons emitter to end up in one of its two ground states depending on the and single quantum emitters are a vital building block towards input mode of a single photon. Initially considered in ref. 25 and the distribution of quantum information between remote sys- analysed in a series of theoretical works15,26–28, this mechanism was tems1–4. Deterministic photon–atom state transfer has previ- first realized experimentally to create an all-optical single-photon ously been demonstrated with protocols that include active switch11, and to demonstrate deterministic extraction of a single feedback or synchronized control pulses5–10. Here we demon- photon from optical pulses13. The effect was also demonstrated in strate a passive swap operation between the states of a sin- superconducting circuits, where it enabled frequency conversion of gle photon and a single atom. The underlying mechanism is microwave fields12 and highly efficient detection of single micro- single-photon Raman interaction11–15—an interference-based wave photons14. scheme that leads to deterministic interaction between two The configuration that leads to SPRINT includes a Λ -system in photonic modes and the two ground states of a Λ -system. which each of its two transitions is coupled, with the same coopera- Using a nanofibre-coupled microsphere resonator coupled to tivity C, to a different mode of an optical waveguide.
    [Show full text]
  • Passive NAT Detection Using HTTP Logs
    master’s thesis Passive NAT detection using HTTP logs Tomáš Komárek May 2015 Supervisor: Ing. Martin Grill Czech Technical University in Prague Faculty of Electrical Engineering Department of Control Engineering Acknowledgment First and foremost, I would like to acknowledge my supervisor Ing. Martin Grill for his interest, help and guidance throughout my diploma thesis. In addition, I would like to thank Ing. Tomáš Pevný, Ph.D. for his helpful suggestions and comments during this work. Special thanks go to all the people at CISCO, for providing a nice and friendly working environment. Last but not least, I would like to thank my girlfriend Teraza and my brother Lukáš who inspire me all the time. Declaration I declare that I worked out the presented thesis independently and I quoted all used sources of information in accord with Methodical instructions about ethical principles for writing academic thesis. Abstract Network devices performing NAT prove to be a double edge sword. They can easily overcome the problem with the deficit of IPv4 addresses as well as introduce a vulnerability to the network. Therefore detecting NAT devices is an important task in the network security domain. In this thesis, a novel passive NAT detection algorithm is proposed. It infers NAT devices in the networks using statistical behavior analysis of HTTP logs. These network traffic data are often already collected and available at proxy servers, which enables the wide applicability of the solution. On the basis of our experimen- tal evaluations, proposed algorithm detection capabilities are better than the state-of-the art NAT detection approaches.
    [Show full text]
  • Information and Thermodynamics
    The Second Law and Informatics Oded Kafri Varicom Communications, Tel Aviv 68165 Israel. Abstract A unification of thermodynamics and information theory is proposed. It is argued that similarly to the randomness due to collisions in thermal systems, the quenched randomness that exists in data files in informatics systems contributes to entropy. Therefore, it is possible to define equilibrium and to calculate temperature for informatics systems. The obtained temperature yields correctly the Shannon information balance in informatics systems and is consistent with the Clausius inequality and the Carnot cycle. 1 Introduction Heat is the energy transferred from one body to another. The second law of thermodynamics gives us universal tools to determine the direction of the heat flow. A process is likely to happen if at its end the entropy increases. Similarly, energy distribution of particles will evolve to a distribution that maximizes the Boltzmann -H function [1] namely, an equilibrium state, where entropy and temperature are well defined. Information technology (IT) is governed by energy flow. Processes like data transmission, registration and manipulation are all energy consuming. It is accepted that energy flow in computers and IT are subject to the same physical laws as in heat engines or chemical reactions. Nevertheless, no consistent thermodynamic theory for IT was proposed. Hereafter, a thermodynamic theory of communication is considered. The discussion starts by drawing a thermodynamic analogy between a spontaneous heat flow from a hot body to a cold one and energy flow from a broadcasting antenna to receiving antennas. This analogy may look quiet natural. When a file is transmitted from a transmitter to the receivers, the transmitted file's energy, thermodynamically speaking, is heat.
    [Show full text]
  • Objective Functions for Information-Theoretical Monitoring Network Design: What Is “Optimal”?
    Hydrol. Earth Syst. Sci., 25, 831–850, 2021 https://doi.org/10.5194/hess-25-831-2021 © Author(s) 2021. This work is distributed under the Creative Commons Attribution 4.0 License. Objective functions for information-theoretical monitoring network design: what is “optimal”? Hossein Foroozand and Steven V. Weijs Department of Civil Engineering, University of British Columbia, Vancouver, British Columbia, Canada Correspondence: Steven V. Weijs ([email protected]) Received: 1 April 2020 – Discussion started: 17 April 2020 Revised: 26 November 2020 – Accepted: 7 December 2020 – Published: 19 February 2021 Abstract. This paper concerns the problem of optimal mon- sensors, to identify optimal subsets. Jointly, these papers (Al- itoring network layout using information-theoretical meth- fonso et al., 2010a,b; Li et al., 2012; Ridolfi et al., 2011; ods. Numerous different objectives based on information Samuel et al., 2013; Stosic et al., 2017; Keum and Coulibaly, measures have been proposed in recent literature, often fo- 2017; Banik et al., 2017; Wang et al., 2018; Huang et al., cusing simultaneously on maximum information and mini- 2020; Khorshidi et al., 2020) have proposed a wide variety of mum dependence between the chosen locations for data col- different optimization objectives. Some have suggested that lection stations. We discuss these objective functions and either a multi-objective approach or a single objective de- conclude that a single-objective optimization of joint entropy rived from multiple objectives is necessary to find an optimal suffices to maximize the collection of information for a given monitoring network. These methods were often compared to number of stations.
    [Show full text]
  • The Use of Information Theory in Evolutionary Biology Christoph
    The Use of Information Theory in Evolutionary Biology Christoph Adami1,2,3 1Department of Microbiology and Molecular Genetics 2Department of Physics and Astronomy 3BEACON Center for the Study of Evolution in Action Michigan State University, East Lansing, MI 48824 Abstract Information is a key concept in evolutionary biology. Information is stored in biological organism’s genomes, and used to generate the organism as well as to maintain and control it. Information is also “that which evolves”. When a population adapts to a local environment, information about this environment is fixed in a representative genome. However, when an environment changes, information can be lost. At the same time, information is processed by animal brains to survive in complex environments, and the capacity for information processing also evolves. Here I review applications of information theory to the evolution of proteins as well as to the evolution of information processing in simulated agents that adapt to perform a complex task. 1. Introduction Evolutionary biology has traditionally been a science that used observation and the analysis of specimens to draw inferences about common descent, adaptation, variation, and selection1,2. In contrast to this discipline that requires fieldwork and meticulous attention to detail stands the mathematical theory of population genetics3,4, which developed in parallel but somewhat removed from evolutionary biology, as it could treat exactly only very abstract cases. The mathematical theory cast Darwin’s insight about inheritance, variation, and selection into formulae that could predict particular aspects of the evolutionary process, such as the probability that an allele that conferred a particular advantage would go to fixation, how long this process would take, and how it would be modified by different forms of inheritance.
    [Show full text]
  • Information Theory in Neuroscience
    entropy Editorial Information Theory in Neuroscience Eugenio Piasini 1,* and Stefano Panzeri 2,* 1 Computational Neuroscience Initiative and Department of Physics and Astronomy, University of Pennsylvania, Philadelphia, PA 19104, USA 2 Neural Computation Laboratory, Center for Neuroscience and Cognitive Systems @UniTn, Istituto Italiano di Tecnologia, 38068 Rovereto (TN), Italy * Correspondence: [email protected] (E.P.); [email protected] (S.P.) Received: 26 December 2018; Accepted: 9 January 2019; Published: 14 January 2019 Abstract: This is the Editorial article summarizing the scope and contents of the Special Issue, Information Theory in Neuroscience. Keywords: information theory; neuroscience As the ultimate information processing device, the brain naturally lends itself to be studied with information theory. Because of this, information theory [1] has been applied to the study of the brain systematically for many decades and has been instrumental in many advances. It has spurred the development of principled theories of brain function [2–8]. It has led to advances in the study of consciousness [9]. It has also led to the development of many influential neural recording analysis techniques to crack the neural code, that is to unveil the language used by neurons to encode and process information [10–15]. The influence of information theory on the study of neural information processing continues today in many ways. In particular, concepts from information theory are beginning to be applied to the large-scale recordings of
    [Show full text]
  • Information Theory and Coding
    Information Theory and Coding Subject Code : 10EC55 IA Marks : 25 No. of Lecture Hrs/Week : 04 Exam Hours : 03 Total no. of Lecture Hrs. : 52 Exam Marks : 100 PART - A Unit – 1: Information Theory: Introduction, Measure of information, Average information content of symbols in long independent sequences, Average information content of symbols in long dependent sequences. Mark-off statistical model for information source, Entropy and information rate of mark-off source. 6 Hours Unit – 2: Source Coding: Encoding of the source output, Shannon‟s encoding algorithm. Communication Channels, Discrete communication channels, Continuous channels. 6 Hours Unit – 3: Fundamental Limits on Performance: Source coding theorem, Huffman coding, Discrete memory less Channels, Mutual information, Channel Capacity. 6 Hours Unit – 4: Channel coding theorem, Differential entropy and mutual information for continuous ensembles, Channel capacity Theorem. 6 Hours PART - B Unit – 5: Introduction to Error Control Coding: Introduction, Types of errors, examples, Types of codes Linear Block Codes: Matrix description, Error detection and correction, Standard arrays and table look up for decoding. 7 Hours Unit – 6: Binary Cycle Codes, Algebraic structures of cyclic codes, Encoding using an (n-k) bit shift register, Syndrome calculation. BCH codes. 7 Hours Unit – 7: RS codes, Golay codes, Shortened cyclic codes, Burst error correcting codes. Burst and Random Error correcting codes. 7 Hours Unit – 8: Convolution Codes, Time domain approach. Transform domain approach. 7Hours Text Books: Digital and analog communication systems, K. Sam Shanmugam, John Wiley, 1996. Digital communication, Simon Haykin, John Wiley, 2003. Reference Books: ITC and Cryptography, Ranjan Bose, TMH, II edition, 2007 Digital Communications - Glover and Grant; Pearson Ed.
    [Show full text]