DOCSLIB.ORG

Projecto Em Engenharia Informatica

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Projecto Em Engenharia Informatica UNIVERSIDADE DE LISBOA Faculdade de Cienciasˆ Departamento de Informatica´ ANALISE´ DE CORPORA JORNALISTICO´ ATRAVES´ DAS OCORRENCIASˆ DE N-GRAMAS Joao˜ Antonio´ Monteiro de Oliveira PROJECTO MESTRADO EM ENGENHARIA INFORMATICA´ Especializac¸ao˜ em Sistemas de Informac¸ao˜ 2015 UNIVERSIDADE DE LISBOA Faculdade de Cienciasˆ Departamento de Informatica´ ANALISE´ DE CORPORA JORNALISTICO´ ATRAVES´ DAS OCORRENCIASˆ DE N-GRAMAS Joao˜ Antonio´ Monteiro de Oliveira PROJECTO Projecto orientado pelo Prof. Doutor Francisco Jose´ Moreira Couto e co-orientado pelo Prof. Doutor Bruno Emanuel da Grac¸a Martins MESTRADO EM ENGENHARIA INFORMATICA´ Especializac¸ao˜ em Sistemas de Informac¸ao˜ 2015 Agradecimentos Gostaria de comec¸ar por agradecer ao meu orientador, o Professor Francisco Couto e ao meu co-orientador Bruno Martins por todo o apoio prestado durante todo o processo da criac¸ao˜ do trabalho que realizei e da tese resultante deste. Igualmente, gostaria de agradecer ao Professor Mario´ Silva e a todos os membros do projecto REACTION do INESC-ID que ajudaram a guiar o meu trabalho e me abriram as portas a uma experienciaˆ academica´ que me enriqueceu, tanto ao n´ıvel academico´ como ao n´ıvel pessoal. Devo a minha gratidao˜ a` minha fam´ılia, ao meu pai em especial, que me ajudaram a ver a importanciaˆ deste projecto quando a minha motivac¸ao˜ ja´ nao˜ era a mesma e a concluir este. Adicionalmente, gostaria de agradecer aos meus colegas de curso e amigos por toda a ajuda e conv´ıvio que partilharam comigo. Finalmente, tambem´ gostaria de agradecer a` FCT (Fundac¸ao˜ para a Cienciaˆ e Tecnologia) pelo seu apoio financeiro. iii Aos meus pais, Teresa Maria e Antonio´ Valdemar, pelo apoio e motivac¸ao˜ Resumo O mundo jornal´ıstico mudou drasticamente com a transic¸ao˜ dos meios tradicionais de publicac¸ao˜ para o mundo digital. O aumento da informac¸ao˜ dispon´ıvel e o melhoramento dos meios de processamento desta facilitam a sua interpretac¸ao,˜ sendo assim poss´ıvel analisar a importanciaˆ de certos elementos informativos (memes) ao longo do tempo. O estudo destas tendenciasˆ permite avaliar as flutuac¸oes˜ de interesse dos memes, possibili- tando a previsao˜ de novos picos de popularidade e a correlac¸ao˜ destes com os eventos que estiveram na origem do referido interesse. Considerando esta realidade, o trabalho proposto nesta tese tem como objectivo a transformac¸ao˜ de um corpus jornal´ıstico em dados estat´ısticos que permitam a analise´ de tendenciasˆ ao longo de um per´ıodo de tempo. A soluc¸ao˜ proposta deve ter em conta o tamanho do cor- pus a ser processado e como este afecta o tempo de processamento. Em termos anal´ıticos, deve considerar-se a performance da soluc¸ao˜ e a validade dos resultados produzidos. Para responder aos objectivos propostos, construi a aplicac¸ao˜ Web Publico´ N-Gram Vi- ewer para a criac¸ao˜ de dados estat´ısticos e visualizac¸ao˜ de tendencaisˆ de N-Gramas. A analise´ da performance da soluc¸ao˜ tem como base o tempo de execuc¸ao˜ dos diversos algo- ritmos de criac¸ao˜ de N-Gramas para conjuntos de dados de tamanho variavel´ e o numero´ de maquinas´ usado para o processamento destes. Ao n´ıvel da validade dos resultados produzidos, sao˜ analisados exemplos onde se verifica a correlac¸ao˜ entre o numero´ de ocorrenciasˆ gerado e a ocorrenciaˆ de eventos que justifiquem o aumento de popularidade. Palavras-chave: Prospecc¸ao˜ de informac¸ao˜ em textos, Analise´ de N-Gramas, Indexac¸ao˜ de textos, jornalismo computacional, memes vii Abstract The news world has changed drastically with the transition from the tradicional pub- lishing methods to the digital world. The increase of available information, as well as the improvement of the methods to process that information, facilitate its interpretation, al- lowing the analysis of the importance of memes along a period of time. The study of these tendencies allows the analysis and prediction of the future occurrences of the memes. Considering this reality, the point of this thesis is the creation of a software solution for the transformation of a corpus into statistic data and the study of the resulting trends for the given time period. The program must have in consideration the size of the corpus and how it will affect its performance. In terms of analysis, besides software performance, it is also necessary to confirm the validity of the produced statistics. In order to hit the goals defined for this thesis, the Publico´ N-Gram Viewer Web ap- plication was built. The solution creates statistical data from corpus text files given as input and allows the search for N-Gram interest tendencies in a time interval. The per- formance analysis is based on the time necessary for the completion of the algorithms for the creation of the N-Grams, given inputs of variable size and the number of nodes used for processing. Regarding the consistency of the produced results, examples are consid- ered where the correlation between the number of occurrences for a given N-Gram match events that occurred that justify the increase in popularity. Keywords: Text data mining, N-gram analysis, full text search, computational journalism, memes ix Conteudo´ Lista de Figuras xv Lista de Tabelas xvii 1 Introduc¸ao˜ 1 1.1 Motivac¸ao..................................˜ 1 1.2 Objectivos . .2 1.3 Metodologia . .2 1.4 Estrutura do documento . .3 1.5 Conclusoes˜ do cap´ıtulo . .4 2 Trabalho relacionado 5 2.1 Back-End . .5 2.1.1 Finding and Exploring Memes in Social Media . .5 2.1.2 MapReduce: Simplified Data Processing on Large Clusters . .5 2.1.3 Computing n-Gram Statistics in MapReduce . .6 2.2 Interface . .7 2.2.1 Google NGram Viewer . .7 2.2.2 MemeTracker . .8 2.3 Conclusoes˜ do cap´ıtulo . .9 3 Analise´ 11 3.1 Arquitectura . 11 3.1.1 Camada de Dados . 12 3.1.2 Camada Logica´ . 13 3.1.3 Camada de Apresentac¸ao......................˜ 14 3.2 Conlusoes˜ do cap´ıtulo . 14 4 Implementac¸ao˜ 17 4.1 Back-End . 17 4.1.1 Configurac¸ao/Inicializac¸˜ ao.....................˜ 17 4.1.2 Tokenizador . 21 xi 4.1.3 MapReduce . 22 4.1.4 Algoritmo Principal . 23 4.1.5 Algoritmo de Categoria de Not´ıcias . 26 4.1.6 Algoritmo de Classificac¸ao˜ de Entidades . 26 4.1.7 Algoritmo de Classificac¸ao˜ Gramatical . 27 4.1.8 Construc¸ao˜ das Estruturas de Dados . 28 4.1.9 Implementac¸oes˜ descartadas . 29 4.2 Web Service . 30 4.3 Front-End . 35 4.3.1 Publico´ Online NGram Search . 36 4.3.2 Publico´ Online NGram Admin Client . 40 4.4 Principais Problemas Encontrados . 44 4.4.1 Actualizac¸ao˜ dos dados nas estruturas MapDB . 44 4.4.2 Criac¸ao˜ da aplicac¸ao˜ administrativa . 44 4.4.3 Criac¸ao˜ da estrutura de dados TreeMap na func¸ao˜ Reduce . 45 4.4.4 Implementac¸ao˜ do cluster Hadoop . 45 5 Resultados 47 5.1 MapReduce . 47 5.1.1 Algoritmo Base . 48 5.1.2 Algoritmo de Categorias de Not´ıcias . 49 5.1.3 Algoritmo de Classificac¸ao˜ de Entidades . 50 5.1.4 Algoritmo de Classificac¸ao˜ Gramatical . 52 5.2 Estat´ısticas . 52 5.2.1 Pesquisa regular . 52 5.2.2 Pesquisa por Categoria . 54 6 Conclusao˜ 57 6.1 Trabalho Futuro . 58 Bibliografia 60 Abreviaturas 61 A Anexos 63 Indice´ 71 xii xiv Lista de Figuras 2.1 Google Ngram Viewer . .7 2.2 MemeTracker . .8 3.1 Arquitectura da plataforma . 12 4.1 Publico´ Online N-Gram Search - News Articles . 34 4.2 Publico´ Online N-Gram Search - Start Screen . 36 4.3 Publico´ Online N-Gram Search - Normal Search . 37 4.4 Publico´ Online N-Gram Search - Category Search . 38 4.5 Publico´ Online N-Gram Search - Lexical Search . 38 4.6 Publico´ Online N-Gram Search - Entity Search . 39 4.7 Publico´ Online N-Gram Search - On Hover Info . 39 4.8 Publico´ Online N-Gram Search - Artigos que contem´ o N-Grama pesquisado 40 4.9 Publico´ N-Gram Search Admin Program - Main Window . 40 4.10 Publico´ N-Gram Search Admin Program - Upload Window . 41 4.11 Publico´ N-Gram Search Admin Program - Upload Conclu´ıdo . 42 4.12 Publico´ N-Gram Search Admin Program - Verificar Progresso . 43 4.13 Publico´ N-Gram Search Admin Program - Definic¸oes˜ . 43 5.1 Tempo de execuc¸ao˜ pelo tamanho do ficheiro corpus - Algoritmo Base . 48 5.2 Func¸oes˜ Map - Algoritmo Principal . 48 5.3 Tempo de execuc¸ao˜ pelo tamanho do ficheiro corpus - Algoritmo Catego- ria Not´ıcias . 50 5.4 Tempo de execuc¸ao˜ pelo tamanho do ficheiro corpus - Algoritmo Entidades 51 5.5 Tempo de execuc¸ao˜ pelo tamanho do ficheiro corpus - Algoritmo Gramatical 52 5.6 Publico´ N-Gram Search - Pedro Passos Coelho . 53 5.7 Publico´ N-Gram Search Categoria - Pedro Passos Coelho . 54 5.8 Publico´ N-Gram Search Classe Gramatical - Pedro Passos Coelho . 55 xv Lista de Tabelas 5.1 Configurac¸oes˜ das maquinas´ do cluster ................... 47 xvii Cap´ıtulo 1 Introduc¸ao˜ A crescente transic¸ao˜ dos meios tradicionais de comunicac¸ao˜ social para o espac¸o digital veio trazer um novo conjunto de paradigmas relativos as` actividades normalmente associ- adas ao jornalismo, fornecendo um maior leque de analises´ sobre a informac¸ao˜ dispon´ıvel. Esta informac¸ao˜ encontra-se em constante crescimento, com dados a serem produzidos diariamente´ na Internet. O conceito de Big Data e´ actualmente uma realidade, sendo poss´ıvel processar grandes quantidades de informac¸ao˜ atraves´ de tecnicas´ especialmente desenhadas para esse proposito´ e do aumento da capacidade computacional dispon´ıvel. Um estudo poss´ıvel sobre grandes quantidades de dados consiste na analise´ da correlac¸ao˜ entre o numero´ de ocorrenciasˆ de entidades e eventos durante um determinado per´ıodo de tempo.
Load more

Recommended publications
  • Improving Efficiency of Map Reduce Paradigm with ANFIS for Big Data (IJSTE/ Volume 1 / Issue 12 / 015)
    IJSTE - International Journal of Science Technology & Engineering | Volume 1 | Issue 12 | June 2015 ISSN (online): 2349-784X Improving Efficiency of Map Reduce Paradigm with ANFIS for Big Data Gor Vatsal H. Prof. Vatika Tayal Department of Computer Science and Engineering Department of Computer Science and Engineering NarNarayan Shashtri Institute of Technology Jetalpur , NarNarayan Shashtri Institute of Technology Jetalpur , Ahmedabad , India Ahmedabad , India Abstract As all we know that map reduce paradigm is became synonyms for computing big data problems like processing, generating and/or deducing large scale of data sets. Hadoop is a well know framework for these types of problems. The problems for solving big data related problems are varies from their size , their nature either they are repetitive or not etc., so depending upon that various solutions or way have been suggested for different types of situations and problems. Here a hybrid approach is used which combines map reduce paradigm with anfis which is aimed to boost up such problems which are likely to repeat whole map reduce process multiple times. Keywords: Big Data, fuzzy Neural Network, ANFIS, Map Reduce, Hadoop ________________________________________________________________________________________________________ I. INTRODUCTION Initially, to solve problem various problems related to large crawled documents, web requests logs, row data , etc a computational processing model is suggested by jeffrey Dean and Sanjay Ghemawat is Map Reduce in 2004[1]. MapReduce programming model is inspired by map and reduce primitives which are available in Lips and many other functional languages. It is like a De Facto standard and widely used for solving big data processing and related various operations.
    [Show full text]
  • Character-Word LSTM Language Models
    Character-Word LSTM Language Models Lyan Verwimp Joris Pelemans Hugo Van hamme Patrick Wambacq ESAT – PSI, KU Leuven Kasteelpark Arenberg 10, 3001 Heverlee, Belgium [email protected] Abstract A first drawback is the fact that the parameters for infrequent words are typically less accurate because We present a Character-Word Long Short- the network requires a lot of training examples to Term Memory Language Model which optimize the parameters. The second and most both reduces the perplexity with respect important drawback addressed is the fact that the to a baseline word-level language model model does not make use of the internal structure and reduces the number of parameters of the words, given that they are encoded as one-hot of the model. Character information can vectors. For example, ‘felicity’ (great happiness) is reveal structural (dis)similarities between a relatively infrequent word (its frequency is much words and can even be used when a word lower compared to the frequency of ‘happiness’ is out-of-vocabulary, thus improving the according to Google Ngram Viewer (Michel et al., modeling of infrequent and unknown words. 2011)) and will probably be an out-of-vocabulary By concatenating word and character (OOV) word in many applications, but since there embeddings, we achieve up to 2.77% are many nouns also ending on ‘ity’ (ability, com- relative improvement on English compared plexity, creativity . ), knowledge of the surface to a baseline model with a similar amount of form of the word will help in determining that ‘felic- parameters and 4.57% on Dutch. Moreover, ity’ is a noun.
    [Show full text]
  • Study and Analysis of Different Cloud Storage Platform
    International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395 -0056 Volume: 03 Issue: 06 | June-2016 www.irjet.net p-ISSN: 2395-0072 Study And Analysis Of Different Cloud Storage Platform S Aditi Apurva, Dept. Of Computer Science And Engineering, KIIT University ,Bhubaneshwar, India. ---------------------------------------------------------------------***--------------------------------------------------------------------- Abstract - Cloud Storage is becoming the most sought 1.INTRODUCTION after storage , be it music files, videos, photos or even The term cloud is the metaphor for internet. The general files people are switching over from storage on network of servers and connection are collectively their local hard disks to storage in the cloud. known as Cloud .Cloud computing emerges as a new computing paradigm that aims to provide reliable, Google Cloud Storage offers developers and IT customized and quality of service guaranteed organizations durable and highly available object computation environments for cloud users. storage. Cloud storage is a model of data storage in Applications and databases are moved to the large which the digital data is stored in logical pools, the centralized data centers, called cloud. physical storage spans multiple servers (and often locations), and the physical environment is typically owned and managed by a hosting company. Analysis of cloud storage can be problem specific such as for one kind of files like YouTube or generic files like google drive and can have different performances measurements . Here is the analysis of cloud storage based on Google’s paper on Google Drive,One Drive, Drobox, Big table , Facebooks Cassandra This will provide an overview of Fig 2: Cloud storage working. how the cloud storage works and the design principle .
    [Show full text]
  • Large-Scale Youtube-8M Video Understanding with Deep Neural Networks
    Large-Scale YouTube-8M Video Understanding with Deep Neural Networks Manuk Akopyan Eshsou Khashba Institute for System Programming Institute for System Programming ispras.ru ispras.ru [email protected] [email protected] many hand-crafted approaches to video-frame feature Abstract extraction, such as Histogram of Oriented Gradients (HOG), Histogram of Optical Flow (HOF), Motion Video classification problem has been studied many Boundary Histogram (MBH) around spatio-temporal years. The success of Convolutional Neural Networks interest points [9], in a dense grid [10], SIFT [11], the (CNN) in image recognition tasks gives a powerful Mel-Frequency Cepstral Coefficients (MFCC) [12], the incentive for researchers to create more advanced video STIP [13] and the dense trajectories [14] existed. Set of classification approaches. As video has a temporal video-frame features then encoded to video-level feature content Long Short Term Memory (LSTM) networks with bag of words (BoW) approach. The problem with become handy tool allowing to model long-term temporal BoW is that it uses only static video-frame information clues. Both approaches need a large dataset of input disposing of the time component, the frame ordering. data. In this paper three models provided to address Recurrent Neural Networks (RNN) show good results in video classification using recently announced YouTube- modeling with time-based input data. A few papers [15, 8M large-scale dataset. The first model is based on frame 16] describe solving video classification problem using pooling approach. Two other models based on LSTM Long Short-Term Memory (LSTM) networks and achieve networks. Mixture of Experts intermediate layer is used in good results.
    [Show full text]
  • Sector and Sphere: the Design and Implementation of a High Performance Data Cloud Yunhong Gu University of Illinois at Chicago
    Sector and Sphere: The Design and Implementation of a High Performance Data Cloud Yunhong Gu University of Illinois at Chicago Robert L Grossman University of Illinois at Chicago and Open Data Group ABSTRACT available, the data is moved to the processors. To simplify, this is the supercomputing model. An alternative Cloud computing has demonstrated that processing very approach is to store the data and to co-locate the large datasets over commodity clusters can be done computation with the data when possible. To simplify, simply given the right programming model and this is the data center model. infrastructure. In this paper, we describe the design and implementation of the Sector storage cloud and the Cloud computing platforms (GFS/MapReduce/BigTable Sphere compute cloud. In contrast to existing storage and and Hadoop) that have been developed thus far have been compute clouds, Sector can manage data not only within a designed with two important restrictions. First, clouds data center, but also across geographically distributed data have assumed that all the nodes in the cloud are co- centers. Similarly, the Sphere compute cloud supports located, i.e., within one data center, or that there is User Defined Functions (UDF) over data both within a relatively small bandwidth available between the data center and across data centers. As a special case, geographically distributed clusters containing the data. MapReduce style programming can be implemented in Second, these clouds have assumed that individual inputs Sphere by using a Map UDF followed by a Reduce UDF. and outputs to the cloud are relatively small, although the We describe some experimental studies comparing aggregate data managed and processed is very large.
    [Show full text]
  • Cloud Computing Synopsis and Recommendations
    Special Publication 800-146 Cloud Computing Synopsis and Recommendations Recommendations of the National Institute of Standards and Technology Lee Badger Tim Grance Robert Patt-Corner Jeff Voas NIST Special Publication 800-146 Cloud Computing Synopsis and Recommendations Recommendations of the National Institute of Standards and Technology Lee Badger Tim Grance Robert Patt-Corner Jeff Voas C O M P U T E R S E C U R I T Y Computer Security Division Information Technology Laboratory National Institute of Standards and Technology Gaithersburg, MD 20899-8930 May 2012 U.S. Department of Commerce John Bryson, Secretary National Institute of Standards and Technology Patrick D. Gallagher, Under Secretary of Commerce for Standards and Technology and Director CLOUD COMPUTING SYNOPSIS AND RECOMMENDATIONS Reports on Computer Systems Technology The Information Technology Laboratory (ITL) at the National Institute of Standards and Technology (NIST) promotes the U.S. economy and public welfare by providing technical leadership for the nation’s measurement and standards infrastructure. ITL develops tests, test methods, reference data, proof of concept implementations, and technical analysis to advance the development and productive use of information technology. ITL’s responsibilities include the development of management, administrative, technical, and physical standards and guidelines for the cost-effective security and privacy of other than national security-related information in Federal information systems. This Special Publication 800-series reports on ITL’s research, guidance, and outreach efforts in computer security and its collaborative activities with industry, government, and academic organizations. National Institute of Standards and Technology Special Publication 800-146 Natl. Inst. Stand. Technol. Spec. Publ.
    [Show full text]
  • Mapreduce: Simplified Data Processing On
    MapReduce: Simplified Data Processing on Large Clusters Jeffrey Dean and Sanjay Ghemawat [email protected], [email protected] Google, Inc. Abstract given day, etc. Most such computations are conceptu- ally straightforward. However, the input data is usually MapReduce is a programming model and an associ- large and the computations have to be distributed across ated implementation for processing and generating large hundreds or thousands of machines in order to finish in data sets. Users specify a map function that processes a a reasonable amount of time. The issues of how to par- key/value pair to generate a set of intermediate key/value allelize the computation, distribute the data, and handle pairs, and a reduce function that merges all intermediate failures conspire to obscure the original simple compu- values associated with the same intermediate key. Many tation with large amounts of complex code to deal with real world tasks are expressible in this model, as shown these issues. in the paper. As a reaction to this complexity, we designed a new Programs written in this functional style are automati- abstraction that allows us to express the simple computa- cally parallelized and executed on a large cluster of com- tions we were trying to perform but hides the messy de- modity machines. The run-time system takes care of the tails of parallelization, fault-tolerance, data distribution details of partitioning the input data, scheduling the pro- and load balancing in a library. Our abstraction is in- gram's execution across a set of machines, handling ma- spired by the map and reduce primitives present in Lisp chine failures, and managing the required inter-machine and many other functional languages.
    [Show full text]
  • Big Data Challenges and Hadoop As One of the Solution of Big Data with Its Modules Tapan P
    International Journal of Scientific & Engineering Research, Volume 5, Issue 6, June-2014 133 ISSN 2229-5518 Big Data challenges and Hadoop as one of the solution of big data with its Modules Tapan P. Gondaliya, Dr. Hiren D. Joshi Abstract— this is an era of the tools, trend and technology. And these tools, trend and technology era is mainly depending upon the information. Or this information is called the data and that information is stored in database just like a My SQL, Oracle, Access and other many more database available in different companies. But problem is in our daily life we used tools and devices for the entertainment, websites, Scientific instrument, and many more different devices that generate a large amount of data and that is in zeta bytes or in Petabytes that is called big data and that data create a big problem in our day to day life. Very tough to organize or manage this kind of big data but through Apache Hadoop it’s trouble-free. Basically in this paper we describe the main purpose of Apache Hadoop and how can Hadoop organize or manage the different kinds of data and what are the main techniques are to be used behind that. Index Terms— Big Data, Hadoop, Map Reduce, Hadoop Distributed File System, Big Data Challanges —————————— —————————— 1 INTRODUCTION his is the era of digitalization and we all live in digital T world. In this era we need more and more facility for our life is being easier. So we used different kind of new things for entertaining our life using audio video and digital images, through social stuff like a Facebook Google+ Tweeter, using mobile as well as sensor devices, we used company or gov- ernment sector that all this increases the digitalization and create a large amount of structured, semi structured and un- structured data.
    [Show full text]
  • The Pagerank Algorithm and Application on Searching of Academic Papers
    The PageRank algorithm and application on searching of academic papers Ping Yeh Google, Inc. 2009/12/9 Department of Physics, NTU Disclaimer (legal) The content of this talk is the speaker's personal opinion and is not the opinion or policy of his employer. Disclaimer (content) You will not hear physics. You will not see differential equations. You will: ● get a review of PageRank, the algorithm used in Google's web search. It has been applied to evaluate journal status and influence of nodes in a graph by researchers, ● see some linear algebra and Markov chains associated with it, and ● see some results of applying it to journal status. Outline Introduction Google and Google search PageRank algorithm for ranking web pages Using MapReduce to calculate PageRank for billions of pages Impact factor of journals and PageRank Conclusion Google The name: homophone to the word “Googol” which means 10100. The company: ● founded by Larry Page and Sergey Brin in 1998, ● ~20,000 employees as of 2009, ● spread in 68 offices around the world (23 in N. America, 3 in Latin America, 14 in Asia Pacific, 23 in Europe, 5 in Middle East and Africa). The mission: “to organize the world's information and make it universally accessible and useful.” Google Services Sky YouTube iGoogle web search talk book search Chrome calendar scholar translate blogger.com Android product news search maps picasaweb video groups Gmail desktop reader Earth Photo by mr.hero on panoramio (http://www.panoramio.com/photo/1127015)‏ 6 Google Search http://www.google.com/ or http://www.google.com.tw/ The abundance problem Quote Langville and Meyer's nice book “Google's PageRank and beyond: the science of search engine rankings”: The men in Jorge Luis Borges’ 1941 short story, “The Library of Babel”, which describes an imaginary, infinite library.
    [Show full text]
  • Mapreduce: a Flexible Data Processing Tool
    contributed articles DOI:10.1145/1629175.1629198 of MapReduce has been used exten- MapReduce advantages over parallel databases sively outside of Google by a number of organizations.10,11 include storage-system independence and To help illustrate the MapReduce fine-grain fault tolerance for large jobs. programming model, consider the problem of counting the number of by JEFFREY DEAN AND SaNjay GHEMawat occurrences of each word in a large col- lection of documents. The user would write code like the following pseudo- code: MapReduce: map(String key, String value): // key: document name // value: document contents for each word w in value: A Flexible EmitIntermediate(w, “1”); reduce(String key, Iterator values): // key: a word Data // values: a list of counts int result = 0; for each v in values: result += ParseInt(v); Processing Emit(AsString(result)); The map function emits each word plus an associated count of occurrences (just `1' in this simple example). The re- Tool duce function sums together all counts emitted for a particular word. MapReduce automatically paral- lelizes and executes the program on a large cluster of commodity machines. The runtime system takes care of the details of partitioning the input data, MAPREDUCE IS A programming model for processing scheduling the program’s execution and generating large data sets.4 Users specify a across a set of machines, handling machine failures, and managing re- map function that processes a key/value pair to quired inter-machine communication. generate a set of intermediate key/value pairs and MapReduce allows programmers with a reduce function that merges all intermediate no experience with parallel and dis- tributed systems to easily utilize the re- values associated with the same intermediate key.
    [Show full text]
  • Mapreduce: Simplified Data Processing on Large Clusters Jeffry Dean and Sanjay Ghemawat ODSI 2004 Presented by Anders Miltner Problem
    MapReduce: Simplified Data Processing on Large Clusters Jeffry Dean and Sanjay Ghemawat ODSI 2004 Presented by Anders Miltner Problem • When applied on to big data, simple algorithms beCome more complicated • Parallelized Computations • Distributed data • Failure handling • How Can we abstraCt away these Common diffiCulties, allowing the programmer to be able to write simple Code for their simple algorithms? Core Ideas • Require the programmer to state their problem in terms of two funCtions, map and reduCe • Handle the sCalability ConCerns inside the system, instead of requiring the programmer to handle them • Require the programmer to only write Code about the easy to understand functionality Map, Reduce, and MapReduce • map: (k1,v1) -> list (k2, v2) • reduCe: (k2, list v2) -> list (v2) • mapreduCe: list (k1,v1) -> list (k2, list v2) for a rewrite of our production indexing system. Sec- 2.2 Types tion 7 discusses related and future work. Even though the previous pseudo-code is written in terms of string inputs and outputs, conceptually the map and for a rewrite of our production indexing system.2 PrSec-ogramming2.2 TModelypes tion 7 discusses related and future work. reduce functions supplied by the user have associated types: The computationEvtakenesthougha set oftheinputpreviouskey/valuepseudo-codepairs, andis written in terms map (k1,v1) list(k2,v2) produces a setofofstringoutputinputskey/valueand pairs.outputs,Theconceptuallyuser of the map and → 2 Programming Model reduce (k2,list(v2)) list(v2) the MapReducereducelibraryfunctionsexpresses thesuppliedcomputationby theas twusero have associated → functions: Maptypes:and Reduce. I.e., the input keys and values are drawn from a different The computation takes a set of input key/value pairs, and domain than the output keys and values.
    [Show full text]
  • What Bugs Live in the Cloud? a Study of 3000+ Issues in Cloud Systems
    What Bugs Live in the Cloud? A Study of 3000+ Issues in Cloud Systems HaryadiS.Gunawi,MingzheHao, JeffryAdityatama,KurniaJ. Eliazar, Tanakorn Leesatapornwongsa, Thanh Do∗∗ Agung Laksono, Jeffrey F. Lukman, and Tiratat Patana-anake Vincentius Martin, and Anang D. Satria University of University of Chicago∗ Wisconsin–Madison Surya University Abstract tectures have emerged every year in the last decade. Unlike single-server systems, cloud systems are considerably more We conduct a comprehensive study of development and de- complex as they must deal with a wide range of distributed ployment issues of six popular and important cloud systems components, hardware failures, users, and deployment sce- (Hadoop MapReduce, HDFS, HBase, Cassandra, ZooKeeper narios. It is not a surprise that cloud systems periodically ex- and Flume). From the bug repositories, we review in total perience downtimes [7]. There is much room for improving 21,399 submitted issues within a three-year period (2011- cloud systems dependability. 2014). Among these issues, we perform a deep analysis of This paper was started with a simple question: why are 3655 “vital” issues (i.e., real issues affecting deployments) cloud systems not 100% dependable? In our attempt to pro- with a set of detailed classifications. We name the product of vide an intelligent answer, the question has led us to many our one-year study Cloud Bug Study database (CBSDB) [9], more intricate questions. Why is it hard to develop a fully with which we derive numerous interesting insights unique to reliable cloud systems? What bugs “live” in cloud systems? cloud systems. To the best of our knowledge, our work is the How should we properly classify bugs in cloud systems? Are largest bug study for cloud systems to date.
    [Show full text]