DOCSLIB.ORG

A New Self-Organizing Neural Gas Model Based on Bregman Divergences

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
A New Self-Organizing Neural Gas Model Based on Bregman Divergences A New Self-Organizing Neural Gas Model based on Bregman Divergences Esteban J. Palomo∗, Miguel A. Molina-Cabello∗, Ezequiel Lopez-Rubio´ ∗ and Rafael Marcos Luque-Baena∗ ∗Department of Computer Languages and Computer Science University of Malaga,´ Bulevar Louis Pasteur, 35, 29071 Malaga,´ Spain Emails: fejpalomo,miguelangel,ezeqlr,[email protected] Abstract—In this paper, a new self-organizing neural gas by using Bregman divergences the most suitable divergence model that we call Growing Hierarchical Bregman Neural according to input data can be specified. In this paper, a new Gas (GHBNG) has been proposed. Our proposal is based self-organizing neural network called the Growing Hierar- on the Growing Hierarchical Neural Gas (GHNG) in which chical Bregman Neural Gas (GHBNG) is proposed, which Bregman divergences are incorporated in order to compute is grounded in the GHNG model and in which Bregman the winning neuron. This model has been applied to anomaly divergences have been considered. detection in video sequences together with a Faster R-CNN On the other hand, the proliferation in recent years of as an object detector module. Experimental results not only a huge amount of visual information in the form of data confirm the effectiveness of the GHBNG for the detection sequences has led to a growth in the field of intelligent video of anomalous object in video sequences but also its self- surveillance. In particular, one of the most important tasks organization capabilities. to consider is to automatically detect moving objects that are not very frequent in a scene and can be considered as anomalies. In recent years, the appearance of deep learning 1. Introduction networks for the detection of objects in an image has meant a turning point in the detection of objects in video sequences The Self-organizing Map (SOM) [1] has been widely [8]. Thus, it is possible to use pre-trained networks with used for data clustering since its publication. The SOM thousands of data and a large number of object types to performs a mapping between high-dimensional data and detect moving objects in a scene, providing more stable a lower dimensional representation space preserving the results than those obtained by classical approaches. topology of input data. Many SOM-like neural models have In order to show the possible applications of our pro- been proposed over the years, which are based on a fixed posal, we have also applied the GHBNG to anomalous lattice topology among the neurons [2]. The Growing Neural detection in video sequences acquired by fixed IP cameras. Gas (GNG) [3] is a self-organizing neural network which The objects in motion on each frame are obtained by the learns a dynamic graph with variable numbers of neurons Faster RCNN network [9]. Later, the GHBNG estimates the and connections. This graph represents input data in a objects considered as anomalous after a previous training more plastic and flexible way than a fixed-topology map, phase. improving visualization capabilities and understanding of The rest of the paper will have the following structure: data. section 2 exhaustively describes the GHBNG model. Section These self-organizing models have their hierarchical ver- 3 presents several experiments which demonstrate the self- sions, such as the Growing Hierarchical Self-Organizing organization capacity of the GHBNG model, in addition to Map (GHSOM) for the SOM [4] and the Growing Hier- its application for the detection of anomalous objects in archical Neural Gas (GHNG) for the GNG [5], in which a video sequences. Finally section 4 concludes the paper. neuron can be expanded into a new map or graph in a sub- sequent layer of the hierarchy depending on the quantization 2. The GHBNG Model error associated to that neuron or the graph it belongs to. Hierarchical models can reflect hierarchical relations present A Growing Hierarchical Bregman Neural Gas (GHBNG) among input data in a more straightforward way. network is defined as a Growing Hierarchical Neural Gas Another possible problem present in these self- (GHNG) network [5] in which Bregman divergences are organizing models is the use of the Euclidean distance to incorporated in order to compute the winning neuron. A compute the winning neuron, since this distance may not be GHBNG network can be seen as a tree of Growing Neural the most suitable for all input distributions. Hence, Bregman Gas (GNG) networks [3] where a mechanism to control the divergences were taken into account for the GHSOM [6], growth of each GNG graph is established. This mechanism since they are suited for clustering because their minimizer distinguishes between a growth phase where more neurons is the mean [7]. Moreover, the squared Euclidean distance are added until no significant improvement in the quanti- is a particular case of the Bregman divergences. Therefore, zation error is obtained, and a convergence phase where no more units can be created. Thus, each graph contains divergence is uniquely associated to a regular exponential a variable number of neurons so that its size can grow or family of probability density functions, that are defined shrink during learning. Also, each graph is the child of a unit below. This way, a unique probability density function can in the upper level, except for the top level (root) graph which be linked to the cluster associated to a given centroid, has no parent. An example of the structure of a GHBNG which enables probabilistic soft clustering. Furthermore, model is shown in Figure 1. Note that the structure is the expectation maximization can be carried out with a reduced same as the GHNG since the difference between these two computational complexity for general Bregman divergences, self-organizing models resides in the way to compute the so that specific Bregman divergences can be designed to suit winning neuron according to the used Bregman divergence. the application at hand. The definition of the GHBNG is organized in two sub- The property that the mean is the minimizer of a Breg- sections. First a review of Bregman divergences is presented. man divergence is formalized next. Given an input distribu- Then, the basic model for a graph and the corresponding tion for x the following condition holds [12]: learning algorithm are explained (Subsection 2.3). Finally we explain how new graphs are created to yield a hierarchy µ = E [x] = arg min E [Dφ (x; y)] (2) of graphs (Subsection 2.4). y Let N be the number of clusters, and let µi be the mean vector of the i-th cluster Ci, i 2 f1; :::; Ng. Then a point x belongs to Ci if µi minimizes the divergence with respect to x: Ci = x 2 S j i = arg min Dφ x; µj (3) j2f1;:::;Ng So, we can rewrite (2) to partition S into N clusters Ci: µi = E [x j Ci] = arg min E [Dφ (x; y) j Ci] (4) y The above equation implies that the mean of the cluster Figure 1. Structure of a GHBNG model with four graphs. The parent Ci minimizes the Bregman divergence to the samples x neurons are shown in a darker tone. which belong to the cluster. 2.2. Basic model 2.1. Review of Bregman Divergences For clustering and self-organizing network applications it is necessary to learn a weight vector wi of each cluster Next the fundamentals of Bregman divergences and their i [13], so that wi estimates the cluster mean vector µi. application to clustering are reviewed. Let φ : S! R be a Stochastic gradient descent has been proposed in [12] to strictly convex real valued function defined over a convex minimize E [Dφ (x; z)]: set S ⊆ RD, where D is the dimension of the input data [10], [11], [12]. We assume that φ is differentiable on the @Dφ (x; wi) relative interior ri (S) of the set S [7]. Then the Bregman 4wi = −η (5) @wi divergence Dφ : S × ri (S) ! [0; +1) corresponding to φ is defined as where η is a suitable step size. Here we propose a different approach, namely the es- timation of the cluster mean vector E [x j Ci] by stochastic T Dφ (x; y) = φ (x) − φ (y) − (x − y) rφ (y) (1) approximation [14], [15], [16], [17]. This strategy has been successfully applied by the authors to other self-organizing where x 2 S and rφ (y) stands for the gradient vector models in [18], [19], [20]. The goal of stochastic approx- of φ evaluated at y 2 ri (S). Table 1 lists the Bregman imation is to find the value of some parameter θ which divergences that we consider in this paper. satisfies ζ (θ) = 0 (6) Bregman divergences are suited for clustering because their minimizer is the mean. This is the main contribution of where ζ is a function whose values can not be obtained [7], where it is proved that the class of distortion measures directly. What we have is a random variable z which is a with respect to a set of centroids which admit an itera- noisy estimate of ζ: tive minimization procedure is precisely that of Bregman divergences. Moreover, it is also proved that each Bregman E [z (θ) j θ] = ζ (θ) (7) Divergence S φ (x) Dφ (x; y) D 2 2 Squared Euclidean distance R kxk kx − yk D PD PD xk Generalized I-divergence R+ k=1 xk log xk k=1 −xk + yk + xk log y k D PD PD xk xk Itakura-Saito distance − log xk −1 + − log R+ k=1 k=1 yk yk D PD PD Exponential loss R exp xk (exp xk − exp yk − (xk − yk) exp yk) k=1 k=1 D PD PD xk 1−xk Logistic loss (0; 1) (xk log xk + (1 − xk) log (1 − xk)) xk log + (1 − xk) log k=1 k=1 yk 1−yk D TABLE 1.
Load more

Recommended publications
  • Growing Neural Gas with Correntropy Induced Metric
    Growing Neural Gas with Correntropy Induced Metric Naoki Masuyama Chu Kiong Loo Faculty of Computer Science and Information Technology, Faculty of Computer Science and Information Technology, University of Malaya, University of Malaya, 50603 Kuala Lumpur, MALAYSIA 50603 Kuala Lumpur, MALAYSIA Email: [email protected] Email: [email protected] Abstract—This paper discusses the Correntropy Induced Met- of arbitrary shapes without any assumptions on the number ric (CIM) based Growing Neural Gas (GNG) architecture. CIM is of clusters. Boonmee et al. [13] introduced a hybrid GNG a kernel method based similarity measurement from the informa- by considering not only the distance between the clusters, tion theoretic learning perspective, which quantifies the similarity between probability distributions of input and reference vectors. but also the centroids of the each cluster to obtain more We apply CIM to find a maximum error region and node insert practical topology structure. Mohebi and Bagirov [14] applied criterion, instead of euclidean distance based function in original an algorithm based on split and merge of clusters to initialize GNG. Furthermore, we introduce the two types of Gaussian neurons. The initialization algorithm speeds up the learning kernel bandwidth adaptation methods for CIM. The simulation process in large high-dimensional data sets. Palomo et al. experiments in terms of the affect of kernel bandwidth σ in CIM, the self-organizing ability, and the quantitative comparison show [15] combined GNG and tree structure to represent a network that proposed model has the superior abilities than original GNG. topology, which is called Growing Neural Forest (GNF). Index Terms—Growing Neural Gas; Correntropy Induced GNF learns a set of trees so that each tree represents a Metric; Kernel Method; Clustering; connected cluster of data.
    [Show full text]
  • Probabilistic Graphical Model Based on Growing Neural Gas for Long Time Series Classification
    Modern Applied Science; Vol. 9, No. 2; 2015 ISSN 1913-1844 E-ISSN 1913-1852 Published by Canadian Center of Science and Education Probabilistic Graphical Model Based on Growing Neural Gas for Long Time Series Classification Irina N. Palamar1 & Sergey S. Yulin1 1 Soloviev Rybinsk State Aviation Technical University, Rybinsk, Russia Correspondence: Sergey S. Yulin, Soloviev Rybinsk State Aviation Technical University, Rybinsk, Russia. Received: September 10, 2014 Accepted: October 16, 2014 Online Published: December 11, 2014 doi:10.5539/mas.v9n2p108 URL: http://dx.doi.org/10.5539/mas.v9n2p108 Abstract This article proposes a generative probabilistic graphical model with hidden states (Neural Gas Graphical Model (NGGM)) based on data approximation with a grid of "neural gas" nodes aimed at solving the problem of long time series classification. Such time series include information about changes in economic, weather and health values, as well as information about changes in values of operation sensors of technical objects during a quite long period. The most difficult task of classification of such long time series using probabilistic graphical models with hidden states is the selection of the optimum number of hidden states. This work proposes a method for automatic selection of the optimum number of hidden states of the model in the course of the model learning. The model proposed in the article and the methods of its learning are based on a combination of elements used in the metric and Bayesian approaches to classification. The basic NGGM purpose is to match hidden states of a graphical model and nodes (neurons) of the approximating grid.
    [Show full text]
  • Neural Network Classification [Roya Asadi*, Haitham Sabah Hasan, Sameem Abdul Kareem]
    Proc. of the Intl. Conf. on Advances in Computer Science and Electronics Engineering -- CSEE 2014 Copyright © Institute of Research Engineers and Doctors. All rights reserved. ISBN: 978-1-63248-000-2 doi: 10.15224/ 978-1-63248-000-2-36 Review of current Online Dynamic Unsupervised Feed Forward Neural Network classification [Roya Asadi*, Haitham Sabah Hasan, Sameem Abdul Kareem] Abstract— Online Dynamic Unsupervised Feed Forward Neural networks are able to dynamically learn the types of Neural Network (ODUFFNN) classification is suitable to be input values based on their weights and properties. A feed applied in different research areas and environments such as forward neural network is a popular tool for statistical decision email logs, networks, credit card transactions, astronomy and making and is a software version of the brain. The neural satellite communications. Currently, there are a few strong network is flexible algorithm that allows us to encode methods as ODUFFNN classification, although they have general nonlinear relationships between input and desirable outputs [5- problems. The goal of this research is an investigation of the 7]. The online dynamic unsupervised classification must solve critical problems and comparison of current ODUFFNN several problems such as a huge data with high dimensions classification. For experimental results, Evolving Self-Organizing which causes huge memory usage, high processing time and Map (ESOM) and Dynamic Self-Organizing Map (DSOM) as low accuracy [2, 8, 9]; losing data details because of dividing strong related methods are considered; and also they applied some difficult datasets for clustering from the UCI Dataset input values in a few clusters [8, 10, 11]; defining the Repository.
    [Show full text]
  • 222826308.Pdf
    View metadata, citation and similar papers at core.ac.uk brought to you by CORE provided by Aston Publications Explorer Published in: Neural Computation 18, 446-469 (2006) Magnification Control in Self-Organizing Maps and Neural Gas Thomas Villmann Clinic for Psychotherapy, University of Leipzig, 04107 Leipzig, Germany Jens Christian Claussen Institute of Theoretical Physics and Astrophysics, Christian-Albrecht University Kiel, 24098 Kiel, Germany We consider different ways to control the magnification in self-organizing maps (SOM) and neural gas (NG). Starting from early approaches of magnification control in vector quantization, we then concentrate on different approaches for SOM and NG. We show that three structurally similar approaches can be applied to both algorithms: localized learning, concave-convex learning, and winner relaxing learning. Thereby, the approach of concave-convex learning in SOM is extended to a more general description, whereas the concave-convex learning for NG is new. In general, the control mechanisms generate only slightly different behavior comparing both neural algorithms. However, we emphasize that the NG results are valid for any data dimension, whereas in the SOM case the results hold only for the one-dimensional case. 1 Introduction Vector quantization is an important task in data processing, pattern recog- nition and control (Fritzke, 1993; Haykin, 1994; Linde, Buzo, & Gray, 1980; Ripley, 1996). A large number of different types have been dis- cussed, (for an overview, refer to Haykin, 1994; Kohonen, 1995; Duda & Hart, 1973). Neural maps are a popular type of neural vector quan- tizers that are commonly used in, for example, data visualization, feature extraction, principle component analysis, image processing, classification tasks, and acceleration of common vector quantization (de Bodt, Cot- trell, Letremy, & Verleysen, 2004).
    [Show full text]
  • Batch and Median Neural Gas Marie Cottrell, Barbara Hammer, Alexander Hasenfuss, Thomas Villmann
    Batch and median neural gas Marie Cottrell, Barbara Hammer, Alexander Hasenfuss, Thomas Villmann To cite this version: Marie Cottrell, Barbara Hammer, Alexander Hasenfuss, Thomas Villmann. Batch and median neural gas. Neural Networks, Elsevier, 2006, 19, pp.762-771. 10.1016/j.neunet.2006.05.018. hal-00107462 HAL Id: hal-00107462 https://hal.archives-ouvertes.fr/hal-00107462 Submitted on 18 Oct 2006 HAL is a multi-disciplinary open access L’archive ouverte pluridisciplinaire HAL, est archive for the deposit and dissemination of sci- destinée au dépôt et à la diffusion de documents entific research documents, whether they are pub- scientifiques de niveau recherche, publiés ou non, lished or not. The documents may come from émanant des établissements d’enseignement et de teaching and research institutions in France or recherche français ou étrangers, des laboratoires abroad, or from public or private research centers. publics ou privés. Batch and median neural gas Marie Cottrell SAMOS-MATISSE, Universite´ Paris I, Paris, France Barbara Hammer 1 Institute of Computer Science, Clausthal University of Technology, Clausthal-Zellerfeld, Germany Alexander Hasenfuß Institute of Computer Science, Clausthal University of Technology, Clausthal-Zellerfeld, Germany Thomas Villmann Clinic for Psychotherapy, Universitat¨ Leipzig, Leipzig, Germany ccsd-00107462, version 1 - 18 Oct 2006 1 corresponding author: Barbara Hammer, Institute of Computer Science, Clausthal University of Technology, Julius-Albert-Str. 4, D-38678 Clausthal-Zellerfeld, Germany, [email protected] Preprint submitted to Elsevier Science 18 October 2006 Abstract Neural Gas (NG) constitutes a very robust clustering algorithm given euclidian data which does not suffer from the problem of local minima like simple vector quantization, or topo- logical restrictions like the self-organizing map.
    [Show full text]
  • Arxiv:1509.07093V1 [Cs.LG] 23 Sep 2015
    Noname manuscript No. (will be inserted by the editor) A Review of Learning Vector Quantization Classifiers David Nova · Pablo A. Estévez the date of receipt and acceptance should be inserted later Abstract In this work we present a review of the state of the art of Learning Vector Quantization (LVQ) classifiers. A taxonomy is proposed which inte- grates the most relevant LVQ approaches to date. The main concepts associ- ated with modern LVQ approaches are defined. A comparison is made among eleven LVQ classifiers using one real-world and two artificial datasets. Keywords Learning Vector Quantization · Supervised Learning · Neural Networks · Margin Maximization · Likelihood Ratio Maximization 1 Introduction Learning Vector Quantization (LVQ) is a family of algorithms for statistical pattern classification, which aims at learning prototypes (codebook vectors) representing class regions. The class regions are defined by hyperplanes be- tween prototypes, yielding Voronoi partitions. In the late 80’s Teuvo Kohonen introduced the algorithm LVQ1 [36, 38], and over the years produced several variants. Since their inception LVQ algorithms have been researched by a small but active community. A search on the ISI Web of Science in November, 2013, found 665 journal articles with the keywords “Learning Vector Quantization” or “LVQ” in their titles or abstracts. This paper is a review of the progress made in the field during the last 25 years. D. Nova Department of Electrical Engineering, Faculty of Physical and Mathematical Sciences, Uni- versity of Chile, Santiago, Chile E-mail: [email protected] P. A. Estévez arXiv:1509.07093v1 [cs.LG] 23 Sep 2015 Department of Electrical Engineering and Advanced Mining Technology Center, Faculty of Physical and Mathematical Sciences, University of Chile, Santiago, Chile E-mail: [email protected] 2 David Nova, Pablo A.
    [Show full text]
  • 1 Introduction
    Controversial Empirical Results on Batch versus One Pass Online Algorithms CONTROVERSIAL EMPIRICAL RESULTS ON BATCH VERSUS ONE PASS ONLINE ALGORITHMS A. I. González, M. Graña1 Grupo de Inteligencia Computacional Facultad de Informática, UPV/EHU Apdo. 649, 20080 San Sebastián, España [email protected] Abstract – We present controversial empirical results about the relative convergence of batch and online neural network vector quantization (VQ) learning algorithms. It is the commonplace belief that online algorithms have a very slow convergence, while batch versions are faster and give better results. However we propose an online realization of the Self-Organizing Map (SOM) and the Neural Gas algorithm, with learning parameters tuned to perform the training in a one-pass over the sample, that are faster than their batch counterparts and that improve them on some benchmark data sets. Although these results are too limited to claim any kind of superiority of the One Pass algorithms over their batch counterparts, they nevertheless must be taken into account as existence proofs contradicting the claim that the batch realizations are faster and more accurate than online algorithms. Keywords – SOM, Neural Gas, Batch, One pass online 1 Introduction Vector Quantization [9, 10, 12] is a technique that maps a set of input vectors into a finite collection of predetermined codevectors. The set of all codevectors is called the codebook. In designing a vector quantizer, the goal is to construct a codebook for which the expected distortion of approximating any input vector by a codevector is minimized. This task has been tackled by with Competitive Neural Networks (CNN) [1].
    [Show full text]
  • Self-Organizing Neural Networks for Sequence Processing
    University of OsnabruckÄ Institute of Computer Science Department of Mathematics and Computer Science Self-Organizing Neural Networks for Sequence Processing Thesis by Marc Strickert In partial ful¯llment of the requirements for the degree of Doctor of Computer Science (Dr. rer. nat.) Osnabruck,Ä Germany June 7, 2004 Referees: Prof. Dr. Barbara Hammer, Research Group `Learning with Neural Methods on Structured Data' (LNM), Technical University of Clausthal, Germany. Prof. Dr. Helge Ritter, Faculty of Technology, Neuroinformatics Group, University of Bielefeld, Germany. Self-Organizing Neural Networks for Sequence Processing Arbeitsgruppe LNM Institut furÄ Informatik am Fachbereich furÄ Mathematik, Informatik und Angewandte Systemwissenschaft UniversitÄat OsnabruckÄ D-49069 Osnabruck,Ä Germany Arbeitsgruppenleiterin: Dr. Barbara Hammer Institutsdirektor: Prof. Dr. Oliver Vornberger Autor: Dipl.-Systemwiss. Marc Strickert Osnabruck,Ä Juni 2004 Abstract This work investigates the self-organizing representation of temporal data in prototype- based neural networks. Extensions of the supervised learning vector quantization (LVQ) and the unsupervised self-organizing map (SOM) are considered in detail. The principle of Hebbian learning through prototypes yields compact data models that can be easily interpreted by similarity reasoning. In order to obtain a robust prototype dynamic, LVQ is extended by neighborhood cooperation between neurons to prevent a strong dependence on the initial prototype locations. Additionally, implementations of more general, adap- tive metrics are studied with a particular focus on the built-in detection of data attributes involved for a given classi¯cation task. For unsupervised sequence processing, two modi¯- cations of SOM are pursued: the SOM for structured data (SOMSD) realizing an e±cient back-reference to the previous best matching neuron in a triangular low-dimensional neural lattice, and the merge SOM (MSOM) expressing the temporal context as a fractal combi- nation of the previously most active neuron and its context.
    [Show full text]