DOCSLIB.ORG

Elements of Graphical Models

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text

Load more

Steffen L. Lauritzen Elements of Graphical Models Lectures from the XXXVIth International Probability Summer School in Saint-Flour, France, 2006 September 4, 2011 Springer Your dedication goes here Preface Here come the golden words place(s), First name Surname month year First name Surname vii Contents 1 Introduction ................................................... 1 2 Markov Properties ............................................. 5 2.1 Conditional Independence . .5 2.1.1 Basic properties . .5 2.1.2 General conditional independence . .6 2.2 Markov Properties for Undirected Graphs . .9 2.3 Markov Properties for Directed Acyclic Graphs . 15 2.4 Summary . 22 3 Graph Decompositions and Algorithms ........................... 25 3.1 Graph Decompositions and Markov Properties . 25 3.2 Chordal Graphs and Junction Trees . 27 3.3 Probability Propagation and Junction Tree Algorithms. 32 3.4 Local Computation . 32 3.5 Summary . 39 4 Specific Graphical Models ...................................... 41 4.1 Log-linear Models . 41 4.1.1 Interactions and factorization . 41 4.1.2 Dependence graphs and factor graphs . 42 4.1.3 Data and likelihood function . 44 4.2 Gaussian Graphical Models . 50 4.2.1 The multivariate Gaussian distribution . 50 4.2.2 The Wishart distribution . 52 4.2.3 Gaussian graphical models . 53 4.3 Summary . 58 5 Further Statistical Theory....................................... 61 5.1 Hyper Markov Laws . 61 5.2 Meta Markov Models . 66 ix x Contents 5.2.1 Bayesian inference . 71 5.2.2 Hyper inverse Wishart and Dirichlet laws . 72 5.3 Summary . 73 6 Estimation of Structure ......................................... 77 6.1 Estimation of Structure and Bayes Factors . 77 6.2 Estimating Trees and Forests . 81 6.3 Learning Bayesian networks . 85 6.3.1 Model search methods . 85 6.3.2 Constraint-based search . 87 6.4 Summary . 88 References ......................................................... 93 Chapter 1 Introduction Conditional independence The notion of conditional independence is fundamental for graphical models. For three random variables X, Y and Z we denote this as X ??Y jZ and graphi- cally as XZYu u u If the random variables have density w.r.t. a product measure m, the conditional independence is reflected in the relation f (x;y;z) f (z) = f (x;z) f (y;z); where f is a generic symbol for the densities involved. Graphical models 2 4 @ @ 1 u 5 u 7 @@ @@ @ @ u u u @@ @@ 3u 6 u For several variables, complex systems of conditional independence can be de- scribed by undirected graphs. Then a set of variables A is conditionally independent of set B, given the values of a set of variables C if C separates A from B. 1 2 1 Introduction Has tuberculosis Has Positive X-ray? Positive Visit to Asia?to Visit Tuberculosis or canceror Tuberculosis Has lung cancer lung Has Dyspnoea? Smoker? Has bronchitis Has Fig. 1.1 An example of a directed graphical model describing the relationship between risk fac- tors, lung diseases and symptoms. This model was used by Lauritzen and Spiegelhalter (1988) to illustrate important concepts in probabilistic expert systems. A pedigree Graphical model for a pedigree from study of Werner’s syndrome. Each node is itself a graphical model. 1 Introduction 3 A highly complex pedigree p87p99 p98 p88 p89 p90 p100p68p59 p91 p177 p60 p92 p93 p104p182p36 p37 p105p17p153p154p181p78 p18 p77 p49p56 p42 p40 p39 p79 p80 p151p48 p152p85 p174 p183p179p44 p199p28 p27 p51 p84 p70 p15p19 p16 p186p41 p8 p20 p6 p82 p5 p52p61 p62 p75 p2 p178 p185p47 p83 p33 p66 p101p29p65 p102p21 p30 p97 p14 p63 p64 p13 p175 p43p280p314p86 p281 p72p103p288p50p317p295p292 p318p294p31p291 p293p282p25p297p296p343p10p287p73p46p341p193p247p81p358p106 p365p354p355p304 p340p357 p356p148p1 p245p337p306p278p299p171p336p125p279p368 p342p284p11p69p198p170p275p321p12 p26p9 p313p176p259p262p330p235p257p260p244p242p53p329p243p274p34p327p3 p241p248p249p95p271p230p273p228p226p4 p236p348 p234p94p270p7 p22p346p23p298p272p233p326p227p338p307p285p240p325p38p74p303p309 p302p76p239p344p322p323p324p238p195p180 p415p187p311p550p551p310p277 p55p334p96p485p335p45 p67p143 p552p488p487p560p434p305p567 p486p483p578p566p565p481 p568p312p478 p537p145p184p579p246p139p587p580p581p316p191p359p360p361p366p267p423p480p563p527p528p577 p562p522p529p532p530 p424p430 p349p452p427p232p520p194 p422p453p142p456p459p437p372p455p144p436p521p535p192p331p369p258p328 p454p263 p457 p460p505p395p438p477p467p283p458p475p435p558p557p469p251p173p396 p465p463 p559p286p541p265p470p538p461p433p141p426p431p471p24p231p253p300p503p525p269p416p428p429p531p229p264p462p419p504p472p169p250p301p448p449p417p254p494p495p445p266p497 p418 p255p261 p493p252p450p237p491 p492p443p447p444p564p379p539p381p466p376p155p347p308p482p561p588p352p351p290p196p136 p289 p350p339 p553p353p554p315p911p523p910p332p573 p524p526p574p907p276 p912 p333p507p895p889 p908p156p909p137 p575p913p484p190p882p345p383p548p320p938p555 p547p546p620p140p549p189p939p948p940 p545p946p941p582 p451p929p891p388p618p35p386p619p883p902p595p134 p896p363p924p425 p926 p135p928 p384 p925p585p914p420p536 p421p594p440p881p542p892p377p598p888p543p893p597 p596p508p364 p880p608p512p432p625p714p654p904p544p906 p441p138p514p511p397p510p936p905p385p609p921p506p509p622p637p937p887p927p446p479 p920 p606p367p930p468p934p901p917p373p256 p918p916p915 p935p919p621p626p540p571p474p636 p886p723p884p533p501p890p502p931p656p150p500p933p885p632p168p603p645p644p623p268p688p633p903p607 p605p515p387p572 p655p518p593p513 p517p392p464p394p439p635p146p393 p687p390p599p556 p686p378p391p200p473p638p602p398 p685 p516p684p206p771p496p172p628p399p768p770p629p534p570 p627p569 p630p374p932p197p576 p202 p769 p942p205p203p223p208p1585p702p114p701p900p989p898p490p982p981p726p703p204p959p923p984p993p738 p899 p983p728p980p760p727p489 p985p1004p664p1006p682p1003p951p1017p1077p584p955p943p894p922p757p211p992p954p750p499p945p991p161p944p165p986p631p147p949p362p1182 p987 p953 p958p947p319p950 p952p677p613p731p1070p956p129p442 p957p1078p1020p201p370p1014p1122p754p970p735p704p737p162p734p973p118p1124p1016p966p611p968p1042p1043p616 p1123p586 p971p614p212p740p742p743p676p617p744 p1015p749p1075p1013p666p711p710p1038p733p967p709p753p659p660p1035p1588p706 p612p1101p988p662p712p683 p1018p159p1111p1097 p222p1100p716p476p1096p1008p1045p657p1007p604p1036p640p1074p1039p1099 p1009 p643p1055 p1037p1048p641p1071p590p610p1027p218p836p673p962p1054p1026p1024 p1022 p1010p1053 p639p1073p1098p679p976p977p975p725p678 p1049p1052 p1025 p724p680p979p715p157p960p974p897p961p965p729p642p130 p978 p1011p408p1072p1040p752p1044p1041p407p751 p409p160p651 p1051p648p401p601p404p838p382p1021p1023p403p1083p661 p405p719p389p600p658p674p498 p1012p730p624p739 p718p380p667 p406p964p1080p672 p649 p668 p1031p720p721p402p1076 p717p120p110 p722p1587 p671p759 p132p113p375p1079p221p1586p210p400 p216p217p1558p995p163p873p1459p1509p111p839p777p775p793p788p1127p1508p787p1267p990p1458p149p772p790p998p841p789p1375p792p1005p647p1455p1191p220p1372p774p1374 p1510 p791p1368p785 p786p1271p1332p225p1376p669p1371 p1266p1370p1066p1187p1086 p994p1069 p207p1063 p997p1269p1064p755p782p1194p1068p1144p776p1193 p1061 p1270p1084p1060p1148p1373 p646p1065p1454p1057p1337p1369 p1333p1002p1593p1336p1335p1149 p1265p1001p1087p1272p1456p1140p116p1121p784p1329p1059 p1088p778 p1338p1125p779p689p848p1203 p1085p996 p1334p1204p1331 p1268p1142p371 p1330p1190p857p213p1520p854p1126p1521p736p867p107p732p1457p855p1185p844p1145p780p1062 p846p1143p1141p1000p1578p834p1186p802p1067p1209p1184p1146p969p856p188p783p835p615p797p1028p1197p1046 p801p781p696p1219p1202p1147p1205p1607 p833p1207p1606p697p758p1198p1188p663p695p705p1611 p1208 p1047 p1206 p1605p798p1613p1189p741p1104p999p707 p803p1092p1105p1609p1030p866p1608p1108 p1612 p1577 p800p1110p691p1614p1177p690p1106p1107p1102p699p837p1128p1109p158p215p1175p1176p963p1103p670p1135p1139p874p1130 p1610p1174 p799 p692 p698p858p1019p796p1029p127p1050p1171p209p410p1137p1419p1199p1132p1136p1201p1133p1094p1095p1134p1243p681p166p117p794p589p1495p1183p634p1232p713p694p1032p1056p828p1230p1089p700 p795p1218p591p1181p1034p1033p652p214p708p830p827p1231p1093p1421p761p412p519p1172p1058p1200p1082 p829 p1418p1090p109p1229p762p1223p1228p764p763p1170p1227p122p748 p831p119p1091p1420p767p653p766p852p745p746p747 p693p1081p826p650 p1244 p832p1436p1129p112p972 p1417p1225p1590p1168p843p1438p1169p1435p1180p1437p1166p1165p1220p765p1163p1162p414p413p1226p1434p1222p864 p224p1221p871p1592 p851p1589p1138p1439p123p1349p1591 p850 p411p1161p1179p131p1173p845p1224p853 p1260p1392p1394p1512p1286p1490p1294p128p1393p1388p1391p1396p1389p121p1390p1544p1464p1291p863p1293p1289p1292p1288p1328p1257p1450p1451p1453p1287p1449 p1261p879 p1290p1415p1403p108 p1262p1506p773p133p1264p1326p1256p840p1463p1192p1411 p1263p1195p1246p1395p1255p1414 p1258p1452p1250p1440p1327p1584p1410p1252p1253p1245p1402p1466 p1524p1523p1522 p1412 p1441p1325p1311 p1416 p1259p1405p1150 p1254p1404p1308 p1413p1249p1473p1247p1505p1504p1345 p1400p1155p1497p1348 p1251 p1248 p1344p756 p1401 p1324p1409 p1467p1579p1283p1498p1566p1275p1304p1279p1340p1210 p1496p878p1399p1342 p1347p1407p1550p1276p1406 p1341 p1339 p1343p1233 p1346p1427p1277p1474p1431p1397p126 p1274 p1478p1309p1433p1429p1299p1428p1468p583 p1398p1280 p1282p1285p1323p1284p1281p1408p1432 p1307p823p1297p1296p1430p818p819 p824p849p820p1196p1319p1300p1237p822 p1273 p825p1301p1320 p1318p1242p1302p1321p1313p167 p821p1310p1303p814p1322p1298p1422 p1551p1315 p1317p812p1239p817p1316p816 p1314p1295p1306p1442p1426p1278p1352p860p1541p1350p1241p862p1113p1312 p813p1240p665 p859p1443p1236 p815p1114p1238 p1305p1569p1530p1235p1213p1465p870p1517p811p847p869p1447p1131p1537p1462p1117p1519p592p1516p1351p1120p1115p1445p1444p1538
Recommended publications
  • Arxiv:1606.02359V1 [Stat.ME] 7 Jun 2016

    Arxiv:1606.02359V1 [Stat.ME] 7 Jun 2016

    STRUCTURE LEARNING IN GRAPHICAL MODELING MATHIAS DRTON AND MARLOES H. MAATHUIS Abstract. A graphical model is a statistical model that is associated to a graph whose nodes correspond to variables of interest. The edges of the graph reflect allowed conditional dependencies among the variables. Graphical models admit computationally convenient fac- torization properties and have long been a valuable tool for tractable modeling of multivariate distributions. More recently, applications such as reconstructing gene regulatory networks from gene expression data have driven major advances in structure learning, that is, estimat- ing the graph underlying a model. We review some of these advances and discuss methods such as the graphical lasso and neighborhood selection for undirected graphical models (or Markov random fields), and the PC algorithm and score-based search methods for directed graphical models (or Bayesian networks). We further review extensions that account for effects of latent variables and heterogeneous data sources. 1. Introduction This article gives an overview of commonly used techniques for structure learning in graphical modeling. Structure learning is a model selection problem in which one estimates a graph that summarizes the dependence structure in a given data set. 1.1. What is a Graphical Model? A graphical model captures stochastic dependencies among a collection of random variables Xv, v 2 V (Lauritzen, 1996). Each model is associated to a graph G = (V; E), where the vertex set V indexes the variables and the edge set E imposes a set of conditional independencies. Specifically, Xv and Xw are required to be conditionally independent given XC := (Xu : u 2 C), denoted by Xv ?? Xw j XC , if every path between nodes v and w in G is suitably blocked by the nodes in C.
  • Finding Optimal Tree Decompositions

    Finding Optimal Tree Decompositions

    MSc thesis Computer Science Finding Optimal Tree Decompositions Tuukka Korhonen June 4, 2020 Faculty of Science University of Helsinki Supervisor(s) Assoc. Prof. Matti J¨arvisalo,Dr. Jeremias Berg Examiner(s) Assoc. Prof. Matti J¨arvisalo,Dr. Jeremias Berg Contact information P. O. Box 68 (Pietari Kalmin katu 5) 00014 University of Helsinki,Finland Email address: [email protected].fi URL: http://www.cs.helsinki.fi/ HELSINGIN YLIOPISTO – HELSINGFORS UNIVERSITET – UNIVERSITY OF HELSINKI Tiedekunta — Fakultet — Faculty Koulutusohjelma — Utbildningsprogram — Study programme Faculty of Science Computer Science Tekij¨a— F¨orfattare— Author Tuukka Korhonen Ty¨onnimi — Arbetets titel — Title Finding Optimal Tree Decompositions Ohjaajat — Handledare — Supervisors Assoc. Prof. Matti J¨arvisalo, Dr. Jeremias Berg Ty¨onlaji — Arbetets art — Level Aika — Datum — Month and year Sivum¨a¨ar¨a— Sidoantal — Number of pages MSc thesis June 4, 2020 94 pages, 1 appendix page Tiivistelm¨a— Referat — Abstract The task of organizing a given graph into a structure called a tree decomposition is relevant in multiple areas of computer science. In particular, many NP-hard problems can be solved in polynomial time if a suitable tree decomposition of a graph describing the problem instance is given as a part of the input. This motivates the task of finding as good tree decompositions as possible, or ideally, optimal tree decompositions. This thesis is about finding optimal tree decompositions of graphs with respect to several notions of optimality. Each of the considered notions measures the quality of a tree decomposition in the context of an application. In particular, we consider a total of seven problems that are formulated as finding optimal tree decompositions: treewidth, minimum fill-in, generalized and fractional hypertreewidth, total table size, phylogenetic character compatibility, and treelength.
  • Chapter 6 Inference with Tree-Clustering

    Chapter 6 Inference with Tree-Clustering

    Chapter 6 Inference with Tree-Clustering As noted in Chapters 3, 4, topological characterization of tractability in graphical models is centered on the graph parameter called induced-width or tree-width. In this chap- ter, we take variable elimination algorithms such as bucket-elimination one step further, showing that they are a restricted version of schemes that are based on the notion of tree-decompositions which is applicable across the whole spectrum of graphical models. These methods have received different names in different research areas, such as join-tree clustering, clique-tree clustering and hyper-tree decompositions. We will refer to these schemes by the umbrella name tree-clustering or tree-decomposition. The complexity of these methods is governed by the induced-width, the same parameter that controls the performance of bucket elimination. We will start by extending the bucket-elimination algorithm into an algorithms that process a special class of tree-decompositions, called bucket-trees, and will then move to the general notion of tree-decomposition. 6.1 Bucket-Tree Elimination The bucket-elimination algorithm, BE-bel, for belief updating computes the belief of the first node in the ordering, given all the evidence or just the probability of evidence. However, it is often desirable to answer the belief query for every variable in the network. A brute-force approach will require running BE-bel n times, each time with a different variable order. We will show next that this is unnecessary. By viewing bucket-elimination 103 104 CHAPTER 6. INFERENCE WITH TREE-CLUSTERING as message passing algorithm along a rooted bucket-tree, we can augment it with a second message passing from root to leaves which is equivalent to running the algorithm for each variable separately.

  • Unifying Cluster-Tree Decompositions for Reasoning in Graphical Models∗

    Unifying Cluster-Tree Decompositions for Reasoning in Graphical models¤ Kalev Kask¤, Rina Dechter¤, Javier Larrosa¤¤ and Avi Dechter¤¤¤ ¤Bren School of Information and Computer Science, University of California, Irvine, CA 92697-3425, USA ¤¤ Universitat Politcnica de Catalunya (UPC), Barcelona, Spain ¤¤¤ Business school, California State University, Northridge, CA, USA April 7, 2005 Abstract The paper provides a unifying perspective of tree-decomposition algorithms appearing in various automated reasoning areas such as join-tree clustering for constraint-satisfaction and the clique-tree al- gorithm for probabilistic reasoning. Within this framework, we in- troduce a new algorithm, called bucket-tree elimination (BT E), that extends Bucket Elimination (BE) to trees, and show that it can pro- vide a speed-up of n over BE for various reasoning tasks. Time-space tradeo®s of tree-decomposition processing are analyzed. 1 Introduction This paper provides a unifying perspective of tree-decomposition algorithms that appear in a variety of automated reasoning areas. Its main contribution ¤This work was supported in part by NSF grant IIS-0086529 and by MURI ONR award N0014-00-1-0617 1 is bringing together, within a single coherent framework, seemingly di®erent approaches that have been developed over the years within a number of di®erent communities 1. The idea of embedding a database that consists of a collection of functions or relations in a tree structure and, subsequently, processing it e®ectively by a tree-processing algorithm, has been discovered and rediscovered in dif- ferent contexts. Database researchers observed, almost three decades ago, that relational database schemes that constitute join-trees enable e±cient query processing [22].
  • Graphical Models and Message Passing

    Graphical Models and Message Passing

    Graphical Models and Message passing Yunshu Liu ASPITRG Research Group 2013-07-16 References: [1]. Steffen Lauritzen, Graphical Models, Oxford University Press, 1996 [2]. Michael I. Jordan, Graphical models, Statistical Science, Vol.19, No. 1. Feb., 2004 [3]. Christopher M. Bishop, Pattern Recognition and Machine Learning, Springer-Verlag New York, Inc. 2006 [4]. Daphne Koller and Nir Friedman, Probabilistic Graphical Models - Principles and Techniques, The MIT Press, 2009 [5].Kevin P. Murphy, Machine Learning - A Probabilistic Perspective, The MIT Press, 2012 Outline Preliminaries on Graphical Models Directed graphical model Undirected graphical model Directed graph and Undirected graph Message passing and sum-product algorithm Factor graphs Sum-product algorithms Junction tree algorithm Outline I Preliminaries on Graphical Models I Message passing and sum-product algorithm Preliminaries on Graphical Models Motivation: Curse of dimensionality I Matroid enumeration I Polyhedron computation I Entropic Region I Machine Learning: computing likelihoods, marginal probabilities ··· Graphical Models: Help capturing complex dependencies among random variables, building large-scale statistical models and designing efficient algorithms for Inference. Preliminaries on Graphical Models Definition of Graphical Models: A graphical model is a probabilistic model for which a graph denotes the conditional dependence structure between random variables. MIT : Accepted by MIT Stanford : Accepted by Stanford MIT Example: Suppose MIT and GPA Stanford accepted
  • Graph Properties of DAG Associahedra and Related Polytopes

    Graph Properties of DAG Associahedra and Related Polytopes

    DEGREE PROJECT IN MATHEMATICS, SECOND CYCLE, 30 CREDITS STOCKHOLM, SWEDEN 2018 Graph properties of DAG associahedra and related polytopes JONAS BEDEROFF ERIKSSON KTH ROYAL INSTITUTE OF TECHNOLOGY SCHOOL OF ENGINEERING SCIENCES Graph properties of DAG associahedra and related polytopes JONAS BEDEROFF ERIKSSON Degree Projects in Mathematics (30 ECTS credits) Degree Programme in Mathematics (120 credits) KTH Royal Institute of Technology year 2018 Supervisor at KTH: Liam Solus Examiner at KTH: Svante Linusson TRITA-SCI-GRU 2018:161 MAT-E 2018:25 Royal Institute of Technology School of Engineering Sciences KTH SCI SE-100 44 Stockholm, Sweden URL: www.kth.se/sci GRAPH PROPERTIES OF DAG ASSOCIAHEDRA AND RELATED POLYTOPES JONAS BEDEROFF ERIKSSON Sammanfattning. En riktad acyklisk graf kan betraktas som en representant för relatio- ner av betingat oberoende mellan stokastiska variabler. Ett grundläggande problem inom kausal inferens är, givet att vi får data från en sannolikhetsfördelning som uppfyller en mängd relationer av betingat oberoende, att hitta den underliggande graf som represen- terar dessa relationer. Den giriga SP-algoritmen strävar efter att hitta den underliggande grafen genom att traversera kanter på en polytop kallad DAG-associahedern. Därav spelar kantstrukturen hos DAG-associahedern en stor roll för vår förståelse för den giriga SP- algoritmens komplexitet. I den här uppsatsen studerar vi grafegenskaper hos kantgrafer av DAG-associahedern, relaterade polytoper och deras viktiga delgrafer. Exempel på grafe- genskaper som vi studerar är diameter, radie och center. Våra diameterresultat för DAG- associahedern ger upphov till en ny kausal inferensalgoritm med förbättrade teoretiska på- litlighetsgarantier och komplexitetsbegränsningar jämfört med den giriga SP-algoritmen. Abstract. A directed acyclic graph (DAG) can be thought of as encoding a set of condi- tional independence (CI) relations among random variables.