Graphs and Conditional Independence

Examples Conditional independence Abstract conditional independence Markov properties for undirected graphs

Steﬀen Lauritzen, University of Oxford

CIMPA Summerschool, Hammamet 2011, Tunisia

September 5, 2011

Steﬀen Lauritzen, University of Oxford Graphs and Conditional Independence Examples Conditional independence Abstract conditional independence Markov properties for undirected graphs A directed graphical model

Directed graphical model (Bayesian network) showing relations between risk factors, diseases, and symptoms.

Steﬀen Lauritzen, University of Oxford Graphs and Conditional Independence Examples Conditional independence Abstract conditional independence Markov properties for undirected graphs A pedigree

Graphical model for a pedigree from study of Werner’s syndrome. Each node is itself a graphical model.

p87p99 p98 p88 p89 p90 p100p68p59 p91 p177 p60 p92 p93 p104p182p36 p37 p105p17p153p154p181p78 p18 p77 p49p56 p42 p40 p39 p79 p80 p151p48 p152p85 p174 p183p179p44 p199p28 p27 p51 p84 p70 p15p19 p16 p186p41 p8 p20 p6 p82 p5 p52p61 p62 p75 p2 p178 p185p47 p83 p33 p66 p101p29p65 p102p21 p30 p97 p14 p63 p64 p13 p175

p43p280p314p86 p281 p72p103p288p50p317p295p292 p318p294p31p291 p293p282p25p297p296p343p10p287p73p46p341p193p247p81p358p106 p365p354p355p304 p340p357 p356p148p1 p245p337p306p278p299p171p336p125p279p368 p342p284p11p69p198p170p275p321p12 p26p9 p313p176p259p262p330p235p257p260p244p242p53p329p243p274p34p327p3 p241p248p249p95p271p230p273p228p226p4 p236p348 p234p94p270p7 p22p346p23p298p272p233p326p227p338p307p285p240p325p38p74p303p309 p302p76p239p344p322p323p324p238p195p180

p415p187p311p550p551p310p277 p55p334p96p485p335p45 p67p143 p552p488p487p560p434p305p567 p486p483p578p566p565p481 p568p312p478 p537p145p184p579p246p139p587p580p581p316p191p359p360p361p366p267p423p480p563p527p528p577 p562p522p529p532p530 p424p430 p349p452p427p232p520p194 p422p453p142p456p459p437p372p455p144p436p521p535p192p331p369p258p328 p454p263 p457 p460p505p395p438p477p467p283p458p475p435p558p557p469p251p173p396 p465p463 p559p286p541p265p470p538p461p433p141p426p431p471p24p231p253p300p503p525p269p416p428p429p531p229p264p462p419p504p472p169p250p301p448p449p417p254p494p495p445p266p497 p418 p255p261 p493p252p450p237p491 p492p443p447p444p564p379p539p381p466p376p155p347p308p482p561p588p352p351p290p196p136 p289 p350p339

p553p353p554p315p911p523p910p332p573 p524p526p574p907p276 p912 p333p507p895p889 p908p156p909p137 p575p913p484p190p882p345p383p548p320p938p555 p547p546p620p140p549p189p939p948p940 p545p946p941p582 p451p929p891p388p618p35p386p619p883p902p595p134 p896p363p924p425 p926 p135p928 p384 p925p585p914p420p536 p421p594p440p881p542p892p377p598p888p543p893p597 p596p508p364 p880p608p512p432p625p714p654p904p544p906 p441p138p514p511p397p510p936p905p385p609p921p506p509p622p637p937p887p927p446p479 p920 p606p367p930p468p934p901p917p373p256 p918p916p915 p935p919p621p626p540p571p474p636 p886p723p884p533p501p890p502p931p656p150p500p933p885p632p168p603p645p644p623p268p688p633p903p607 p605p515p387p572 p655p518p593p513 p517p392p464p394p439p635p146p393 p687p390p599p556 p686p378p391p200p473p638p602p398 p685 p516p684p206p771p496p172p628p399p768p770p629p534p570 p627p569 p630p374p932p197p576 p202 p769

p942p205p203p223p208p1585p702p114p701p900p989p898p490p982p981p726p703p204p959p923p984p993p738 p899 p983p728p980p760p727p489 p985p1004p664p1006p682p1003p951p1017p1077p584p955p943p894p922p757p211p992p954p750p499p945p991p161p944p165p986p631p147p949p362p1182 p987 p953 p958p947p319p950 p952p677p613p731p1070p956p129p442 p957p1078p1020p201p370p1014p1122p754p970p735p704p737p162p734p973p118p1124p1016p966p611p968p1042p1043p616 p1123p586 p971p614p212p740p742p743p676p617p744 p1015p749p1075p1013p666p711p710p1038p733p967p709p753p659p660p1035p1588p706 p612p1101p988p662p712p683 p1018p159p1111p1097 p222p1100p716p476p1096p1008p1045p657p1007p604p1036p640p1074p1039p1099 p1009 p643p1055 p1037p1048p641p1071p590p610p1027p218p836p673p962p1054p1026p1024 p1022 p1010p1053 p639p1073p1098p679p976p977p975p725p678 p1049p1052 p1025 p724p680p979p715p157p960p974p897p961p965p729p642p130 p978 p1011p408p1072p1040p752p1044p1041p407p751 p409p160p651 p1051p648p401p601p404p838p382p1021p1023p403p1083p661 p405p719p389p600p658p674p498 p1012p730p624p739 p718p380p667 p406p964p1080p672 p649 p668 p1031p720p721p402p1076 p717p120p110 p722p1587 p671p759 p132p113p375p1079p221p1586p210p400

p216p217p1558p995p163p873p1459p1509p111p839p777p775p793p788p1127p1508p787p1267p990p1458p149p772p790p998p841p789p1375p792p1005p647p1455p1191p220p1372p774p1374 p1510 p791p1368p785 p786p1271p1332p225p1376p669p1371 p1266p1370p1066p1187p1086 p994p1069 p207p1063 p997p1269p1064p755p782p1194p1068p1144p776p1193 p1061 p1270p1084p1060p1148p1373 p646p1065p1454p1057p1337p1369 p1333p1002p1593p1336p1335p1149 p1265p1001p1087p1272p1456p1140p116p1121p784p1329p1059 p1088p778 p1338p1125p779p689p848p1203 p1085p996 p1334p1204p1331 p1268p1142p371 p1330p1190p857p213p1520p854p1126p1521p736p867p107p732p1457p855p1185p844p1145p780p1062 p846p1143p1141p1000p1578p834p1186p802p1067p1209p1184p1146p969p856p188p783p835p615p797p1028p1197p1046 p801p781p696p1219p1202p1147p1205p1607 p833p1207p1606p697p758p1198p1188p663p695p705p1611 p1208 p1047 p1206 p1605p798p1613p1189p741p1104p999p707 p803p1092p1105p1609p1030p866p1608p1108 p1612 p1577 p800p1110p691p1614p1177p690p1106p1107p1102p699p837p1128p1109p158p215p1175p1176p963p1103p670p1135p1139p874p1130 p1610p1174 p799 p692 p698p858p1019p796p1029p127p1050p1171p209p410p1137p1419p1199p1132p1136p1201p1133p1094p1095p1134p1243p681p166p117p794p589p1495p1183p634p1232p713p694p1032p1056p828p1230p1089p700 p795p1218p591p1181p1034p1033p652p214p708p830p827p1231p1093p1421p761p412p519p1172p1058p1200p1082 p829 p1418p1090p109p1229p762p1223p1228p764p763p1170p1227p122p748 p831p119p1091p1420p767p653p766p852p745p746p747 p693p1081p826p650 p1244 p832p1436p1129p112p972 p1417p1225p1590p1168p843p1438p1169p1435p1180p1437p1166p1165p1220p765p1163p1162p414p413p1226p1434p1222p864 p224p1221p871p1592 p851p1589p1138p1439p123p1349p1591 p850 p411p1161p1179p131p1173p845p1224p853

p1260p1392p1394p1512p1286p1490p1294p128p1393p1388p1391p1396p1389p121p1390p1544p1464p1291p863p1293p1289p1292p1288p1328p1257p1450p1451p1453p1287p1449 p1261p879 p1290p1415p1403p108 p1262p1506p773p133p1264p1326p1256p840p1463p1192p1411 p1263p1195p1246p1395p1255p1414 p1258p1452p1250p1440p1327p1584p1410p1252p1253p1245p1402p1466 p1524p1523p1522 p1412 p1441p1325p1311 p1416 p1259p1405p1150 p1254p1404p1308 p1413p1249p1473p1247p1505p1504p1345 p1400p1155p1497p1348 p1251 p1248 p1344p756 p1401 p1324p1409 p1467p1579p1283p1498p1566p1275p1304p1279p1340p1210 p1496p878p1399p1342 p1347p1407p1550p1276p1406 p1341 p1339 p1343p1233 p1346p1427p1277p1474p1431p1397p126 p1274 p1478p1309p1433p1429p1299p1428p1468p583 p1398p1280 p1282p1285p1323p1284p1281p1408p1432 p1307p823p1297p1296p1430p818p819 p824p849p820p1196p1319p1300p1237p822 p1273 p825p1301p1320 p1318p1242p1302p1321p1313p167 p821p1310p1303p814p1322p1298p1422 p1551p1315 p1317p812p1239p817p1316p816 p1314p1295p1306p1442p1426p1278p1352p860p1541p1350p1241p862p1113p1312 p813p1240p665 p859p1443p1236 p815p1114p1238 p1305p1569p1530p1235p1213p1465p870p1517p811p847p869p1447p1131p1537p1462p1117p1519p592p1516p1351p1120p1115p1445p1444p1538 p1518p1152p1423p1118 p1446p1424p1159 p1116p1153p1158p877p1425p1156p1160p164p1476 p1151p1448p876p1234p872p1555p1212p861p1211 p1214p1154p1477p865p1178p1556p1565p1216p1217p1489p868p1564p1112p1472p875p1470p1471p810p1119p1380p1461p1557p1540 p1215p675p808p807p1387p1378p806p804p1601p1386p1552p1377p1381 p805p1603p1595p1167p1460p842 p1539p1525p1385p1600p1164p1599p219p1594p1583p1597 p1529p1382 p809p1596 p1563p1598p115p1383p1379p1157p1384p1514p1527p1526p1602p1488p1567p1528p1604p124p1486p1487

p1560p1511p1484 p1559p1572p1507p1492p1571p1494 p1493p1562p1491p1570p1574p1535p1502p1500p1503p1576 p1501p1549p1548p1469p1480p1533p1582p1534p1536p1475p1361p1362p1482p1483p1360 p1481p1547p1546p1353p1355p1356p1357p1358p1359 p1354p1545p1575p1513p1479p1531p1542p1485p1561p1499p1367p1365p1532p1366p1363p1364p1573p1515p1568p1580p1581p1554p1553 p1543 Family relationship of 1641 members of Greenland Eskimo population.

P(X ∈ A | Y = y) = P(X ∈ A) or, equivalently, if

P{(X ∈ A) ∩ (Y ∈ B)} = P(X ∈ A)P(Y ∈ B).

For discrete variables this is equivalent to

pij = pi+p+j P where pij = P(X = i, Y = j) and pi+ = j pij etc., whereas for continuous variables the requirement is a factorization of the joint density: fXY (x, y) = fX (x)fY (y). When X and Y are independent we write X ⊥⊥ Y .

Random variables X and Y are conditionally independent given the random variable Z if

L(X | Y , Z) = L(X | Z).

We then write X ⊥⊥ Y | Z (or X ⊥⊥P Y | Z) Intuitively: Knowing Z renders Y irrelevant for predicting X . Factorisation of densities:

X ⊥⊥ Y | Z ⇐⇒ fXYZ (x, y, z)fZ (z) = fXZ (x, z)fYZ (y, z) ⇐⇒ ∃a, b : f (x, y, z) = a(x, z)b(y, z).

2 4

@ @ 1 u 5 u 7 @@ @@ @ @ u u u @@ @@ 3u 6 u For several variables, complex systems of conditional independence can for example be described by undirected graphs. Then a set of variables A is conditionally independent of set B, given the values of a set of variables C if C separates A from B. For example in picture above

1 ⊥⊥{4, 7} | {2, 3}, {1, 2} ⊥⊥ 7 | {4, 5, 6}.

Directed graphs are also natural models for conditional indpendence: 2 4 - @ @ 1 u 5 u 7 @@R @-@R @ @ u u u @@R @-@R 3u 6 u Any node is conditional independent of its non-descendants, given its immediate parents. So, for example, in the above picture we have 5 ⊥⊥{1, 4} | {2, 3}, 6 ⊥⊥{1, 2, 4} | {3, 5}.

Steﬀen Lauritzen, University of Oxford Graphs and Conditional Independence Examples Independence Conditional independence Formal deﬁnition Abstract conditional independence Fundamental properties Markov properties for undirected graphs Proof of (C5): We have X ⊥⊥ Y | (Z, W ) ⇒ f (x, y, z, w) = a(x, z, w)b(y, z, w). Similarly X ⊥⊥ Z | (Y , W ) ⇒ f (x, y, z, w) = g(x, y, w)h(y, z, w). If f (x, y, z, w) > 0 for all (x, y, z, w) it thus follows that g(x, y, w) = a(x, z, w)b(y, z, w)/h(y, z, w).

The left-hand side does not depend on z. So for fixed z = z0: g(x, y, w) = ã(x, w)b˜(y, w). Insert this into the second expression for f to get f (x, y, z, w) = ã(x, w)b˜(y, w)h(y, z, w) = a∗(x, w)b∗(y, z, w) which shows X ⊥⊥ (Y , Z) | W .

Steﬀen Lauritzen, University of Oxford Graphs and Conditional Independence Examples Conditional independence Graphoids and semi-graphoids Abstract conditional independence Examples Markov properties for undirected graphs Conditional independence can be seen as encoding abstract irrelevance. With the interpretation: Knowing C, A is irrelevant for learning B, (C1)–(C4) translate into: (I1) If, knowing C, learning A is irrelevant for learning B, then B is irrelevant for learning A; (I2) If, knowing C, learning A is irrelevant for learning B, then A is irrelevant for learning any part D of B; (I3) If, knowing C, learning A is irrelevant for learning B, it remains irrelevant having learnt any part D of B; (I4) If, knowing C, learning A is irrelevant for learning B and, having also learnt A, D remains irrelevant for learning B, then both of A and D are irrelevant for learning B. The property analogous to (C5) is slightly more subtle and not generally obvious.

An independence model ⊥σ is a ternary relation over subsets of a ﬁnite set V . It is graphoid if for all subsets A, B, C, D:

(S1) if A ⊥σ B | C then B ⊥σ A | C (symmetry);

(S2) if A ⊥σ (B ∪ D) | C then A ⊥σ B | C and A ⊥σ D | C (decomposition);

(S3) if A ⊥σ (B ∪ D) | C then A ⊥σ B | (C ∪ D) (weak union);

(S4) if A ⊥σ B | C and A ⊥σ D | (B ∪ C), then A ⊥σ (B ∪ D) | C (contraction);

(S5) if A ⊥σ B | (C ∪ D) and A ⊥σ C | (B ∪ D) then A ⊥σ (B ∪ C) | D (intersection). Semigraphoid if only (S1)–(S4) holds. It is compositional if also

(S6) if A ⊥σ B | C and A ⊥σ D | C then A ⊥σ (B ∪ D) | C (composition).

Let G = (V , E) be ﬁnite and simple undirected graph (no self-loops, no multiple edges).

For subsets A, B, S of V , let A ⊥G B | S denote that S separates A from B in G, i.e. that all paths from A to B intersect S.

Fact: The relation ⊥G on subsets of V is a compositional graphoid. This fact is the reason for choosing the name ‘graphoid’ for such independence model.

For a system V of labeled random variablesX v , v ∈ V , we use the shorthand A ⊥⊥ B | C ⇐⇒ XA ⊥⊥ XB | XC ,

where XA = (Xv , v ∈ A) denotes the variables with labels in A. The properties (C1)–(C4) imply that ⊥⊥ satisﬁes the semi-graphoid axioms for such a system, and the graphoid axioms if the joint density of the variables is strictly positive. A regular multivariate Gaussian distribution, deﬁnes a compositional graphoid independence model.

Let L, M, and N be linear subspaces of a Hilbert space H and

L ⊥ M | N ⇐⇒ (L N) ⊥ (M N),

where L N = L ∩ N⊥.L and M are said to meet orthogonally in N. (O1) If L ⊥ M | N then M ⊥ L | N; (O2) If L ⊥ M | N and U is a linear subspace of L, then U ⊥ M | N; (O3) If L ⊥ M | N and U is a linear subspace of M, then L ⊥ M | (N + U); (O4) If L ⊥ M | N and L ⊥ R | (M + N), then L ⊥ (M + R) | N. Intersection does not hold in general whereas composition (S6) does. Steﬀen Lauritzen, University of Oxford Graphs and Conditional Independence Examples Conditional independence Deﬁnitions Abstract conditional independence Structural relations among Markov properties Markov properties for undirected graphs

G = (V , E) simple undirected graph; An independence model ⊥σ satisﬁes (P) the pairwise Markov property if

α 6∼ β ⇒ α ⊥σ β | V \{α, β};

(L) the local Markov property if

∀α ∈ V : α ⊥σ V \ cl(α) | bd(α);

(G) the global Markov property if

A ⊥G B | S ⇒ A ⊥σ B | S.

Steﬀen Lauritzen, University of Oxford Graphs and Conditional Independence Examples Conditional independence Deﬁnitions Abstract conditional independence Structural relations among Markov properties Markov properties for undirected graphs Pairwise Markov property

2 4

@ @ 1 u 5 u 7 @@ @@ @ @ u u u @@ @@ 3u 6 u Any non-adjacent pair of random variables are conditionally independent given the remaning. For example, 1 ⊥σ 5 | {2, 3, 4, 6, 7} and 4 ⊥σ 6 | {1, 2, 3, 5, 7}.

2 4

@ @ 1 u 5 u 7 @@ @@ @ @ u u u @@ @@ 3u 6 u Every variable is conditionally independent of the remaining, given its neighbours. For example, 5 ⊥σ {1, 4} | {2, 3, 6, 7} and 7 ⊥σ {1, 2, 3} | {4, 5, 6}.

2 4

@ @ 1 u 5 u 7 @@ @@ @ @ u u u @@ @@ 3u 6 u To ﬁnd conditional independence relations, one should look for separating sets, such as {2, 3}, {4, 5, 6}, or {2, 5, 6} For example, it follows that 1 ⊥σ 7 | {2, 5, 6} and 2 ⊥σ 6 | {3, 4, 5}.

For any semigraphoid it holds that

(G) ⇒ (L) ⇒ (P)

If ⊥σ satisﬁes graphoid axioms it further holds that

(P) ⇒ (G) so that in the graphoid case

(G) ⇐⇒ (L) ⇐⇒ (P).

The latter holds in particular for ⊥⊥ , when f (x) > 0.

(G) implies (L) because bd(α) separates α from V \ cl(α). Assume (L). Then β ∈ V \ cl(α) because α 6∼ β. Thus

bd(α) ∪ ((V \ cl(α)) \{β}) = V \{α, β},

Hence by (L) and weak union (S3) we get that

α ⊥σ (V \ cl(α)) | V \{α, β}.

Decomposition (S2) then gives α ⊥σ β | V \{α, β} which is (P).

(P) ⇒ (G) for graphoids: Assume (P) and A ⊥G B | S. We must show A ⊥σ B | S. Wlog assume A and B non-empty. Proof is reverse induction on n = |S|. If n = |V | − 2 then A and B are singletons and (P) yields A ⊥σ B | S directly. Assume |S| = n < |V | − 2 and conclusion established for |S| > n: First assume V = A ∪ B ∪ S. Then either A or B has at least two elements, say A. If α ∈ A then B ⊥G (A \{α}) | (S ∪ {α}) and also α ⊥G B | (S ∪ A \{α}) (as ⊥G is a semi-graphoid). Thus by the induction hypothesis

(A \{α}) ⊥σ B | (S ∪ {α}) and {α} ⊥σ B | (S ∪ A \{α}).

Now intersection (S5) gives A ⊥σ B | S.

For A ∪ B ∪ S ⊂ V we choose α ∈ V \ (A ∪ B ∪ S). Then A ⊥G B | (S ∪ {α}) and hence the induction hypothesis yields A ⊥σ B | (S ∪ {α}). Further, either A ∪ S separates B from {α} or B ∪ S separates A from {α}. Assuming the former gives α ⊥σ B | A ∪ S.

Using intersection (S5) we get (A ∪ {α}) ⊥σ B | S and from decomposition (S2) we derive that A ⊥σ B | S. The latter case is similar.

Steﬀen Lauritzen, University of Oxford Graphs and Conditional Independence