FeedRank: A Tamper-resistant Method for the Ranking of Cyber Threat Intelligence Feeds
Roland Meier(1), Cornelia Scherrer(1), David Gugelmann(2), Vincent Lenders(3), Laurent Vanbever(1)
(1) (2) (3)
armasuisse WorldTravel
hotel in Amsterdam, flights to Amsterdam, vacation in Amsterdam, trip to Amsterdam, hotel in Andorra la Vella, flights to Andorra la Vella, vacation in Andorra la Vella, trip to Andorra la Vella, hotel in Ankara, flights to Ankara, vacation in Ankara, trip to Ankara, hotel in Astana, flights to Astana, vacation in Astana, trip to Astana, hotel in Athens, flights to Athens, vacation in Athens, trip to Athens, hotel in Baku, flights to Baku, vacation in Baku, trip to Baku, hotel in Belgrade, flights to Belgrade, vacation in Belgrade, trip to Belgrade, hotel in Berlin, flights to Berlin, vacation in Berlin, trip to Berlin, hotel in Bern, flights to Bern, vacation in Bern, trip to Bern, hotel in Bratislava, flights to Bratislava, vacation in Bratislava, trip to Bratislava, hotel in Brussels, flights to Brussels, vacation in Brussels, trip to Brussels, hotel in Bucharest, flights to Bucharest, vacation in Bucharest, trip to Bucharest, hotel in Budapest, flights to Budapest, vacation in Budapest, trip to Budapest, hotel in Chişinău, flights to Chişinău, vacation in Chişinău, trip to Chişinău, hotel in Copenhagen, flights to Copenhagen, vacation in Copenhagen, trip to Copenhagen, hotel in Dublin, flights to Dublin, vacation in Dublin, trip to Dublin, hotel in Helsinki, flights to Helsinki, vacation in Helsinki, trip to Helsinki, hotel in Kiev, flights to Kiev, vacation in Kiev, trip to Kiev, hotel in Lisbon, flights to Lisbon, vacation in Lisbon, trip to Lisbon, hotel in Ljubljana, flights to Ljubljana, vacation in Ljubljana, trip to Ljubljana, hotel in London, flights to London, vacation in London, trip to London, hotel in Luxembourg, flights to Luxembourg, vacation in Luxembourg, trip to Luxembourg, hotel in Madrid, flights to Madrid, vacation in Madrid, trip to Madrid, hotel in Minsk, flights to Minsk, vacation in Minsk, trip to Minsk, hotel in Monaco, flights to Monaco, vacation in Monaco, trip to Monaco, hotel in Moscow, flights to Moscow, vacation in Moscow, trip to Moscow, hotel in Nicosia, flights to Nicosia, vacation in Nicosia, trip to Nicosia, hotel in Oslo, flights to Oslo, vacation in Oslo, trip to Oslo, hotel in Paris, flights to Paris, vacation in Paris, trip to Paris, hotel in Podgorica, flights to Podgorica, vacation in Podgorica, trip to Podgorica, hotel in Prague, flights to Prague, vacation in Prague, trip to Prague, hotel in Pristina, flights to Pristina, vacation in Pristina, trip to Pristina, hotel in Reykjavík, flights to Reykjavík, vacation in Reykjavík, trip to Reykjavík, hotel in Riga, flights to Riga, vacation in Riga, trip to Riga, hotel in Rome, flights to Rome, vacation in Rome, trip to Rome, hotel in San Marino, flights to San Marino, vacation in San Marino, trip to San Marino, hotel in Sarajevo, flights to Sarajevo, vacation in Sarajevo, trip to Sarajevo, hotel in Skopje, flights to Skopje, vacation in Skopje, trip to Skopje, hotel in Sofia, flights to Sofia, vacation in Sofia, trip to Sofia, hotel in Stockholm, flights to Stockholm, vacation in Stockholm, trip to Stockholm, hotel in Tallinn, flights to Tallinn, vacation in Tallinn, trip to Tallinn, hotel in Tbilisi, flights to Tbilisi, vacation in Tbilisi, trip to Tbilisi, hotel in Tirana, flights to Tirana, vacation in Tirana, trip to Tirana, hotel in Vaduz, flights to Vaduz, vacation in Vaduz, trip to Vaduz, hotel in Valletta, flights to Valletta, vacation in Valletta, trip to Valletta, hotel in Vatican City, flights to Vatican City, vacation in Vatican City, trip to Vatican City, hotel in Vienna, flights to Vienna, vacation in Vienna, trip to Vienna, hotel in Vilnius, flights to Vilnius, vacation in Vilnius, trip to Vilnius, hotel in Warsaw, flights to Warsaw, vacation in Warsaw, trip to Warsaw, hotel in Yerevan, flights to Yerevan, vacation in Yerevan, trip to Yerevan, hotel in Zagreb, flights to Zagreb, vacation in Zagreb, trip to Zagreb
hotel in Amsterdam, flights to Amsterdam, vacation in Amsterdam, trip to Amsterdam, hotel in Andorra la Vella, flights to Andorra la Vella, vacation in Andorra la Vella, trip to Andorra la Vella, hotel in Ankara, flights to Ankara, vacation hotel in Amsterdam, flights to Amsterdam, vacation in Amsterdam, trip to Amsterdam, hotel in Andorra la Vella, flights to Andorra la Vella, vacation in Andorra la Vella, trip to Andorra la Vella, hotel in Ankara, flights to Ankara, vacation in Ankara, trip to Ankara, hotel in Astana, flights to Astana, vacation in Astana, trip to Astana, hotel in Athens, flights to Athens, vacation in Athens, trip to Athens, hotel in Baku, flights to Baku, vacation in Baku, trip to Baku, hotel in Belgrade, flights to Belgrade, vacation in Belgrade, trip to Belgrade, hotel in Berlin, flights to Berlin, vacation in Berlin, trip to Berlin, hotel in Bern, flights to Bern, vacation in Bern, trip to Bern, hotel in Bratislava, flights to Bratislava,invacation Ankara,in Bratislava, trip to Bratislava, hoteltripin Brussels, flightstoto BrusselsAnkara,, vacation in Brussels, trip tohotelBrussels, in Astana, flights hotel in Bucharest, flights to Bucharest, vacation in Bucharest, trip to Bucharest, hotel in Budapest, flights to Budapest, vacation in Budapest, trip to Budapest, hotel in Chişinău, flights to Chişinău, vacation in Chişinău, trip to Chişinău, hotel in Copenhagen, flights to Copenhagen, vacation in Copenhagen, trip to Copenhagen, hotel in Dublin, flights to Dublin, vacation in Dublin, trip to Dublin, hotel in Helsinki, flights to Helsinki, vacation in Helsinki, trip to Helsinki, hotel in Kiev, flights to Kiev, vacation in Kiev, trip to Kiev, hotel in Lisbon, flights to Lisbon, vacation in Lisbon, trip to Lisbon, hotel in Ljubljana, flights to Ljubljana, vacation in Ljubljana, trip to Ljubljana, hotel in London, flights to London, vacation in London, trip to London, hotel in Luxembourg, flights to Luxembourg, vacation in Luxembourg, trip to Luxembourg, hotel in Madrid, flights to Madrid, vacation in Madrid, trip to Madrid, hotel in Minsk, flights to Minsk, vacation in Minsk, trip to Minsk, hotel intoMonaco, flightsAstana,to Monaco, vacation in Monaco,vacationtrip to Monaco, hotel in Moscowin, flightsAstana,to Moscow, vacation in trip to Astana Moscow, trip to Moscow, hotel in Nicosia, flights to Nicosia, vacation in Nicosia, trip to Nicosia, hotel in Oslo, flights to Oslo, vacation in Oslo, trip to Oslo, hotel in Paris, flights to Paris, vacation in Paris, trip to Paris, hotel in Podgorica, flights to Podgorica, vacation in Podgorica, trip to Podgorica, hotel in Prague, flights to Prague, vacation in Prague, trip to Prague, hotel in Pristina, flights to Pristina, vacation in Pristina, trip to Pristina, hotel in Reykjavík, flights to Reykjavík, vacation in Reykjavík, trip to Reykjavík, hotel in Riga, flights to Riga, vacation in Riga, trip to Riga, hotel in Rome, flights to Rome, vacation in Rome, trip to Rome, hotel in San Marino, flights to San Marino, vacation in San Marino, trip to San Marino, hotel in Sarajevo, flights to Sarajevo, vacation in Sarajevo, trip to Sarajevo, hotel in Skopje, flights to Skopje, vacation in Skopje, trip to Skopje, hotel in Sofia, flights to Sofia, vacation in Sofia, trip to Sofia, hotel in Stockholm, flights to Stockholm, vacation in Stockholm, trip to Stockholm, hotel in Tallinn, flights to Tallinn, vacation in Tallinn, trip to Tallinn, hotel in Tbilisi, flights to Tbilisi, vacation in Tbilisi, trip to Tbilisi, hotel in Tirana, flights to Tirana, vacation in Tirana, trip to Tirana, hotel in Vaduz, flights to Vaduz, vacation in Vaduz, trip to Vaduz, hotel in Valletta, flights to Valletta, vacation in Valletta, trip to Valletta, hotel in Vatican City, flights to Vatican City, vacation in Vatican City, trip to Vatican City, hotel in Vienna, flights to Vienna, vacation in Vienna, trip to Vienna, hotel in Vilnius, flights to Vilnius, vacation in Vilnius, trip to Vilnius, hotel in Warsaw, flights to Warsaw, vacation in Warsaw, trip to Warsaw, hotel in Yerevan, flights to Yerevan, vacation in Yerevan, trip to Yerevan, hotel in Zagreb, flights to Zagreb, vacation in Zagreb, trip to Zagreb Ranking Websites
§ Websites can contain arbitrary content § Websites can contain links to any other website § PageRank: A website to which many other websites refer to is likely to be important
4 103.43.75.87 104.131.182.103 104.200.22.130 104.200.23.95 From Websites... 107.170.20.33 107.181.174.34 109.234.35.123 109.234.35.128 109.234.35.75 109.237.111.168 109.248.222.47 138.201.118.102 138.201.93.46 138.201.95.72 139.59.147.0 141.8.226.58 WorldTravel 146.120.110.46 146.185.137.40 146.185.249.189 148.163.73.29 149.154.152.108 149.154.157.14 149.154.159.179 149.154.68.190 149.202.109.202 149.202.52.215 151.236.14.51 151.236.15.226 158.255.6.109 158.255.6.115 158.69.223.5 164.132.40.47 176.107.185.19 176.121.14.95 176.31.47.100 176.53.21.105 176.9.172.166 178.62.232.244 178.63.238.185 185.100.85.150 185.102.136.67 185.102.136.77 185.106.122.38 185.115.140.170 185.117.153.176 185.117.72.105 185.117.72.94 185.127.25.176 185.129.148.6 185.14.28.30 185.14.29.188 185.14.29.64 185.14.30.97 185.141.25.108 185.141.25.150 185.146.169.16 185.159.128.119 185.162.8.101 185.162.8.94 185.17.120.130 185.179.190.31 185.22.67.108 185.22.67.27 185.46.11.239 185.46.11.73 185.5.250.135 185.67.2.156 185.75.46.122 185.75.46.4 185.75.46.73 185.80.148.137 185.82.216.213 185.82.216.45 185.82.217.29 185.86.78.3 185.92.220.35 188.120.239.230 188.127.230.60 188.127.231.116 188.127.231.124
hotel in Amsterdam, flights to Amsterdam, vacation in Amsterdam, trip to Amsterdam, hotel in Andorra la Vella188.127.239.53, flights to Andorra la Vella, vacation in Andorra la Vella, trip to Andorra188.127.251.99la Vella, hotel in Ankara, flights to Ankara, vacation in 188.138.88.184 188.166.168.250 Ankara, trip to Ankara, hotel in Astana, flights to Astana, vacation in Astana, trip to Astana, hotel in Athens, flights to Athens, vacation in Athens, trip to Athens, hotel in Baku, flights to Baku, vacation in Baku, trip to Baku, hotel in Belgrade, flights to Belgrade, vacation in Belgrade, trip to Belgrade, hotel in Berlin, flights to Berlin, vacation in Berlin, trip to Berlin, hotel in Bern, flights to Bern, vacation in Bern, trip to Bern, hotel in Bratislava, flights to Bratislava, vacation in Bratislava, trip to Bratislava, hotel in Brussels, flights to Brussels, vacation in Brussels, trip to Brussels, 192.121.16.196hotel in Bucharest, flights to Bucharest, vacation in Bucharest , trip to192.71.213.69Bucharest, hotel in Budapest, flights to Budapest, vacation in193.0.178.74 193.124.180.6 193.124.185.87 Budapest, trip to Budapest, hotel in Chişinău, flights to Chişinău, vacation in Chişinău, trip to Chişinău, hotel in Copenhagen, flights to Copenhagen, vacation in Copenhagen, trip to Copenhagen, hotel in Dublin, flights to Dublin, vacation in Dublin, trip to Dublin, hotel in Helsinki, flights to Helsinki, vacation in Helsinki, trip to Helsinki, hotel in Kiev, flights to Kiev, vacation in Kiev, trip to Kiev, hotel in Lisbon, flights to Lisbon, vacation in Lisbon, trip to Lisbon, hotel in Ljubljana, flights to Ljubljana, vacation in Ljubljana, trip to Ljubljana, hotel in London, flights to London, vacation in London,193.32.68.48trip to London, hotel in Luxembourg, flights to Luxembourg, 193.70.86.51vacation in Luxembourg, trip to Luxembourg, hotel in Madrid,193.9.28.13flights to 193.9.28.16 193.9.28.254 Madrid, vacation in Madrid, trip to Madrid, hotel in Minsk, flights to Minsk, vacation in Minsk, trip to Minsk, hotel in Monaco, flights to Monaco, vacation in Monaco, trip to Monaco, hotel in Moscow, flights to Moscow, vacation...in Moscow ,to Cyber Threat trip to Moscow, hotel in Nicosia, flights to Nicosia, vacation in Nicosia, trip to Nicosia, hotel in Oslo, flights to Oslo, vacation in Oslo, trip to Oslo, hotel in Paris, flights to Paris, vacation in Paris, trip to Paris, hotel in Podgorica, flights to Podgorica, vacation in Podgorica, trip to Podgorica, hotel in Prague, flights to Prague, vacation in Prague193.9.28.49, trip to Prague, hotel in Pristina, flights to Pristina, vacation194.1.236.126in Pristina, trip to Pristina, hotel in Reykjavík, flights to Reykjavík,194.28.87.26vacation in 194.31.59.5 194.58.56.105 Reykjavík, trip to Reykjavík, hotel in Riga, flights to Riga, vacation in Riga, trip to Riga, hotel in Rome, flights to Rome, vacation in Rome, trip to Rome, hotel in San Marino, flights to San Marino, vacation in San Marino, trip to San Marino, hotel in Sarajevo, flights to Sarajevo, vacation in Sarajevo, trip to Sarajevo, hotel in Skopje, flights to Skopje, vacation in Skopje, trip to Skopje, hotel in Sofia, flights to Sofia, vacation in Sofia, trip to Sofia, hotel in Stockholm, flights to Stockholm, vacation in Stockholm, trip to Stockholm, hotel in Tallinn, flights to Tallinn, vacation in Tallinn, trip194.58.56.162to Tallinn, hotel in Tbilisi, flights to Tbilisi, vacation in Tbilisi , trip195.123.211.6to Tbilisi, hotel in Tirana, flights to Tirana, vacation in Tirana, Intelligencetrip195.123.218.226to Tirana, Feeds195.154.241.208 195.154.69.90 hotel in Vaduz, flights to Vaduz, vacation in Vaduz, trip to Vaduz, hotel in Valletta, flights to Valletta,195.19.192.99vacation in Valletta, trip to Valletta, hotel in Vatican City, flights 195.43.95.198to Vatican City, vacation in Vatican City, trip to Vatican City, hotel195.64.154.114in Vienna, 195.64.154.126 195.64.154.14 Ranking Cyber Threat Intelligence Feeds
51.254.240.45 91.234.34.98 185.22.67.27 49.154.152.108 88.214.237.57 185.82.216.213 81.177.181.164 § Feeds can contain arbitrary entries 89.108.84.132 91.219.29.81 91.219.31.15 95.163.107.41 § Feeds can copy entries from any other feed 91.209.77.86 § FeedRank: A feed whose entries are confirmed by other 185.82.216.213 89.108.84.132 feeds is likely to be of high quality 37.139.30.95 185.22.67.108 91.219.29.81 107.170.20.33 91.239.235.130 83.217.26.168 46.38.52.225
46.165.253.93 149.154.157.14 91.239.235.130 82.202.221.108 83.217.26.168 107.170.20.33 46.38.52.225 91.234.34.98 149.154.12.108 6 FeedRank
Evaluate cyber threat intelligence feeds in a way that
§ Allows us to identify high quality feeds
§ Is customizable for different preferences of network defenders
§ Does not require a ground truth
§ Scales to the large ecosystem of feeds
§ Is robust against dishonest feed providers Properties of High Quality Feeds
§ Completeness Contain all malicious endpoints
§ Accuracy Do not list benign endpoints
§ Speed Be complete and accurate upon changes
8 FeedRank Operates in 3 Steps
Input Analysis Output
Feed Collection Correlation FeedRank A A B B A C D C t0 C t1 t0 t1 Contribution B D D t0 A C t1 B
t0 t1 D
9 Step 1: Collect Feed Data
Input Analysis Output
Feed Collection Correlation FeedRank A A B B A C D C t0 C t1 t0 t1 Contribution B D D t0 A C t1 B
t0 t1 D
10 Step 2a: Analyze Correlations Between Feeds
Input Analysis Output
Feed Collection Correlation FeedRank A A B B A C D C t0 C t1 t0 t1 Contribution B D D t0 A C t1 B
t0 t1 D
11 Step 2a: Analyze Correlations Between Feeds
§ Correlation Graph: Directed graph with § vertices ← feeds § edges ← correlations
12 Step 2a: Analyze Correlations Between Feeds
§ Correlation Graph: Directed graph with t=0 § vertices ← feeds § edges ← correlations Feed A 1,2,3,4
Feed B 5,6,7
13 Step 2a: Analyze Correlations Between Feeds
§ Correlation Graph: Directed graph with t=0 t=1 § vertices ← feeds § edges ← correlations Feed A Feed A 1,2,3,4 1,2,3,4
Feed B Feed B 5,6,7 3,4,5,6,7
14 Step 2a: Analyze Correlations Between Feeds
§ Correlation Graph: Directed graph with t=0 t=1 § vertices ← feeds § edges ← correlations Feed A Feed A 1,2,3,4 1,2,3,4
Feed B Feed B 5,6,7 3,4,5,6,7
15 Step 2a: Analyze Correlations Between Feeds
§ Correlation Graph: Directed graph with t=0 t=1 § vertices ← feeds § edges ← correlations Feed A Feed A 1,2,3,4 1,2,3,4
50%
Feed B Feed B 5,6,7 3,4,5,6,7
16 Step 2b: Determine the Contribution of Each Feed
Input Analysis Output
Feed Collection Correlation FeedRank A A B B A C D C t0 C t1 t0 t1 Contribution B D D t 0 A C t1 B
t0 t1 D
17 Step 2b: Determine the Contribution of Each Feed
§ Measure the percentage of entries that originated from each feed
18 Step 2b: Determine the Contribution of Each Feed
§ Measure the percentage of entries that originated from each feed
All listed endpoints (IPs)
19 Step 2b: Determine the Contribution of Each Feed
§ Measure the percentage of entries B that originated from each feed
50% contributed by feed A C D
20 Step 3: Compute a Ranking
Input Analysis Output
Feed Collection Correlation FeedRank A A B B A C D C t0 C t1 t0 t1 Contribution B D D t0 A C t1 B
t0 t1 D
21 Properties of High Quality Feeds
§ Completeness Contain all malicious endpoints ▸ Contribution analysis
§ Accuracy Do not list benign endpoints ▸Correlation analysis (edge weights)
§ Speed Be complete and accurate upon changes ▸Correlation analysis (edge directions)
22 FeedRank is robust against dishonest feed providers
Tampering strategies:
§ Add entries
§ Remove entries
§ Replace entries
23 FeedRank is robust against dishonest feed providers
Tampering strategies:
Random entries are not confirmed by other feeds; § Add entries Copied entries appear on other feeds first
§ Remove entries
§ Replace entries
24 FeedRank is robust against dishonest feed providers
Tampering strategies:
Random entries are not confirmed by other feeds; § Add entries Copied entries appear on other feeds first
Removing confirmed entries results in a lower score; § Remove entries Removing unconfirmed entries truly improves the feed’s quality
§ Replace entries
25 FeedRank is robust against dishonest feed providers
Tampering strategies:
Random entries are not confirmed by other feeds; § Add entries Copied entries appear on other feeds first
Removing confirmed entries results in a lower score; § Remove entries Removing unconfirmed entries truly improves the feed’s quality
§ Replace entries Combination of the above
26 Evaluation
§ Contribution analysis
§ FeedRank vs. individual metrics
§ Tamper-resistance
§ Case-study: Find the the best feeds w.r.t. completeness
§ Case-study: Find the fastest feeds
27 Dataset
Real feeds Dishonest feeds § 27 freely available IP feeds § RandomFeed contains 50k random IPs § Between 20 and ~50k entries § CopyFeed copies entries from the two best real feeds
28 5 Feeds Contribute 80% of the IPs
But a feed’s size is not a good indicator of contribution.
60k Size
Contribution
# Entries
0 F1 F8 F2 F3 F6 F4 F7 F17 F17 F9 F16 F11 F10 F5 F14 F12 F19 F21 F20 F23 F25 F24 F26 F27 F22 F15 F18 Feeds Tamper-resistance
Dishonest feeds can cheat with individual metrics, but not with FeedRank.
§ Size
§ Update rate
30 Tamper-resistance
Dishonest feeds can cheat with individual metrics, but not with FeedRank.
Individual metrics
RandomFeed CopyFeed
31 Tamper-resistance
Dishonest feeds can cheat with individual metrics, but not with FeedRank.
Individual metrics FeedRank
16. RandomFeed CopyFeed 20.
32 FeedRank
Evaluate cyber threat intelligence feeds in a way that
Correlation & Contribution analysis § Allows us to identify high quality feeds
Weights for correlation and contribution § Is customizable for different preferences of network defenders
Based solely on the contents of feeds § Does not require a ground truth
Well-known graph algorithms § Scales to the large ecosystem of feeds
Reputation-based PageRank algorithm § Is robust against dishonest feed providers FeedRank
Evaluate cyber threat intelligence feeds in a way that
Correlation & Contribution analysis § Allows us to identify high quality feeds
Weights for correlation and contribution § Is customizable for different preferences of network defenders
Based solely on the contents of feeds § Does not require a ground truth
Well-known graph algorithms § Scales to the large ecosystem of feeds
Reputation-based PageRank algorithm § Is robust against dishonest feed providers
Questions?