Visualizing Mirrors Mirrors.Ustc.Edu.Cn 服务器日志分析
Total Page:16
File Type:pdf, Size:1020Kb
Visualizing Mirrors mirrors.ustc.edu.cn 服务器日志分析 . 李博杰 [email protected] ©USTC LUG August 14, 2012 . .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. 李博杰 [email protected] Visualizing Mirrors . Outline 1. Requests & Traffic By Time By IP By Other Measures 2. Files Files Characteristics How Files Are Requested 3. Sessions 4. Distributions Insight CentOS Fedora Ubuntu Eclipse 5. Technical Details 6. Query Optimization . .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. 李博杰 [email protected] Visualizing Mirrors . Notes The data is access log of mirrors.ustc.edu.cn in 51 days. See ‘Technical Details’ section for more info about dataset. Some graphs are in log-scale for clarity. Please note whether x axis, y axis or both are in log-scale. The graph title sometimes lies. Because there may be many points in a graph, sampling is made to reduce file size (they are vector graphics), hence there may be some ‘straight lines’. I have checked the data to make sure the graphs illustrate real trends. Title length is limited, so the title itself may not explain well, please keep an eye on the axis and keys of the graph. Graphs are shown in the hope of conveying information without words. Any questions or suggestions, please email me. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. 李博杰 [email protected] Visualizing Mirrors . Requests & Traffic in a day Requests & Traffic within a day 400000 5e+10 350000 4e+10 300000 250000 3e+10 200000 Requests Traffic (Bytes) 2e+10 150000 100000 1e+10 50000 Request count (Bezier smoothed) Traffic (Bezier smoothed) 0 0 00:00 02:00 04:00 06:00 08:00 10:00 12:00 14:00 16:00 18:00 20:00 22:00 00:00 Time of the day . .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. 李博杰 [email protected] Visualizing Mirrors . Requests & Traffic in a week Requests & Traffic in different weekdays 7e+07 8e+12 7e+12 6e+07 6e+12 5e+07 5e+12 4e+07 4e+12 Requests 3e+07 Traffic (Bytes) 3e+12 2e+07 2e+12 1e+07 1e+12 Request count Traffic 0 0 Monday Tuesday Wednesday Thursday Friday Saturday Sunday Time of the day . .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. 李博杰 [email protected] Visualizing Mirrors . Requests & Traffic across 50 days Requests & Traffic in 50 days 1.2e+07 1.2e+12 1e+07 1e+12 8e+06 8e+11 6e+06 Requests 6e+11 Traffic (Bytes) 4e+06 4e+11 2e+06 2e+11 Request count Traffic 0 0 05-20 05-27 06-03 06-10 06-17 06-24 07-01 07-08 07-15 Time of the day . .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. 李博杰 [email protected] Visualizing Mirrors . Statistics Requests Traffic Total 328976877 36892 GB Avg. per Day 6450527 723.4 GB Max. per Day 8632963 1049.5 GB Min. per Day 4868022 421.5 GB Avg. per Hour 268771 30.14 GB Max. per Hour 561925 79.75 GB Min. per Hour 99506 2.97 GB Avg. per Minute 4480 514.4 MB Max. per Minute 14714 N/A Min. per Minute 441 N/A Avg. per Second 74.66 8779 KB Max. per Second 2117 N/A Min. per Second 1 N/A Because the time recorded is only completion time of the request, and large requests can span hours, so Max./Min. per minute/second is not applicable. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. 李博杰 [email protected] Visualizing Mirrors . Cumulative Requests per Hour Sorted Requests per Hour 600000 550000 500000 450000 400000 350000 Requests 300000 250000 200000 150000 100000 Hour Percentage 50000 0 10 20 30 40 50 60 70 80 90 100 Hour Percentage (sorted by Requests count). .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. 李博杰 [email protected] Visualizing Mirrors . Cumulative Requests per Minute Sorted Requests per Minute 16000 14000 12000 10000 8000 Requests 6000 4000 2000 Minutes Percentage 0 0 10 20 30 40 50 60 70 80 90 100 Minutes Percentage (sorted by Requests count). .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. 李博杰 [email protected] Visualizing Mirrors . Cumulative Requests per Second Sorted Requests per Second 450 400 350 300 250 Requests 200 150 100 50 Seconds Percentage 0 0 10 20 30 40 50 60 70 80 90 100 Seconds Percentage (sorted by Requests count). .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. 李博杰 [email protected] Visualizing Mirrors . Cumulative Traffic over IPs: 20%-80% law Cumulative Traffic over unique IPs 4e+13 3.5e+13 3e+13 2.5e+13 2e+13 Cumulative Traffic 1.5e+13 1e+13 5e+12 0 1 10 100 1000 10000 100000 1e+06 1e+07 Percentage of unique IP (log-scale) (sorted by Traffic. DESC). .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. 李博杰 [email protected] Visualizing Mirrors . Cumulative Requests over IPs: 20%-80% law Cumulative Requests over unique IPs 3.5e+08 3e+08 2.5e+08 2e+08 1.5e+08 Cumulative Requests 1e+08 5e+07 0 1 10 100 1000 10000 100000 1e+06 1e+07 Percentage of unique IP (log-scale) (sorted by Request. Num. DESC). .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. 李博杰 [email protected] Visualizing Mirrors . IPv4 vs. IPv6 Requests Traffic IPv4 318575688 (96.84%) 34180 GB (92.65%) IPv6 10401189 (3.15%) 2712 GB (7.35%) It can be seen that IPv6 still have a long way to go… . .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. 李博杰 [email protected] Visualizing Mirrors . Requests & Traffic TOP 40: xxx.0.0.0/24 Request & Traffic among IPv4 first fields 0.08 Requests Traffic 0.07 0.06 0.05 0.04 Percentage 0.03 0.02 0.01 0 IPv6 222 202 113 218 58 114 61 183 180 121 124 116 219 210 119 122 59 221 125 123 60 115 117 118 220 211 203 112 111 14 182 110 27 1 120 175 101 159 223 . .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. 李博杰 [email protected] Visualizing Mirrors . Traffic TOP 40: IPv4 addrs Request & Traffic among popular IPv4 addrs 0.005 Requests Traffic 0.0045 0.004 0.0035 0.003 0.0025 Percentage 0.002 0.0015 0.001 0.0005 0 219.133.0.1 218.242.250.212 203.114.244.88 114.212.189.93 114.113.226.53 180.169.73.90 180.96.19.25 66.197.225.53 218.3.125.243 61.234.123.57 159.226.126.177 203.198.202.225 124.74.45.130 220.181.145.27 114.213.255.162 202.108.130.138 202.119.45.31 222.66.23.57 124.127.250.34 208.53.156.36 221.216.135.54 116.228.240.198 113.108.76.195 114.80.133.7 210.13.71.73 180.149.134.10 124.126.245.14 218.94.63.55 220.248.0.145 112.65.134.2 222.56.17.109 124.74.78.2 124.207.104.18 58.211.218.74 1.202.225.132 116.226.65.12 220.248.0.154 222.94.140.45 210.73.5.33 116.247.98.50 . .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. 李博杰 [email protected] Visualizing Mirrors . Request count TOP 40: IPv4 addrs Request & Traffic among popular IPv4 addrs 0.03 Requests Traffic 0.025 0.02 0.015 Percentage 0.01 0.005 0 63.245.214.78 209.132.181.102 113.111.38.40 129.143.116.10 211.86.56.227 223.5.20.10 49.123.105.219 159.226.20.217 60.208.111.199 119.97.142.81 202.38.95.60 203.244.218.6 221.219.75.222 202.104.151.152 121.49.96.70 59.77.33.100 182.89.199.227 183.45.54.83 218.13.224.81 59.37.44.133 59.44.42.194 222.134.53.246 210.21.243.170 61.130.247.168 116.228.202.66 123.185.172.126 219.134.89.202 114.113.29.21 183.31.242.39 58.19.126.37 27.17.19.75 203.114.244.88 220.178.52.108 124.42.77.160 113.111.40.89 218.94.63.55 180.153.97.82 222.171.60.177 210.34.196.99 222.92.29.130 . .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. 李博杰 [email protected] Visualizing Mirrors . USTC Mirrors Usage (IPv4 only) IP range Requests Traffic Note 202.38.64.0-202.38.95.255 994661 (0.30%) 149.3 GB (0.40%) CERNET 210.45.64.0-210.45.79.255 191141 (0.06%) 68.31 GB (0.19%) CERNET 210.45.112.0-210.45.127.255 243976 (0.07%) 37.59 GB (0.10%) CERNET 211.86.144.0-211.86.159.255 81035 (0.02%) 24.96 GB (0.07%) CERNET 222.195.64.0-222.195.95.255 319435 (0.10%) 86.88 GB (0.24%) CERNET 114.214.160.0-114.214.255.255 0 0 CERNET 210.72.22.0-210.72.22.255 3622 (0.00%) 11.86 MB (0.00%) TechNet (?) 218.22.21.0-218.22.21.31 1 (0.00%) 0.01 MB (0.00%) China Telecom 218.104.71.160-218.104.71.175 0 0 China Unicom 202.141.160.0-202.141.175.255 123455 (0.04%) 12.60 GB (0.03%) China Telecom 202.141.176.0-202.141.191.255 187 (0.00%) 120.4 MB (0.00%) China Mobile Total 1957513 (0.60%) 379.77 GB (1.03%) USTC IPv4 Data source of USTC IP range: http://lib.ustc.edu.cn/ustcip.html . .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. 李博杰 [email protected] Visualizing Mirrors . Requests & Traffic of distributions Request & Traffic of Distributions 0.35 0.3 0.25 0.2 Percentage 0.15 0.1 0.05 Requests Traffic 0 eclipsefedoraubuntucentosdebiantdf cygwinCTANarchlinuxmozilla-currentopensusegentookde-applicationdatakde backtrackepel gnu CRANNULLfreebsdubuntu-releasesdebian-securityscientificlinuxlinux-kerneldebian-backportskdemoddebian-cdmeegoslackwaredeepinsourceware.orgCPANlinuxmintlinux-2.6.gitpuppylinux.gitdebian-multimedia4 qomo3.7 . .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. 李博杰 [email protected] Visualizing Mirrors . Requests & Traffic of distributions Request & Traffic of Distributions 0.35 0.3 0.25 0.2 Percentage 0.15 0.1 0.05 Requests Traffic 0 centosNULLeclipsefedoraubuntuubuntu-releasesmozilla-currenttdf debianCTANbacktrackopensusegentoodeepin-cdkde-applicationdatadebian-cdcygwinUbuntulinuxmintarchlinuxkde gnu linuxmint-cdCRANdebian-multimediaqomodebian-securityepel puppyscientificlinuxdebian-backportsfreebsddeepinCPANkdemodturnkeylinuxslackwaredebian-uogentoo-portagelinux-kernel .