class 3 Data Models prof. Stratos Idreos HTTP://DASLAB.SEAS.HARVARD.EDU/CLASSES/CS165/ NO LAPTOP/PHONE POLICY class is based on participation! + there is enough evidence that laptops and phones slow you down + We will bring printed slides CS165, Fall 2019 Stratos Idreos NO LAPTOP/PHONE POLICY class is based on participation! + there is enough evidence that laptops and phones slow you down + We will bring printed slides slides are not notes! slides are mainly there to trigger discussion note keeping is your task: starting class 3 we will do collaborative note taking CS165, Fall 2019 Stratos Idreos for the semester project: install MonetDB & PostgreSQL/MySQL and play with SQL explain + SQL query to see query plans in MonetDB (use read-only mode) repeat throughout the semester compare with your system in terms of performance this is part of final deliverable CS165, Fall 2019 Stratos Idreos for the semester project: install MonetDB & PostgreSQL/MySQL and play with SQL explain + SQL query to see query plans in MonetDB (use read-only mode) repeat throughout the semester compare with your system in terms of performance this is part of final deliverable & several tools (code.harvard, gdb, perf, valgrind, testing infrastructure) do not underestimate this… CS165, Fall 2019 Stratos Idreos how to read research papers 1) abstract-intro-related work-conclusions what is the problem why is it important 2) core part-analysis why past solutions do not work basic idea what is the core idea what matters what is success any gaps? 3) follow a few citations and repeat goal: by the end of the semester understand these papers fully CS165, Fall 2019 Stratos Idreos declarative/ physical/logical independence recovery/ ~2019 ~1960s consistency philosophy/ sciences papyrus/ dbs dbs paper vs vs dbs dbs xml/p2p dbs … nosql dbs vs hand code history/timeline CS165, Fall 2019 Stratos Idreos db noSQL simple complex clean legacy just enough tuning … abstractions expensive declarative processing … it is all the same… memory hierarchy I/O vs CPU modeling as apps become optimization more complex as apps need to be data layout concepts more scalable tradeoffs … newSQL->db CS165, Fall 2019 Stratos Idreos the evolution of photo of Anand here data-driven applications CS165, Fall 2019 Stratos Idreos the evolution of photo of Anand here data-driven applications (1) use own data (recommendations) CS165, Fall 2019 Stratos Idreos the evolution of photo of Anand here data-driven applications use public data use own data (2) (1) (search) (recommendations) CS165, Fall 2019 Stratos Idreos the evolution of photo of Anand here data-driven applications (3) use social data (social networks) use public data use own data (2) (1) (search) (recommendations) CS165, Fall 2019 Stratos Idreos the evolution of photo of Anand here data-driven applications (3) use social data (social networks) use public data use own data (2) (1) (search) (recommendations) all of the above (4) (all of the above) CS165, Fall 2019 Stratos Idreos the evolution of photo of Anand here data-driven applications (3) use social data (social networks) use public data use own data (2) (1) (search) (recommendations) all of the above (4) (all of the above) (5) training data (machine learning) CS165, Fall 2019 Stratos Idreos the evolution of photo of Anand here data-driven applications (3) use social data (social networks) use public data use own data (2) (1) (search) Give(recommendations) me item X given a constraint C all of the above (4) (all of the above) (5) training data (machine learning) CS165, Fall 2019 Stratos Idreos food for (big data-driven) thought amazon google the cyborg knows & manages Cyborg all your models linkedin bank X CS165, Fall 2019 Stratos Idreos next few weeks: (1) data models and languages (today) (2) data system architecture (next few classes) (3) column-stores & hardware-conscious designs CS165, Fall 2019 Stratos Idreos logical design physical design system design CS165, Fall 2019 Stratos Idreos essential steps in using a database system experts/system admins clean schema load tune user/apps query CS165, Fall 2019 Stratos Idreos relational model+SQL database professors courses students (id,name,…) (id,name, (id,name,…) profId,…) key column/attribute table/relation CS165, Fall 2019 Stratos Idreos relational model+SQL database professors courses students (id,name,…) (id,name, (id,name,…) profId,…) key column/attribute table/relation create table for professors: create table professors (id:integer, name: char(40), telephone: char(10), …) CS165, Fall 2019 Stratos Idreos relational model+SQL database professors courses students (id,name,…) (id,name, (id,name,…) profId,…) key column/attribute table/relation create table for professors: create table professors (id:integer, name: char(40), telephone: char(10), …) insert into professors (76897689, “john smith”, …) CS165, Fall 2019 Stratos Idreos relational model+SQL database professors courses students (id,name,…) (id,name, (id,name,…) profId,…) key column/attribute table/relation create table for professors: create table professors (id:integer, name: char(40), telephone: char(10), …) insert into professors (76897689, “john smith”, …) give me the names of all students: select name from students CS165, Fall 2019 Stratos Idreos relational model+SQL database professors courses students (id,name,…) (id,name, (id,name,…) profId,…) key column/attribute table/relation create table for professors: create table professors (id:integer, name: char(40), telephone: char(10), …) insert into professors (76897689, “john smith”, …) give me the names of all students: select name from students where GPA>3.0 CS165, Fall 2019 Stratos Idreos schema employee (id:int, name:varchar(50), office:char(5), telephone:char(10), city:varchar(30), salary:int) data (1, name1, office1, tel1, city1, salary1) SQL:insert into employee (2, name2, office2, tel2, city2, salary2) (1, name1, office1, tel1, city1, salary1) (3, name3, office3, tel3, city3, salary3) (4, name4, office4, tel4, city4, salary4) (5, name5, office5, tel5, city5, salary5) cardinality=9 (6, name6, office6, tel6, city6, salary6) (7, name7, office7, tel7, city7, salary7) (8, name8, office8, tel8, city8, salary8) (9, name9, office9, NULL, city9, salary9) value does not exist CS165, Fall 2019 Stratos Idreos relational model+SQL database professor course student (id,name,…) (id,name, (id,name,…) profId,…) give me all students enrolled in cs165 CS165, Fall 2019 Stratos Idreos relational model+SQL database professor course student (id,name,…) (id,name, (id,name,…) profId,…) enrolled (studentId, courseId,…) give me all students enrolled in cs165 CS165, Fall 2019 Stratos Idreos relational model+SQL database professor course student (id,name,…) (id,name, (id,name,…) profId,…) enrolled (studentId, courseId,…) foreign key give me all students enrolled in cs165 CS165, Fall 2019 Stratos Idreos relational model+SQL database professor course student (id,name,…) (id,name, (id,name,…) profId,…) enrolled (studentId, courseId,…) foreign key give me all students enrolled in cs165 select student.name from student, enrolled, course where course.name=“cs165” and enrolled.courseId=course.id and student.id=enrolled.studentId CS165, Fall 2019 Stratos Idreos relational model+SQL database professor course student (id,name,…) (id,name, (id,name,…) profId,…) enrolled (studentId, courseId,…) foreign key give me all students enrolled in cs165 select student.name from student, enrolled, course where course.name=“cs165” and enrolled.courseId=course.id and student.id=enrolled.studentId join CS165, Fall 2019 Stratos Idreos enrolled student (studentId,courseId,…) (id,name,…) how do we join CS165, Fall 2019 Stratos Idreos normalization say schema about university db contains one table… AllData(student ID,student name,student address, course name, grade, professor name, professor ID, professor telephone,…) good CS165, Fall 2019 Stratos Idreos star schema … dimension table 1 (id1,…) fact table dimension table 2 (id1,id2,…) (id2,…) … … CS165, Fall 2019 Stratos Idreos snowflake schema … CS165, Fall 2019 Stratos Idreos Alex Liu, class 2016 165/265 project adaptive denormalization 1st prize in ACM SIGMOD undergrad research competition Special Interest Group on Management of Data CS165, Fall 2019 Stratos Idreos NORMALIZED DATA DENORMALIZED DATA only fast scans but expensive to create, storage & updates good for updates, storage but we need joins CS165, Fall 2019 Stratos Idreos adaptive denormalization normalized data CS165, Fall 2019 Stratos Idreos adaptive denormalization normalized data possible denormalized space CS165, Fall 2019 Stratos Idreos adaptive denormalization continuously physically reorganize data based on incoming query patterns (joins) normalized data possible denormalized space CS165, Fall 2019 Stratos Idreos adaptive denormalization continuously physically reorganize data based on incoming query patterns (joins) denormalized fragments queries only need to fast scan normalized data possible denormalized space CS165, Fall 2019 Stratos Idreos adaptive denormalization continuously physically reorganize data based on incoming query patterns (joins) denormalized fragments queries only need to fast scan normalized data possible denormalized space CS165, Fall 2019 Stratos Idreos adaptive denormalization continuously physically reorganize data based on incoming query patterns (joins) denormalized fragments queries only need to fast scan normalized data possible denormalized space CS165, Fall 2019 Stratos Idreos adaptive denormalization continuously physically reorganize