2/20/18 Lecture 11: Last Two Lectures Finish Web Technologies & Ø Last Thursday: String manipulation & Regular Expressions Ø guest lecture from the amazing Sam Lau Begin SQL Databases Ø reviewed in section and in future labs & HWs Ø Last Tuesday: HTTP, XML, and JSON Ø Pandas web tables support Slides by: Ø Using the browser developer mode Joseph E. Gonzalez ? Ø JSON and basics of XML [email protected] Ø Started HTTP request/response protocol and GET vs POST Ø Didn’t finish REST and web-services … REST – Representational State Transfer REST APIs Ø A way of architecting widely accessible, efficient, and extensible web services (typically using HTTP) Example: GET /website/images Get all images Ø Client-Server: client and server are able to evolve independently POST /website/images Add an image GET /website/images/{id} Get a an image Ø Stateless: The server does not store any of the clients session state PUT /website/images/{id} Update an image DELETE /website/images/{id} Delete an image Ø Cacheable: system should clearly define what functionality can be cached (e.g., GET vs POST requests) Ø Uniform Interface: provide a consistent interface for getting and Client Server updating data in a system Scraping Ethics Ø Don’t violate terms of use for the service or data Ø Scraping can cause result in degraded services for others Ø Many services are optimized for human user access patterns Ø Requests can be parallelized/distributed to saturate server Ø Each query may result in many database requests Ø How to scrape ethically: Demo Ø Used documented REST APIs – read terms of service TwitterAPI_REST_Example.ipynb Ø Examine at robots.txt (e.g., https://en.wikipedia.org/robots.txt) Ø Throttle request rates (sleep) Ø Avoid getting Berkeley (or your organization) blocked from websites & services 1 2/20/18 Databases and SQL Part 1 Slides by: What is a database? Joseph E. Gonzalez & Joseph Hellerstein, ? [email protected] [email protected] Defining Databases Database Management Systems Ø A database is an organized collection of data. Ø Data storage Ø Provide reliable storage to survive system crashes and disk failures Ø Special data-structures to improve performance Ø Data management Ø Configure how data is logically organized and who has access Ø Ensure data consistency properties (e.g., positive bank account values) Ø A database management systems (DBMS) is a software Ø Facilitate access system that stores, manages, and facilitates access to Ø Enable efficient access to the data one or more databases. Ø Supports user defined computation (queries) over data Is Pandas a Database Management System? Why should I use a DBMS? Why can’t I just have my CSV files? Ø Data Storage? Ø DBMSs organize many related sources of information Ø Pandas doesn’t store data, this is managed by the filesystem Ø Data Management? Ø DBMSs enforce guarantees on the data Ø Pandas does support changing the organization of data but doesn’t Ø Can be used to prevent data anomalies manage who can access the data Ø Ensure safe concurrent operations on data Ø Facilitate Access? Ø DBMSs can be scalable Ø Pandas does support rich tools for computation over data Ø Optimized to compute on data that does not fit in memory Ø Parallel computation and optimized data structures Ø Pandas is not generally considered a database management system but it often interacts with DBMSs Ø DBMSs prevent data loss from software/hardware failures 2 2/20/18 Common DBMS Systems https://db-engines.com/en/ranking Widely Used DBMS Technologies Relational database management systems are widely used! Relational Database Management Systems Relational Data Abstraction Ø Relational databases are the traditional DBMS technology Database Management System Relations (Tables) Optimized Data Structures Name Prod Price Sue iPod $200.00 B+Trees Joey sid Bikesname $333.99rating age Ø Logically organize data in relations (tables) Alice 28 Caryuppy $999.009 35.0 31 lubber 8 55.5 Describes relationship: 44 guppy 5 35.0 Page 1 Page 2 Name Prod Price bid bname color Optimized Sales relation: 58 rusty 10 35.0 Name purchased Storage Sue iPod $200.00 101 Interlake blue Page 3 Page 4 Abstraction Page Prod at Price. 102 Interlake red Header Joey Bike $333.99 104 Marine red Page 5 Page 6 Tuple (row) Alice Car $999.00 How is data 103 Clipper green physically Attribute (column) stored? RelationalPhysical Data Data Independence: Abstraction Database management systemsDatabase hide Management how data Systemis Relationsstored from (Tables) end user applicationsOptimized Data Structures Name Prod Price Sueà SystemiPod can$200.00 optimize storage and computationB+Trees Joey sid Bikesname $333.99rating age Alicewithout28 Caryuppy changing$999.009 35.0 applications. 31 lubber 8 55.5 44 guppy 5 35.0 Page 1 Page 2 bid bname Bigcolor Idea in Data StructuresOptimized 58 rusty 10 35.0 In a time long ago … Storage 101 Interlake blue Page 3 Page 4 DataAbstraction Systems & Page 102 Interlake red Header 104 Marine red ComputerPage 5 Page 6 Science 103 Clipper green It wasn’t always like this … 3 2/20/18 Ted Codd and the Relational Model Ø [1969] Relational model: a mathematical Before 1970’s databases abstraction of a database as sets Ø Independence of data from the physical properties of were not routinely organized as tables. stage storage and representation Instead they exposed specialized Ø [1972] Relational Algebra & Calculus: a collection data structures designed for of operations and a way defining logical specific applications. outcomes for data transformations Ø Algebra: beginning of technologies like Pandas Ø Calculus: the foundation of modern SQL Edgar F. “Ted” Codd (1923 - 2003) Turing Award 1981 Relational Database Management Systems Ø Traditionally DBMS referred to relational databases What Ø Logically organize data in relations (tables) Ø Structured Query Language (SQL) to define, manipulate not and compute on data. Ø A common language spoken by many data systems How Ø Some variations and deviations from the standard … Ø Describes logical organization of data as well as computation on data. SQL SQL is a Declarative Language Review of Relational Terminology Ø Declarative: “Say what you want, not how to get it.” Ø Database: Set of Relations (i.e., one or more tables) Ø Declarative Example: I want a table with columns “x” and “y” constructed from tables “A” and ”B” where the values in “y” are greater than 100.00. Ø Relation (Table): Ø Imperative Example: For each record in table “A” find the corresponding Ø Schema: description of columns, their types, and constraints record in table “B” then drop the records where “y” is less than or equal to Ø Instance: data satisfying the schema 100 then return the ”x” and “y” values. Ø Advantages of declarative programming Ø Attribute (Column) Ø Enable the system to find the best way to achieve the result. Ø Often more compact and easier to learn for non-programmers Ø Tuple (Record, Row) Ø Challenges of declarative programming Ø Schema of database is set of schemas of its relations Ø System performance depends heavily on automatic optimization Ø Limited language (not Turing complete) 4 2/20/18 Two sublanguages of SQL Ø DDL – Data Definition Language Ø Define and modify schema Ø DML – Data Manipulation Language Creating Tables & Ø Queries can be written intuitively. SELET * FROM Populating Tables THINGS; CAPITALIZATION IS optional BUT … DATABASE PEOPLE PREFER TO YELL CREATE TABLE … CREATE TABLE Sailors ( sid INTEGER, sname CHAR(20), Columns have Common SQL Types (there are others...) sid sname rating age rating INTEGER, names and types 1 Fred 7 22 age REAL, PRIMARY KEY (sid)); 2 Jim 2 39 Specify Ø CHAR(size): Fixed number of characters 3 Nancy 8 27 Primary Key CREATE TABLE Boats ( column(s) bid INTEGER, Ø TEXT: Arbitrary number of character strings bname CHAR (20), bid bname color color CHAR(10), PRIMARY KEY (bid)); Specify Ø INTEGER & BIGINT: Integers of various sizes 101 Nina red Foreign Key 102 Pinta blue relationships Ø REAL & DOUBLE PRECISION: Floating point numbers 103 Santa Maria red CREATE TABLE Reserves ( sid INTEGER, bid INTEGER, Ø DATE & DATETIME: Date and Date+Time formats day DATE, PRIMARY KEY (sid, bid, day), sid bid day FOREIGN KEY (sid) REFERENCES Sailors, FOREIGN KEY (bid) REFERENCES Boats); 1 102 9/12 See documentation for database system (e.g., Postgres) 2 102 9/13 Semicolon at end of command More Creating Tables Inserting Records into a Table INSERT INTO students (name, gpa, age, dept, gender) ß Optional CREATE TABLE students( VALUES name TEXT PRIMARY KEY, ('Sergey Brin', 2.8, 40, 'CS', 'M'), gpa REAL CHECK (gpa >= 0.0 and gpa <= 4.0), ('Danah Boyd', 3.9, 35, 'CS', 'F'), age INTEGER, Ø ('Bill Gates', 1.0, 60, 'CS', 'M'), Fields must be entered in dept TEXT, ('Hillary Mason', 4.0, 35, 'DATASCI', 'F'), order (record) gender CHAR); Ø ('Mike Olson', 3.7, 50, 'CS', 'M'), Comma between records Ø ('Mark Zuckerberg', 4.0, 30, 'CS', 'M'), Must use the single quote Imposing Integrity (‘Sheryl Sandberg', 4.0, 47, 'BUSINESS', 'F'), (’) for strings. Constraints ('Susan Wojcicki', 4.0, 46, 'BUSINESS', 'F'), ('Marissa Meyer', 4.0, 45, 'BUSINESS', 'F'); Useful to ensure data quality… -- This is a comment. -- Does the order matter? No 5 2/20/18 Deleting and Modifying Records Deleting and Modifying Records Ø Records are deleted by specifying a condition: Ø What is wrong with the following DELETE FROM students UPDATE students WHERE LOWER(name) = 'sergey brin' SET gpa = 1.0 + gpa String Function WHERE dept = ‘CS’; Ø Modifying records UPDATE students CREATE TABLE students( name TEXT PRIMARY KEY, SET gpa = 1.0 + gpa gpa FLOAT CHECK (gpa >= 0.0 and gpa <= 4.0), WHERE dept = ‘CS’; age INTEGER, dept TEXT, Ø Notice that there is no way to modify records by location gender CHAR); Update would violate Integrity Constraints SQL DML: Basic Single-Table Queries SELECT [DISTINCT] <column expression list> FROM <single table> [WHERE <predicate>] [GROUP BY <column list> [HAVING <predicate>] ] Querying Tables [ORDER BY <column list>]; Ø Elements of the basic select statement Ø [Square brackets] are optional expressions.