CSIT Advance Python for Data Science A.Andrew Bergeran M.Sc.,B.Ed.,M.B.A [email protected] 9444473301 1 Table of Contents Index Description Page. No Python Introduction Chapter 1 Python Environment 3 Technology Specifications Chapter 2 Python Data Types 14 Chapter 3 Decision Making and Loops 18 Chapter 4 String, Tuples, Sets, Dictionaries 20 Functions Chapter 5 27 ZIP, MAP, Filter, Lambda Chapter 6 File Handling and Exception Handling 31 Chapter 7 Object Oriented Programming 33 Chapter 8 Regular Expression 34 Chapter 9 CGI – Common Gateway Interface 38 Chapter 10 SQLite DB , JSON 40 Data Science Chapter 11 Pandas - Data structures and analysis 43 Chapter 12 NumPy - Numerical Computing 45 Matplotlib - 2D/3D plotting Chapter 13 46 Excel - Data Visualization Chapter 14 SciPy - Scientific Computing 51 Chapter 15 SymPy - Symbolic mathematics 52 Chapter 16 SciKit - Machine Learning 53 Beautiful Soup - HTML/XML Parser Chapter 17 56 Web Scraping 2 Introduction Chapter 1 Python is an easy to learn, powerful programming language. It has efficient high-level data structures and a simple but effective approach to object-oriented programming. Python‟s elegant syntax and dynamic typing, together with its interpreted nature, make it an ideal language for scripting and rapid application development in many areas on most platforms. Developed by Guido van Rossum. The Python interpreter and the extensive standard library are freely available in source or binary form for all major platforms from the Python Web site, https://www.python.org and may be freely distributed. The Python interpreter is easily extended with new functions and data types implemented in C or C++ (or other languages callable from C). Python is also suitable as an extension language for customizable applications. Python is a general purpose Interpreted programming language that is often applied in scripting roles. So Python is language as well as scripting. Python is a readable , dynamic, pleasant, flexible, fast, and powerful language. Multi-Purpose ( Web, GUI, Scripting, etc. ). Python is a Platform Independent Language. Open Source. OOP Python is widely used in Artificial Intelligence, Machine Learning; Data Analytics Python has very powerful statistical and visualization libraries. Python has efficient for high-level Data Structures and Data Science. 3 History Chapter 1 Invented in the Netherlands, early 90s by Guido van Rossum. Python was conceived in the late 1980s and its implementation was started in December 1989 . Rossum is fan of „Monty Python‟s Flying Circus‟, this is a famous TV show in Netherlands, Named after Monty Python. Python 1.0 releases in 1994 1.x Python 2.0 releases in 2000 2.x Python 3.0 releases in 2008 3.x Python 3.5 releases in 2015 Python Features Interpreted Language - Interpreter reads the source code of the program, line by line, parses the source code, and interprets the instructions. Python Interpreters are available for many operating systems. Object-oriented programming language. Python is an open source scripting language. Large standard libraries to solve common tasks. Cross Platform Language. Python provides no braces to indicate blocks of code for class and function definitions or flow control. Blocks of code are denoted by line indentation. Python has a very simple and elegant syntax. It's much easier to read and write Python programs compared to other languages like: C++, Java, C#. Interpreted, Object Oriented, Open Source, Cross Platform, Line Indentation Large Standard Libraries, Very simple and Elegant Syntax, Automatic memory management, Support third party utilities, Mixable 4 Scope Chapter 1 • Scientific and Numeric • System Programming • Web Applications • Testing Scripts • Graphical User Interface Programming • Component Integration • Database Application • Network Programming • Game Development Users of Python Google –Web Spider and Search Engine NASA – Scientific Calculations Intel, Cisco, Hewlett-Packard, Seagate, IBM use python for hardware testing ESRI – Environmental Systems Research Institute GIS – Geographic information systems mapping products You Tube – Video Sharing Research Scalars Statistics Projects Data Scientists 5 Compiling and Interpreting Chapter 1 Many languages require compiling your program into a form that the machine understands. Compiled Program is executed. Python is instead directly interpreted into machine instructions. Interpreter executes the program directly without Compile Python Interpreter The Python code we write must always be run by the Interpreter. (Source code can execute instructions directly) Source code is translated to byte code, which is then run by the Python Virtual Machine. Your Code is Automatically Compiled Compilation is a translation step, and the byte code is a low-level platform- independent. Note that the Python byte code is not binary machine code. 6 Installation & Configuration Chapter 1 Software Requirements: Key Points • OS : Windows 7 Service Pack1 / Windows 8/ 10 • Python 3.6.2 (Optional IDE) • JDK 1.7 • NetBeans IDE 8.0.2 • Python Plugins for NetBeans 8.0 • Install IIS and configure CGI • DB Browser for SQLite Run the downloaded file Python 3.6.2. This brings up the Python install wizard, just accept the default settings, and wait until the install is finished. IDLE - Integrated Development and Learning Environment Python 3.6.2 (v3.6.2:5fd33b5, Jul 8 2017, 04:14:34) Go to https://www.python.org/downloads/ 7 Installation & Configuration Chapter 1 IDLE - Integrated Development and Learning Environment 8 Chapter 1 Python Environment Mode Python Interpreter can be used in Interactive Mode and Scripting mode. Interactive Mode: Quickly interact with python on a Command Line Interface. Scripting mode: Write multiple line coding to be save & execute as a program. Python script source code in a file with the .py extension. 9 Integrated Development Environments for Python Chapter 1 • Atom, an open source cross-platform IDE with autocomplete. • Eclipse with the Pydev plug-in. • Eric an IDE for Python and Ruby. • Koding a free online development environment. • IDE an IDE for Python, Perl, PHP and Ruby. Komodo • NetBeans is written in Java and runs everywhere where a JVM is installed. • PIDA open source IDE. • PyCharm, a proprietary and Open Source IDE. • PyScripter Free and open-source software Python IDE. • Python Tools for Visual Studio Free and open-source plug-in for Visual Studio. Python Standard Library Numeric and Mathematical Modules numbers,math,cmath,decimal, fractions,random,statistics File and Directory Access pathlib, os.path, fileinput, filecmp Data Persistence sqlite3, pickle Data Compression and Archiving zlib, gzip, bz2, zipfile, tarfile File Formats csv Cryptographic Services hashlib, hmac, secrets 10 Operating System Services Chapter 1 os,io,time Concurrent Execution threading Networking socket,ssl Internet Data Handling email,json,base64 Structured Markup Processing Tools html, xml.dom Internet Protocols cgi,urllib,http https://docs.python.org/3/library/ Python Libraries for Data Science Numerical libraries - NumPy, SciPy, SymPy NumPy = Numerical Python (Array Package) advance math functionalities SciPy = Scientific Python. It is a library of algorithms and mathematical tool SymPy = Symbolic Mathematics. (algebraic evaluation, complex numbers ) 11 Mathematical libraries - Matplotlib, NumPy, SymPy Chapter 1 Matplotlib = powerful visualizations (2d Plotting). Data Structure and Analysis - Pandas Pandas = data manipulation, aggregation, and visualization. Scientific Computing - SciPy, Scikit SciPy, Scikit-learn (Machine Learning) Web Scraping BeautifulSoup - xml and html parsing library Scrapy - Scrapy is a library for making crawling programs NLTK - Natural Language Toolkit, (Linguistics, Cognitive Science, Artificial Intelligence) Graphics Frameworks Panda3d, PyGame UI Frameworks PyGTK, PyQt,Tkinter pip Package Manager pip, a package manager used to install and manage Python software packages pip install <Package Name> pip list # displays the list of currently installed modules pip uninstall <Package Name> umpy (1.14.0) bs4 (0.0.1) pip (9.0.1) pyexcel (0.5.7) Goto Command Prompt pyexcel-io (0.5.6) pyexcel-xls (0.5.5) C:\Users\Admin>cd\ requests (2.18.4) setuptools (28.8.0) C:\>cd Python simplejson (3.13.2) urllib3 (1.22) C:\Python>cd Scripts xlrd (1.1.0) XlsxWriter (1.0.2) C:\Python\Scripts> pip list xlwt (1.3.0) beautifulsoup4 (4.6.0) 12 Chapter 1 Python Keywords Keywords are the reserved words in Python. We cannot use a keyword as variable name, function name or any other identifier. They are used to define the syntax and structure of the Python language. Keywords are case sensitive. Each keyword have a special meaning and a specific operation. There are 33 keywords in Python 3.6.4 >>> import keyword >>> keyword.kwlist [import, as , from, and, or, not, True, False, None, if else, elif, for, in, while, with, break, continue, class def, pass, return ,del, global, try, except, finally, raise lambda, nonlocal, is, assert, yield] #this is a comment //Single Line Comment. #is treated as a comment. """This is // Multi line Comment Multi-line comments""" Python Simple Programs print ("Hello World") a=5 b=5 c=a+b print ("Ans: ",c) Runtime Input from console a=input("Enter A :") print (a) 13 Python Data Types: Chapter 2 Numbers, Strings, Tuple, Set, List, Dictionaries Python data types are different in some aspects from other programming languages. Python interpreter can determine which type of data are storing, so no need to define the data type of memory location. There are many native (built-in) data types available in Python. Numbers: int, float, Long, Complex Literals: Any number or a string value. Literals can be defined as a data that is given in a variable or constant. Variables Variable is a name which is used to refer memory location. A user defined container that can hold a literal value. The equal sign (=) is used to assign values to variables. a = 10 //Int b = 15.20 //float c = "Python" //String print (a) 1. Assigning single value to multiple variables x=y=z=50 2. Assigning multiple values to multiple variables: a, b, c=5,10,15 Identifiers Identifier is the name given to entities like class, object, functions, and variables.