DOCSLIB.ORG

[320] Web 3: Selenium

[320] Web 3: Selenium

[320] Web 3: Selenium Tyler Caraza-Harter Review Decorators how many functions get defined total? def cache(fn): how many results dicts will there be? results = {} def wrapper(*args): if not args in results: rv = fn(*args) results[args] = rv what is printed? return results[args] return wrapper print(add(1,2)) print(add(3,4)) @cache print(add(1,2)) def add(x, y): print("ADD") print(range_sum(50000000)) #1 return x+y print(range_sum(50000000)) #2 @cache def range_sum(limit): which call is faster? total = 0 for i in range(limit): total += i return total Review Document Object Model What does a web browser do when it gets some HTML in an HTTP response? url: http://domain/rsrc.html HTTP Response Browser HTTP/1.0 200 OK Content-Type: text/html; charset=utf-8 Content-Length: 74 <html> <head><script>...</script></head> <body> <h1>Welcome</h1> <a href="about.html">About</a> <a href="contact.html">Contact</a> </body> </html> url: http://domain/rsrc.html HTTP Response <html> <body> <h1>Welcome</h1> <a href="about.html">About</a> <a href="contact.html">Contact</a> </body> </html> HTTP/1.0 200 OK Content-Type: text/html; charset=utf-8 Content-Length: 74 <html> <head><script>...</script></head> <body> <h1>Welcome</h1> <a href="about.html">About</a> <a href="contact.html">Contact</a> </body> </html> url: http://domain/rsrc.html HTTP Response HTTP/1.0 200 OK Content-Type: text/html; charset=utf-8 Content-Length: 74 <html> before displaying a page, the browser <head><script>...</script></head> <body> uses HTML to generate a Document <h1>Welcome</h1> Object Model <a href="about.html">About</a> (DOM Tree) <a href="contact.html">Contact</a> </body> </html> url: http://domain/rsrc.html HTTP Response HTTP/1.0 200 OK Content-Type: text/html; charset=utf-8 Content-Length: 74 html <html> <head><script>...</script></head> body <body> <h1>Welcome</h1> <a href="about.html">About</a> h1 a a <a href="contact.html">Contact</a> </body> </html> vocab: elements Elements may contain • attributes url: http://domain/rsrc.html HTTP Response HTTP/1.0 200 OK Content-Type: text/html; charset=utf-8 Content-Length: 74 html <html> <head><script>...</script></head> body <body> <h1>Welcome</h1> <a href="about.html">About</a> h1 a a <a href="contact.html">Contact</a> </body> attr: href attr: href </html> Elements may contain • attributes • text url: http://domain/rsrc.html HTTP Response HTTP/1.0 200 OK Content-Type: text/html; charset=utf-8 Content-Length: 74 html <html> <head><script>...</script></head> body <body> <h1>Welcome</h1> <a href="about.html">About</a> h1 a a <a href="contact.html">Contact</a> </body> attr: href attr: href </html> Welcome About Contact Elements may contain • attributes • text • other elements url: http://domain/rsrc.html HTTP Response HTTP/1.0 200 OK Content-Type: text/html; charset=utf-8 Content-Length: 74 html <html> <head><script>...</script></head> body parent <body> <h1>Welcome</h1> <a href="about.html">About</a> h1 a a child <a href="contact.html">Contact</a> </body> attr: href attr: href </html> Welcome About Contact JavaScript (if there's an engine to execute it) may directly edit the DOM! url: http://domain/rsrc.html HTTP Response HTTP/1.0 200 OK Content-Type: text/html; charset=utf-8 Content-Length: 74 html <html> <head><script>...</script></head> body parent <body> <h1>Welcome</h1> <a href="about.html">About</a> h1 a a child <a href="contact.html">Contact</a> </body> attr: href attr: href </html> Welcome About Contact table original .html file doesn't change, but the result is equivalent url: http://domain/rsrc.html HTTP Response Welcome About Contact HTTP/1.0 200 OK Content-Type: text/html; charset=utf-8 Content-Length: 74 browser renders (displays) the <html> DOM tree, based on original file <head><script>...</script></head> and any JavaScript changes <body> <h1>Welcome</h1> <a href="about.html">About</a> <a href="contact.html">Contact</a> </body> </html> Web Scraping: Simple and Complicated requests vs. Selenium requests module - can fetch .html, .js, .etc file Selenium - can fetch .html, .js, .etc file - can run a .js file in browser - can grab HTML version of DOM after JavaScript has modified it IP address: 18.216.110.65 index.html, please [GET] import requests Flask Jupyter: <html> Application r=requests.get(...) <body> <img src="A.png"> <b>Hello</b> <script src="B.js"> </script> </body> </html> computer 1 computer 2 (laptop) (Virtual Machine) requests vs. Selenium requests module note: Selenium is most commonly - can fetch .html, .js, .etc file used for testing websites, but it works great for tricky scraping too Selenium - can fetch .html, .js, .etc file - can run a .js file in browser - can grab HTML version of DOM after JavaScript has modified it IP address: 18.216.110.65 <html> <body> <img src="A.png"> from selenium <b>Hello</b> Flask import webdriver <script src="B.js"> Application driver=webdriver.Chrome() </script> </body> </html> chromedriver index.html, please [GET] computer 1 computer 2 (laptop) (Virtual Machine) Tricky Pages https://tyler.caraza-harter.com/cs320/s20/materials/lec-19/scrape.html Install Selenium Install https://chromedriver.chromium.org/downloads pip install selenium wget https://chromedriver.storage.googleapis.com/80.0.3987.106/chromedriver_linux64.zip unzip chromedriver_linux64.zip echo $PATH mv chromedriver ~/.local/bin/ from selenium import webdriver driver=webdriver.Chrome() sudo apt install chromium-browser chromedriver computer 1 (laptop) Selenium Install https://chromedriver.chromium.org/downloads pip install selenium wget https://chromedriver.storage.googleapis.com/80.0.3987.106/chromedriver_linux64.zip unzip chromedriver_linux64.zip echo $PATH mv chromedriver ~/.local/bin/ from selenium import webdriver driver=webdriver.Chrome() sudo apt install chromium-browser chromedriver Check... trh@instance-1:/tmp$ chromium-browser --version computer 1 Chromium 80.0.3987.87 Built on Ubuntu , ... trh@instance-1:/tmp$ chromedriver --version (laptop) ChromeDriver 80.0.3987.106 (...) Why Drivers? Python Java Ruby JavaScript Python module Java module for Ruby module JavaScript mod for Selenium Selenium for Selenium for Selenium Chrome Driver Firefox Driver Edge Driver Examples Starter Code from selenium import webdriver from selenium.webdriver.chrome.options import Options from selenium.common.exceptions import NoSuchElementException options = Options() #options.headless = True b = webdriver.Chrome(options=options) open browser window b.get(????) go to a URL print(b.page_source) get HTML for current page (including JavaScript changes) try: search for id=???? attributes elem = browser.find_element_by_id(element_id) print("found it") except NoSuchElementException: no such element print("couldn't find it") b.close() Example 1a: Late Loading Table (page1.html) added after 1 second https://tyler.caraza-harter.com/cs320/s20/materials/lec-19/page1.html Example 1b: Headless Mode and Screenshots from selenium import webdriver from selenium.webdriver.chrome.options import Options from selenium.common.exceptions import NoSuchElementException options = Options() options.headless = True b = webdriver.Chrome(options=options) b.get(????) from IPython.core.display import Image b.save_screenshot("out.png") Image("out.png") b.close() Example 2: Auto-Clicking Buttons from selenium import webdriver from selenium.webdriver.chrome.options import Options from selenium.common.exceptions import NoSuchElementException options = Options() options.headless = True b = webdriver.Chrome(options=options) b.get(????) btn = b.find_element_by_id("BTN_ID") btn.click() b.close() auto click https://tyler.caraza-harter.com/cs320/s20/materials/lec-19/page2.html Example 3: Entering Passwords from selenium import webdriver from selenium.webdriver.chrome.options import Options from selenium.common.exceptions import NoSuchElementException options = Options() options.headless = True b = webdriver.Chrome(options=options) b.get(????) pw = b.find_element_by_id("pw") pw.send_keys("fido") b.close() https://tyler.caraza-harter.com/cs320/s20/materials/lec-19/page3.html Example 4: Many Queries https://tyler.caraza-harter.com/cs320/s20/materials/lec-19/page4.html.

View Full Text

Details

  • File Type
    pdf
  • Upload Time
    -
  • Content Languages
    English
  • Upload User
    Anonymous/Not logged-in
  • File Pages
    27 Page
  • File Size
    -

Download

Channel Download Status
Express Download Enable

Copyright

We respect the copyrights and intellectual property rights of all users. All uploaded documents are either original works of the uploader or authorized works of the rightful owners.

  • Not to be reproduced or distributed without explicit permission.
  • Not used for commercial purposes outside of approved use cases.
  • Not used to infringe on the rights of the original creators.
  • If you believe any content infringes your copyright, please contact us immediately.

Support

For help with questions, suggestions, or problems, please contact us