You will continue doing so until the new height is the same as the old height. The basic idea is that you will get the current height of the document (page), tell Selenium to scroll to the bottom of the page, wait for a few seconds, and then calculate the height of the document yet again. Luckily, there is a way for you to automate this using Selenium. There might be 1000 videos so you will have to scroll quite a few times to load them all. However, if the channel you are scraping has a ton of videos, you will have to scroll to the bottom of the page to load older videos. My channel only has 3 videos so far so it hasn't been an issue. Loading all videos on the channel pageīy default, YouTube only shows a few videos on the channel page. Perfect! You have extracted all the data you wanted! But wait! There is one slight issue. "title": "Web Scraping using lxml and Python 2018: Extracting data from " "Steam", "title": "Top 14 MOST famous Python libraries & frameworks", "title": "How to Listen to Trunked Police Radio | Complete Instructions 2020", This is what the chrome window might look like on a successful run: This way you don't have to worry about manually downloading the binary with every Chrome update. webdriver_manager makes it super simple by automatically downloading the required chromedriver binary and returning the path to that.
Generally, you pass in the path to a webdriver executable to webdriver.Chrome but it becomes tedious to keep it updated with the latest Chrome version installed on your system. It imports webdriver, Service, and ChromeDriverManager and then creates a Chrome driver instance. If this is the first time you are using webdriver_manager then it might take a few seconds for the Chrome window to open as webdriver_manager has to download the latest chromedriver.
get( this code will open up a Chrome window and navigate it to my YouTube channel. Chrome(service =Service(ChromeDriverManager().
From import Serviceįrom webdriver_manager.chrome import ChromeDriverManagerĭriver = webdriver.
0 kommentar(er)
