Let us execute some code as we already have Python & an IDE/Selenium installed. Now, setup a web crawler and move ahead.
#IMPORT THESE PACKAGES import selenium from selenium import webdriver #OPTIONAL PACKAGE, BUY MAYBE NEEDED from webdriver_manager.chrome import ChromeDriverManager
Let’s use the following line of code to define our web browser in Selenium:
#THIS INITIALIZES THE DRIVER (AKA THE WEB BROWSER) driver = webdriver.Chrome(ChromeDriverManager().install())
We’ll use the search phrase “cars” to acquire the URL of a specific Google Trends search page. Let’s head over to scraping of Google Trends and look for automobiles:
As you’ll see, there is a lot of information on this page. Let’s imagine we wanted to use scraped Google Trends to get the most common relevant search term:
Then, in your Python script, copy and paste the following code:
POSTS = driver.find_element_by_xpath(‘DELETE THESE WORDS AND REPLACE WITH XPATH’).text
Return to the Google Chrome browser now and save the XPath. To do so, right-click on the line in which the XPath appeared in the inspector tab, then right-click > copy > copy complete XPath.
Return to your Python script and insert the following between the two quotes:
POSTS = driver.find_element_by_xpath(‘/html/body/div[2]/div[2]/div/md-content/div/div/div[4]/trends-widget/ng-include/widget/div/div/ng-include/div/div[1]/div/ng-include/a/div/div[2]/span’).text print(POSTS)
That concludes our discussion. In our console, we’ll now be able to see the top related query for that search term:
To accomplish this, return to your Python script and cut/paste the following code:
#THIS PRETTY MUCH TELLS THE WEB BROWSER WHICH WEBSITE TO GO TO
driver.get('COPY AND PASTE YOUR URL HERE')
This code should be replaced with the following URL from our Google Trends search page:
#THIS PRETTY MUCH TELLS THE WEB BROWSER WHICH WEBSITE TO GO TO
driver.get('https://trends.google.com/trends/explore?q=cars&geo=US')
Next, we’ll use the web browser to get the HTML property of the topmost related query. To begin, learn how and when to enable your development settings in your internet browser and turn them on. Next, right-click on the real top search engine (as shown below) and select inspect; you should see something similar to this:
Here is the final execution of the project
#IMPORT THESE PACKAGES
import selenium
from selenium import webdriver
#OPTIONAL PACKAGE, BUY MAYBE NEEDED
from webdriver_manager.chrome import ChromeDriverManager
#THIS INITIALIZES THE DRIVER (AKA THE WEB BROWSER)
driver = webdriver.Chrome(ChromeDriverManager().install())
#THIS PRETTY MUCH TELLS THE WEB BROWSER WHICH WEBSITE TO GO TO
driver.get('https://trends.google.com/trends/explore?q=cars&geo=US')
POSTS = driver.find_element_by_xpath('/html/body/div[2]/div[2]/div/md-content/div/div/div[4]/trends-widget/ng-include/widget/div/div/ng-include/div/div[1]/div/ng-include/a/div/div[2]/span').text
print(POSTS)
Once the process gets completed, you have finally scraped some information from Google Trends without using an API. The above mentioned is a script used for scraping Google trends.
