Python Web-scraping program to get links from the google search result


A web scraping program which opens a google page, search "Python" and gets the links of all google's search result first of 4 pages and append that link into a new list.

Below are requirement to run the below program:

  • Python3
  • Python Selenium package
  • Webdrivers (We have used Chrome driver in this program)

Steps for webscraping:

  1. Import webdriver from selenium
  2. Create a driver object by giving the path of your webdriver
  3. Using driver open the webpage using get method
  4. Find the google search box using find_element method and enter the Keyword you want to search
  5. Lastly find all the links in the google search result.

Python Program

##Import Statement from selenium import webdriver from selenium.webdriver.common.keys import Keys driver = webdriver.Chrome("./chromedriver") #path for Chromedriver google_page = driver.get('') #Opens a google page in chrome browser search_box = driver.find_element_by_xpath('//input[@name="q"]') # Find the google search box search_box.send_keys('Python') #Enter the keyword "Python" search_box.send_keys(Keys.ENTER) #Clicks the search button pages_links =[] ## Getting links from first 4 pages in google search result counter = 0 for i in range(counter,4): page_no = driver.find_elements_by_xpath("//table[@class='AaVjTc']/tbody/tr/td/a") # Get the pages links which is at the bottom page_no[i].click() #Click that page to go that page value = driver.find_elements_by_xpath('//div[@class="r"]/a') #Gets Links of that page ## Loop for getting the href value of all the links in search result for each_val in value: link = each_val.get_attribute('href') pages_links.append(link) print("Complete links of four pages","\n".join(pages_links)) driver.quit() #Close the browser window.


DevTools listening on ws:// Complete links of four pages
Python Web-scraping program to get links from the google search result
                  Liked Article? Please Buy me a Tea 😭

Learn Python in a Week and Master It


Need Programming Help??

Below Offers ends in

VPN Pro Secure and Fast v3.0.8 Premium Mod Apk Download

Quiz For You:

Search Tags

    Python Webscraping tutorial

    Webscraping using Python Selenium simple Example