Python Web-scraping program to get links from the google search result


A web scraping program which opens a google page, search "Python" and gets the links of all google's search result first of 4 pages and append that link into a new list.

Below are requirement to run the below program:

  • Python3
  • Python Selenium package
  • Webdrivers (We have used Chrome driver in this program)

Steps for webscraping:

  1. Import webdriver from selenium
  2. Create a driver object by giving the path of your webdriver
  3. Using driver open the webpage using get method
  4. Find the google search box using find_element method and enter the Keyword you want to search
  5. Lastly find all the links in the google search result.

Python Program

##Import Statement from selenium import webdriver from selenium.webdriver.common.keys import Keys driver = webdriver.Chrome("./chromedriver") #path for Chromedriver google_page = driver.get('') #Opens a google page in chrome browser search_box = driver.find_element_by_xpath('//input[@name="q"]') # Find the google search box search_box.send_keys('Python') #Enter the keyword "Python" search_box.send_keys(Keys.ENTER) #Clicks the search button pages_links =[] ## Getting links from first 4 pages in google search result counter = 0 for i in range(counter,4): page_no = driver.find_elements_by_xpath("//table[@class='AaVjTc']/tbody/tr/td/a") # Get the pages links which is at the bottom page_no[i].click() #Click that page to go that page value = driver.find_elements_by_xpath('//div[@class="r"]/a') #Gets Links of that page ## Loop for getting the href value of all the links in search result for each_val in value: link = each_val.get_attribute('href') pages_links.append(link) print("Complete links of four pages","\n".join(pages_links)) driver.quit() #Close the browser window.


DevTools listening on ws:// Complete links of four pages
Python Web-scraping program to get links from the google search result

Algorithm Logic Test

Our Quiz prepared by Experts Helps you identify your knowledge in Algorithms. Everyone should atleast attempt this Quiz Once. 

Need any help in Programming? Chat with Us


Online Compiler
Algorithm Quiz

Only 5% Users were able to score above 75% in this Quiz. Can You Crack this?

Deals Ends in

Online Games
Play 2048 Game Online and Relax.
Play 2048 Game Online

Search Tags

    Python Webscraping tutorial

    Webscraping using Python Selenium simple Example