In the age of information, we constantly find ourselves searching the web for answers, insights, or just out of curiosity. From looking up the latest news to researching deep technical topics, web searches have become an integral part of our daily routines. But what if you could automate those searches? Imagine saving time by letting a tool do the heavy lifting for you, searching across multiple websites with just a few clicks.
Today, you’ll learn how to automate web searches using Selenium in Python. We’ll walk you through setting up a simple yet effective Tkinter GUI, configuring Selenium WebDriver, and performing automated searches on popular websites like Google, Bing, and Wikipedia. By the end of this tutorial, you’ll have a powerful tool at your fingertips, ready to simplify your web search tasks and enhance your automation skills.
Let’s get started!
Table of Contents
- Getting Started
- Imports
- Setting Up Logging
- Web Search Information Dictionary
- Performing a Search
- Initiating the Search Function
- Setting Up the Main Window
- Example
- Full Code
Getting Started
Before we dive into the code, make sure you’ve got everything set up. You’ll need to install a few libraries to make it all work smoothly. Just open your terminal or command prompt and run these commands:
$ pip install selenium
$ pip install tk
You’ll also need to install the ChromeDriver that matches your Chrome version. Just find the right URL for your operating system and Chrome version, copy and paste it to download the driver.
Once you’ve got it, go ahead and install it, then make sure to add its path to your Environment Variables. This step ensures everything runs smoothly when you start automating your browser tasks.
Imports
import logging
import tkinter as tk
from tkinter import messagebox, ttk
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException, NoSuchElementException, WebDriverException
from datetime import datetime
As always, let’s start by gathering the necessary tools:
Logging: This will help us track the flow of the program and debug any issues.tkinter: We’ll use this to create a graphical user interface (GUI), display information and errors in message boxes, and make use of themed widgets withttk.Selenium: This library is essential for opening a browser and navigating web pages.WebDriver: This is what we use to interact with the browser.Keys: This helps us handle keyboard actions.By: This is used to locate elements on a web page.WebDriverWait: This ensures that Selenium waits for certain conditions to be met before moving forward.Expected Conditions (EC): These are the predefined conditions that need to be fulfilled.TimeoutException,NoSuchElementException, andWebDriverException: These are used to handle various errors that might occur during browser automation.
Datetime: This is handy for adding timestamps to filenames, as it allows us to manipulate dates and times.
Setting Up Logging
# Set up logging
logging.basicConfig(filename='selenium_test.log', level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')
This is our behind-the-scenes hero, recording all our actions by saving log messages to the selenium_test.log file. Messages with INFO level and higher (like WARNING and ERROR) will be logged with a timestamp and the severity level.
Web Search Information Dictionary
For this step, let’s meet the drivers of our program, where the searches will happen. We’ve created a dictionary that holds the names of the websites we’ll be using as keys. Each key has a value that includes the website’s URL and the name of the search box we’ll use to perform the search.
# Dictionary to map website names to URLs and search box names
websites = {
"Python.org": {"url": "http://www.python.org", "search_box": "q"},
"Google": {"url": "http://www.google.com", "search_box": "q"},
"Wikipedia": {"url": "http://www.wikipedia.org", "search_box": "search"},
"Bing": {"url": "http://www.bing.com", "search_box": "q"}
}
Performing a Search
Now, we’ve reached the core of our script, where the real action happens—the part that performs the search on the selected website. The perform_search() function kicks things off by retrieving the selected site’s URL and search box name from the websites dictionary. Then, webdriver starts a new Chrome browser session.
After that, the function uses driver.get() to navigate to the selected URL. If everything goes smoothly and the website opens, we log a success message. But if any errors pop up during this process, an error message will be displayed.
def perform_search(search_term, selected_site):
site_info = websites[selected_site]
url = site_info["url"]
search_box_name = site_info["search_box"]
# Start the WebDriver and open the website
try:
driver = webdriver.Chrome()
logging.info("WebDriver started successfully.")
driver.get(url)
logging.info(f"Website {url} opened successfully.")
except WebDriverException as e:
logging.error(f"Error starting WebDriver or opening website: {e}")
messagebox.showerror("Error", f"Error starting WebDriver or opening website: {e}")
return
With the website now open, By.NAME helps locate the search box, while WebDriverWait and EC.presence_of_element_located ensure the script pauses until the search box is found, up to 10 seconds. If the search box isn’t found within this time, TimeoutException will step in and display an error message about the search results.
If everything goes well, the script clears any pre-existing text in the search box with elem.clear(), enters the search term using elem.send_keys(search_term), and presses the Enter key with elem.send_keys(Keys.RETURN). Finally, a log message confirms that the search has been successfully initiated.
try:
# Find the search box and perform the search
elem = WebDriverWait(driver, 10).until(
EC.presence_of_element_located((By.NAME, search_box_name))
)
elem.clear()
elem.send_keys(search_term)
elem.send_keys(Keys.RETURN)
logging.info(f"Search for '{search_term}' executed on {selected_site}.")
Once the search is initiated successfully, the script waits for the page to load by checking for the presence of the body tag. If the page loads and the body tag is found, the script checks if the phrase “No results found” is present. If it is, the script raises an AssertionError and displays an error message. If not, it sets a path for saving a screenshot of the loaded page.
The screenshot is then saved using driver.save_screenshot(), and the success is celebrated with a message box confirming that the screenshot has been saved.
# Check if there are results and take a screenshot
WebDriverWait(driver, 10).until(
EC.presence_of_element_located((By.TAG_NAME, "body"))
)
assert "No results found." not in driver.page_source
logging.info("Search results found.")
# Take a screenshot
screenshot_path = f"screenshot_{datetime.now().strftime('%Y%m%d_%H%M%S')}.png"
driver.save_screenshot(screenshot_path)
logging.info(f"Screenshot saved at {screenshot_path}.")
messagebox.showinfo("Success", f"Search completed successfully. Screenshot saved at {screenshot_path}.")
Finally, if any errors occur, the script has built-in error handling that displays error messages and ensures the WebDriver closes properly. This error handling covers scenarios like TimeoutException when a condition isn’t met within the specified time, AssertionError when the expected results aren’t found, NoSuchElementException when an element can’t be found, and WebDriverException for any other WebDriver-related issues.
except (TimeoutException, AssertionError, NoSuchElementException) as e:
logging.error(f"Error with search results: {e}")
messagebox.showerror("Error", f"Error with search results: {e}")
finally:
try:
driver.quit()
except WebDriverException as e:
logging.error(f"Error closing WebDriver: {e}")
Initiating the Search Function
We have already talked about the perform_search() function, but let’s not forget about the start_search() function and why it’s so important:
This function is key because it doesn’t just call perform_search(), it also makes sure everything is ready to go. It grabs the search term you entered using entry.get() and the website you selected from the combobox with site_combobox.get(). If you didn’t provide either of these, it will pop up a warning message asking you to fill them in. But if everything’s in place, it hands things over to perform_search() to get the search going.
def start_search():
search_term = entry.get()
selected_site = site_combobox.get()
if search_term and selected_site:
perform_search(search_term, selected_site)
else:
messagebox.showwarning("Input Error", "Please enter a search term and select a website.")
Setting Up the Main Window
We’re finally at the endgame, this is where we create the main window using tk, set its title, and define its size with geometry.
root = tk.Tk()
root.title("Selenium Search App - The Pycodes")
root.geometry("400x200")
Next, we create a frame to hold our widgets, giving it some padding for a neat layout.
frame = tk.Frame(root)
frame.pack(padx=10, pady=10)
We add a label to indicate the search term entry field and then create the entry widget where the user will input the search term.
label = tk.Label(frame, text="Enter search term:")
label.grid(row=0, column=0, padx=5, pady=5)
entry = tk.Entry(frame)
entry.grid(row=0, column=1, padx=5, pady=5)
After that, we add a label for the website selection and a combobox that lists the website names from our dictionary. We set Python.org as the default selection.
site_label = tk.Label(frame, text="Select website:")
site_label.grid(row=1, column=0, padx=5, pady=5)
site_combobox = ttk.Combobox(frame, values=list(websites.keys()))
site_combobox.grid(row=1, column=1, padx=5, pady=5)
site_combobox.set("Python.org") # Set default value
We then create a “Search” button that, when clicked, calls the start_search() function to initiate the search process.
search_button = tk.Button(frame, text="Search", command=start_search)
search_button.grid(row=2, columnspan=2, pady=10)
Lastly, we use the mainloop() method to start the event loop, keeping the main window running and responsive to user interactions.
root.mainloop()
Example
Full Code
import logging
import tkinter as tk
from tkinter import messagebox, ttk
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException, NoSuchElementException, WebDriverException
from datetime import datetime
# Set up logging
logging.basicConfig(filename='selenium_test.log', level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')
# Dictionary to map website names to URLs and search box names
websites = {
"Python.org": {"url": "http://www.python.org", "search_box": "q"},
"Google": {"url": "http://www.google.com", "search_box": "q"},
"Wikipedia": {"url": "http://www.wikipedia.org", "search_box": "search"},
"Bing": {"url": "http://www.bing.com", "search_box": "q"}
}
def perform_search(search_term, selected_site):
site_info = websites[selected_site]
url = site_info["url"]
search_box_name = site_info["search_box"]
# Start the WebDriver and open the website
try:
driver = webdriver.Chrome()
logging.info("WebDriver started successfully.")
driver.get(url)
logging.info(f"Website {url} opened successfully.")
except WebDriverException as e:
logging.error(f"Error starting WebDriver or opening website: {e}")
messagebox.showerror("Error", f"Error starting WebDriver or opening website: {e}")
return
try:
# Find the search box and perform the search
elem = WebDriverWait(driver, 10).until(
EC.presence_of_element_located((By.NAME, search_box_name))
)
elem.clear()
elem.send_keys(search_term)
elem.send_keys(Keys.RETURN)
logging.info(f"Search for '{search_term}' executed on {selected_site}.")
# Check if there are results and take a screenshot
WebDriverWait(driver, 10).until(
EC.presence_of_element_located((By.TAG_NAME, "body"))
)
assert "No results found." not in driver.page_source
logging.info("Search results found.")
# Take a screenshot
screenshot_path = f"screenshot_{datetime.now().strftime('%Y%m%d_%H%M%S')}.png"
driver.save_screenshot(screenshot_path)
logging.info(f"Screenshot saved at {screenshot_path}.")
messagebox.showinfo("Success", f"Search completed successfully. Screenshot saved at {screenshot_path}.")
except (TimeoutException, AssertionError, NoSuchElementException) as e:
logging.error(f"Error with search results: {e}")
messagebox.showerror("Error", f"Error with search results: {e}")
finally:
try:
driver.quit()
except WebDriverException as e:
logging.error(f"Error closing WebDriver: {e}")
def start_search():
search_term = entry.get()
selected_site = site_combobox.get()
if search_term and selected_site:
perform_search(search_term, selected_site)
else:
messagebox.showwarning("Input Error", "Please enter a search term and select a website.")
# Set up the Tkinter GUI
root = tk.Tk()
root.title("Selenium Search App - The Pycodes")
root.geometry("400x200")
frame = tk.Frame(root)
frame.pack(padx=10, pady=10)
label = tk.Label(frame, text="Enter search term:")
label.grid(row=0, column=0, padx=5, pady=5)
entry = tk.Entry(frame)
entry.grid(row=0, column=1, padx=5, pady=5)
site_label = tk.Label(frame, text="Select website:")
site_label.grid(row=1, column=0, padx=5, pady=5)
site_combobox = ttk.Combobox(frame, values=list(websites.keys()))
site_combobox.grid(row=1, column=1, padx=5, pady=5)
site_combobox.set("Python.org") # Set default value
search_button = tk.Button(frame, text="Search", command=start_search)
search_button.grid(row=2, columnspan=2, pady=10)
root.mainloop()
Happy Coding!
