Remove all

LOADING ...

Content

Python static site parser

10.12.2024

19.08.2026

5 minutes

993

Tags:

Introduction/About the scraper

In this article, I'll give an example of a very simple Python scraper. I'll explain how it works, how it's structured, how to use it, and how to customize it. This parser is ideal for any static website. Plus, as a bonus, I'll show you how to write an identical parser without using external libraries.

A static website is one that consists of pre-created files (HTML, CSS, and JavaScript). All pages are stored on the server in their final form. The text and images on such a site are the same for all visitors, and the server doesn't reassemble the pages each time they visit.

Since I haven't yet received permission to parse other people's websites, I decided to use my own as an example. We'll be parsing the "about the website page", specifically the links.

Creating a scraper

Configuring and preparing a virtual environment

Let's start by installing the necessary packages and importing them. Of course, you can do this without any libraries at all, but this takes time and requires much more coding and control. It's better not to reinvent the wheel and use ready-made solutions. We'll need just three packages:

requests - For sending requests to the target site and retrieving web pages
beautifulsoup4 - For the actual scraping. Finding the necessary elements and extracting the required data from them
lxml - beautifulsop4 can't work with HTML or XML files on its own, so it needs help in the form of a library like this one.

Next, create a directory for the project, add a virtual environment, install the above packages, and create the main script file:

mkdir MyScraper; python -m venv ./MyScraper/venv; source ./MyScraper/venv/bin/activate; pip install BeautifulSoup4 lxml requests; touch ./MyScraper/main.py;

For Linux/Unix(Bash) systems

New-Item -ItemType Directory -Path "MyScraper" -Force python -m venv ./MyScraper/venv ./MyScraper/venv/Scripts/Activate.ps1 pip install BeautifulSoup4 lxml requests New-Item -ItemType File -Path "./MyScraper/main.py" -Force

For Windows(PowerShell) systems

The database and virtual environment are ready, now we can move on to the script itself - main.py

Creating a scraper in Python

I like to start creating parsers by importing all of their dependencies/libraries, defining constants/target website URLs, and creating a basic structure of functions.

Add the program entry point to the very end of the file. This line specifies that when this file is run through the Python interpreter, it will execute the main run function.

Next, we need the scraping function itself. This is where we specify what we're scraping, how we're scraping, and what we're scraping. This creates a soup...

Soup is a standard name for what the BeautifulSoup constructor returns. It's accepted, but you're free to call it what you want.

In this "soup," we find all the links and extract their data, stored in the href attribute. This creates a list that can then be used for any purpose, for example, to crawl newly found pages:

def run(): # Make a request response = requests(URL) # Save all the source code of the page page = response.text # Make a soup soup = BeautifulSoup(page, "lxml") # Find all the links soup_links = soup.find_all('a') links = [] # And after extracting all of them save them for soup_link in soup_links: links.append(soup_link['href'])

All together it will look like this:

# Dependencies import requests from bs4 import BeautifulSoup # Target page URL="https://timthewebmaster.com/ru/about-website/" # This is placeholder for scraper def run(): # Make a request response = requests(URL) # Save all the source code of the page page = response.text # Make a soup soup = BeautifulSoup(page, "lxml") # Find all the links soup_links = soup.find_all('a') links = [] # And after extracting all of them save them for soup_link in soup_links: links.append(soup_link['href']) # Entry point if __name__ == "__main__": run()

This is for those who just want to copy everything.

This is what a simple static site and page scraper looks like. The scraper's output should look something like this:

I also added print functions to the script to make the process visible, but otherwise the script is the same.

However, if the website has any security, such as a limit on the number of the requests, or IP blocking, or if the site is dynamic, this parser won't work. In that case, you'll have to write special scraper for dynamic sites - I wrote about them in a separate article.

Python Parser Without Dependencies (Optional)

This chapter is similar to the previous chapter, with the only difference being that I won't use any external dependencies and will use only built-in Python tools. To replicate the functionality of the previous parser, we'll need to implement the following replacements:

requests library - making requests and getting page sources
beautifulsoup4 library - parsing the necessary data

Our template script will look like this (this is without the request logic and data parsing):

from urllib.request import urlopen, Request from html.parser import HTMLParser URL="https://timthewebmaster.com/ru/about-website/" def run(): pass if __name__ == "__main__": run()

I imported some functions and classes from the standard library. For example, from the urllib library, I need the urlopen function and the Request class, and from the html library, I need the HTMLParser template parser.

We could go even deeper and write our own modules for processing and sending requests and parsing raw text. This would be in C, but let's not dig that deep and stick to Python.

This is really all we need, and we're ready to write the updated parser:

from urllib.request import urlopen, Request from html.parser import HTMLParser URL="https://timthewebmaster.com/ru/about-website/" class SimpleHTMLParser(HTMLParser): def handle_starttag(self, tag, attrs): if tag == "a": for attr in attrs: if attr[0] == "href": print(f"link->{attr[1]}") def run(): # Make a request request = Request(URL, method="GET") # Send the request, open and save it with urlopen(request) as response: page = str(response.read()) parser = SimpleHTMLParser() parser.feed(page) if __name__ == "__main__": run()

The urlopen function works the same way as a regular open function for files, creating a special context manager that can then be used. Incidentally, you don't have to create a request object (of the Request class) and just send a link, but I think this will make it clearer what we're doing.

After receiving the response, we save it as a string and pass it to the custom parser, SimpleHTMLParser. This parser, in turn, processes the resulting page as we configured it. In my opinion, it's even simpler than using external libraries.

Conclusion

These parsers are very simple to implement and are ideal for parsing any static sites, although they still need to be customized for the specific needs of a specific site.

Although the second parser seems simpler to write, at least to me, I recommend using the first option. It will be faster and more reliable, because if you take the lxml parser, for example, it's written in C, and it's been around for quite a while, so there should be almost no errors. But in any case, I hope this article was useful to you.

Types and features of Telegram bots

Next article

COBOL programming language history by version

Previous article