Remove all

LOADING ...

Content

Scraping dynamic websites - methods and code examples

18.08.2026

9 minutes

Tags:

Scraper

Dynamic parser

An introduction to website types and their differences

Back in the day, when I was a child, during my childhood, all websites were static. That is, they consisted only of HTML, CSS, and a few snippets of JavaScript. Back then, if you got a page, you knew that was all the site had to offer (in terms of content).

A static website is one that consists of pre-generated files (HTML, CSS, and JavaScript). All pages are stored on the server in their final form. The text and images are the same for all visitors, and the server doesn't reassemble the pages each time.

Scraping such sites is a breeze; you just need to know what you need to scrape and where it is. Now, however, most websites on the web are dynamic.

According to w3techs, 69.1% of all websites use a CMS (yes, sites with a CMS can be considered dynamic). And that's just the CMS; the number of dynamic websites is naturally much greater, given the recent popularity of various frontend frameworks, which are essentially designed for dynamic websites. Incidentally, my website is also considered dynamic.

What does a dynamic website mean? It means that the page you see in your browser isn't stored separately somewhere on the server, but is assembled on the fly and served when ready. This means a website doesn't have to use JavaScript to be dynamic; the fact that the page is assembled is enough.

A dynamic website is one whose content changes and is reassembled for each user in real time. The website may use a database, templates, or specialized code to display the page based on the user, their actions, or the time of day.

Parsing such websites is significantly more difficult because they require:

Either precise timing - the page can't be downloaded immediately, only after the required content appears on the page.
Or JavaScript rendering - sometimes direct interaction with the site via clicking, scrolling, or focusing on an element is required for the required content to load, and this content is loaded via JavaScript.

Preparation: what, who, and how we'll scrape

As an example of parsing, I'll use my own website. We'll scrape article cards (Title, Link, and Description).

My website uses a paginator and dynamic loading upon reaching a certain height. Also, although not always, websites can update the current URL when loading new content, like mine does. For example, changing the current page number or adding a filter.

There are four methods for parsing, or rather, obtaining the necessary data from this type of website:

1. Browser emulation and rendering of JS scripts

Direct browser emulation via drivers - Selenium
Browser emulation via special software APIs - Playwright / Puppeteer / Cypress

2. Reverse engineering the website and its requests

Using an open, specially developed API for third-party parsers
Finding and using a hidden API not provided by the developer

3. Semi-automated via browser extensions

4. Gaining direct access to databases and the website (i.e., hacking the target)

I will only discuss the first three, as the fourth is illegal. Please don't hack other people's websites. This way, we'll write scrapers that will do almost the same thing, but in different ways and using different technology stacks. I'll also discuss the advantages and disadvantages of each of these methods.

In all subsequent chapters, I'll focus solely on accessing the necessary scraping material—HTML pages—without going into detail about extracting data from the pages themselves using BeautifulSoup.

I'll also write the most simple and straightforward code possible, without optimization gimmicks or gimmicks like multithreading, caching, or rotation. All of this is self-evident and used in real-world projects, but it's unnecessary for tutorials.

We'll start with the first point and the first subpoint—using Selenium.

Scraping a dynamic websites in different ways

Website scraping using Selenium

The main advantage of parsing websites with Selenium is that it supports a huge number of browsers, including Chrome, Edge, Firefox, Safari, and Internet Explorer. And an equally large number of programming languages: Python, Kotlin, JS, Java, C#, and Ruby.

However, it has one significant drawback: it's very slow due to the fact that all interaction occurs through special drivers that emulate browser behavior.

I'll be writing this parser in Python, and the rest of the environment will be built around this language. Create a directory and virtual environment for the project, and install the necessary packages:

mkdir dynamic-scraper; cd dynamic-scraper; python -m venv .venv; source .venv/bin/activate; pip install selenium;

Once the project directory has been created and everything is configured, let's add the main script file. In it, we'll launch the web driver, make the first request to the first page, and save it.

from selenium import webdriver from selenium.webdriver.firefox.options import Options # Save the page def save_to_html(path, text): with open(path, 'w', encoding='utf-8') as file: file.write(text) file.close() TARGET_URL = "https://timthewebmaster.com/en/articles/" def run(): # Setting up browser's options options = Options() # Without GUI options.add_argument('--headless') driver = webdriver.Firefox(options=options) # Get the target page driver.get(TARGET_URL) # Save the target page save_to_html('index.html', driver.page_source) if __name__ == "__main__": run()

So far, there haven't been any changes. We could get the exact same page without Selenium. After all, we still have 15 pages that we can't access except through Selenium.

To parse all available pages, we'll either need to click all 15 buttons or scroll down to the bottom. First, let's see what it looks like if we parse by clicking buttons:

from selenium import webdriver from selenium.webdriver.firefox.options import Options from selenium.webdriver.common.by import By from selenium.webdriver.support.ui import WebDriverWait from selenium.webdriver.support import expected_conditions def save_to_html(path, text): with open(path, 'w', encoding='utf-8') as file: file.write(text) file.close() TARGET_URL = "https://timthewebmaster.com/en/articles/" NEXT_PAGE_BUTTON_ID = "next_pagin_button" LAST_BUTTON_ID = "paginator_last_page_example" NEXT_LOCATOR = "scroll-sentinel-" def run(): # Setting up a browser options = Options() options.add_argument('--headless') driver = webdriver.Firefox(options=options) # On the target website driver.get(TARGET_URL) # Figure out a number of pages pages = int(driver.find_element(By.ID, "paginator_body").find_element(By.ID, LAST_BUTTON_ID).get_attribute("data-page")) for page in range(2, pages): # Find and click the next button next_page_button = driver.find_element(By.ID, NEXT_PAGE_BUTTON_ID) next_page_button.click() # Wait till content are loaded WebDriverWait(driver, 10).until( expected_conditions.presence_of_element_located((By.ID, f"{NEXT_LOCATOR}{page+1}")) ) print(f"STATUS: On {page} page") next_page_button = driver.find_element(By.ID, NEXT_PAGE_BUTTON_ID) next_page_button.click() # Save the page once save_to_html(f'index.html', driver.page_source) if __name__ == "__main__": run()

Initially, we might not know how many pages are available, so after we visit the site, we look at the number of available pages on the paginator.

Then we create a loop and click the "Next" button to get to the next page.

But how do we know if the page has loaded or is still loading? There are many options if you're using Selenium. You can either specify how many seconds you're willing to wait, or wait for the appearance, absence, or visibility of a specific element on the page, using a special construct with WebDriverWait.

Why did I only save the page once? It all depends on the specific site. In my case, my site simply inserts articles onto the same page as we move forward. Some sites reload the entire page when moving to a new page, which would require saving each page.

Without the --headless flag, it would look something like this:

Scraping a website using Playwright

Although Playwright supports Python, I still prefer to work with it as an npm project. Unlike Selenium, it's much faster. However, it doesn't support all browsers, only Chromium (Google Chrome, Edge), WebKit (Safari), Gecko (Firefox), and their derivatives.

But the scraper doesn't really care which browser you use. It only needs the desired HTML page, and then it's a matter of technique...

So, after installing Playwright, to scrape all available articles from the pagination page, you can write the following script:

import { test } from '@playwright/test' import { writeFileSync } from 'fs'; let URL = "https://timthewebmaster.com/en/articles/" function saveToJson(data, filename){ const articlesArray = Array.from(data.values()); writeFileSync(filename, JSON.stringify(articlesArray, null, 2), 'utf-8'); } test('main', async ({ page }) => { let collectedArticles = new Map(); await page.goto(URL); // How many pages let pages = parseInt(await page.locator('#paginator_last_page_example:not([data-page=""])').innerText()) // Click through all for (let i = 0; i < pages; i++){ console.log(`page=${i}`) await page.locator('#next_pagin_button').click() // Wait till required element are loaded if (i < pages - 2) await page.locator(`#scroll-sentinel-${i+3}`).waitFor({ state: 'attached' }); } // Collect all loaded articles const articles = page.locator('#page>div:not(.scroll-sentinel)'); const currentCount = await articles.count(); for (let j = 0; j < currentCount; j++){ const article = articles.nth(j); // Extract the data const title = await article.locator('h2').innerText(); const link = await article.locator('h2>a').getAttribute('href'); collectedArticles.set(`article-${j}`, {"title": title, "link": link}) } saveToJson(collectedArticles, 'results.json') });

This scraper clicks the "Next" button and only at the end scrape the entire loaded page. This could be done in a loop if this site's paginator simply replaced one article with another. But since it simply appends new articles to existing ones, it will work like this.

Scraping such a page without multithreaded code will take about 10-30 seconds. By default, any test in Playwright only lasts 30 seconds, so don't forget to remove the test time limit. timeout: 0 in playwright.config.js

When scraping dynamic sites, the most important rule is to understand this. Everything takes time, especially when it comes to the user interface, so waiting and handling events when certain elements are ready is everything.

Website scraping using an existing API

This is probably the most reliable and fastest way to collect data from a target website. Although it's quite rare, it's usually done only by large sites like Google or very small sites run by enthusiasts and geeks like me. :)

Yes, if you didn't know, Google officially allows you to scrape search results and much more, and it's relatively free. I even have an entire article dedicated to scraping Google's SERP results.

The existence of such an API is usually mentioned directly. After all, it should reduce the server load. What's simpler? Requesting 1,000 entire HTML pages of a website, or 100 lightweight JSON files that already contain all the existing information—just copy and paste.

My website also has an open API located at this address: https://timthewebmaster.com/api/v1/public/

So, after playing around a bit, you can figure out that to get all the articles you need, you can create a GET request to https://timthewebmaster.com/api/v1/public/article/?page=1&page_size=100&format=json and retrieve all the articles at once. In a script, it would look like this:

import json import requests from bs4 import BeautifulSoup TARGET_URL = "https://timthewebmaster.com/api/v1/public/article/?page=1&page_size=10&format=json" def save_to_json(path, list): with open(path, 'w', encoding='utf-8') as file: json.dump(list, file, indent=4, ensure_ascii=False) def run(): # Make an init request response = requests.get(TARGET_URL) all_data = json.loads(response.text) data = all_data while(data['next']): print(f"PAGE: {data['next']}") response = requests.get(data['next']) data = json.loads(response.text) # Save the data all_data['results'].append(data['results']) save_to_json('data.json', all_data) # Logic for organizing and actual scraping for example via bs4 # ... if __name__ == "__main__": run()

Then, all that's left to do is iterate through the resulting list and select only the data you need. Having such an API greatly simplifies the work, essentially reducing it to sorting and packaging the data into more convenient and relevant formats for the client.

But again, this option is very rare. It either won't exist at all, or the API won't be explicit. The next chapter will cover how to find such an API.

Site Parsing Using a Hidden API

The script will be identical to the previous one. The only difference is finding such an API. To find such an API, you can use the developer console. In the console, go to the Network tab and select XHR and JS. Next, interact with the page (click a button, hover over it, scroll down) and monitor the intercepted requests.

The server can return these POST requests, or rather responses to them, in various formats. They can range from a simple piece of HTML code to a full-fledged JSON file. It all depends on the specific site. In my case, a rendered HTML fragment is returned, which then needs to be parsed using BeautifulSoup or other parsers.

Also, informally, the address itself can be a hidden API that allows you to parse all articles from the site.

Site scraing via a browser extension

This method is ideal if you need to parse data from websites under any, even the most sophisticated, protection quickly and immediately, but a real user will have to access such sites. Take, for example, my recent browser extension for scraping the Avito marketplace; it is considered like a very tough website to scrape anything. But with an extension, you can get a pretty useful personal scraper.

Of course, this completely negates the automation aspect of any scraping. But still, it can be sometime useful.

Yes, you can get creative and install such extensions on Selenium or Playwright, but it won't be much use. It won't help much in bypassing bot protection mechanisms. The only advantage of using extensions this way might be what I call "toolset swapping." That is, you can easily break through security using, for example, Python/Selenium, but parse and collect data using JS.

Conclusion

That's how you can scrape dynamic websites. There are two ways: either mimic browser behavior and render JS, or look for hidden (or not so hidden) APIs of these websites. Of course, parsing through these APIs is faster, cheaper, and easier, but sometimes website creators going insane and do everything possible to prevent their resource from being parsed.

It's understandable; it's unclear who makes hundreds and hundreds of requests to a web resource and why, thereby overloading it.

Therefore, before you start scrape a website, make sure you won't be hitting their servers too hard or too noticeably. Collect only the essentials. And if you need some practice, you're welcome to scrape my website :) Have a nice day.

Where to place a telegram bot, and where not to

Previous article