How scraping is done

It all depends on what to scrape and what to scrape with. You can scrape documents and tables, or you can scrape a websites. Moreover, websites are more difficult to scrape from than documents, because there are many websites and each has its own architecture, which greatly complicates scraping.

The general scheme of scraping looks like this:

The first step is also the most difficult. There are many ways to protect content from scraping. Starting from blocking it completely (you can’t get it without a password), ending up with blocking by IP or HTTP headers.

Extracting data is relatively easy. Usually it is text or some statistical information.

A lot can be said about processing and saving information. I will only say that information is usually processed and saved in a pre-specified format from the customer or based on the need.

There is usually a lot of data, and in order to scrape it all, very large computing capabilities are required using multithreading and cloud computing.

Usually, a scraper is written in Python, not in pure Python, of course, but using the appropriate libraries. And the advantage of this language is that for any information, in whatever form and format it was presented, there is a library.