In most cases you would capture the request and change the "page=" parameter (either for an HTML page or an API).
You could also use selenium to click on each "next page". Could be parallelized with multiple tabs / windows.
The only website that blocks me is Bloomberg because they detect mitmproxy (I didn't care enough to make mitmproxy harder to detect).
Another detail is that regular Chrome doesn't let you load insecure certificates while chromedriver allows that.
Anyway, I will write about all that, I already posted some code on my Twitter: https://twitter.com/localhostdotdev (that I will turn into a blog).
In most cases you would capture the request and change the "page=" parameter (either for an HTML page or an API).
You could also use selenium to click on each "next page". Could be parallelized with multiple tabs / windows.
The only website that blocks me is Bloomberg because they detect mitmproxy (I didn't care enough to make mitmproxy harder to detect).
Another detail is that regular Chrome doesn't let you load insecure certificates while chromedriver allows that.
Anyway, I will write about all that, I already posted some code on my Twitter: https://twitter.com/localhostdotdev (that I will turn into a blog).