Scrape at scale running multiple Headful UC Mode of SeleniumBase Instances (Like Selenium Grid)

[juanfrilla]

Hi everyone,

I'm looking for a way to run multiple headful uc mode of seleniumbase instances concurrently, similar to how Selenium Grid works. In my current setup, I use Selenium Grid via Docker Compose, which is great because it allows me to manage and allocate resources efficiently for scalable scraping.

Now with headful uc mode of seleniumbase, however, I’ve only been able to launch a single headful session at a time. Ideally, I’d like to spin up multiple uc sb instances, each with its own attached browser, to perform parallel scraping tasks at scale.

Could anybody guide me in the right direction to do so?

Thanks in advance!

[mdmintz]

There are three different examples for using multiple drivers at the same time with the pure Python formats:

• examples/raw_multi_drivers.py
• examples/cdp_mode/raw_multi_async.py
• examples/cdp_mode/raw_multi_cdp.py

---

Multi-threading in UC Mode is also possible if you use pytest multi-threading provided by pytest-xdist.

You'll need to use command-line options for this format, eg --uc to activate UC Mode, and -n4 (for 4 parallel processes, etc). You can go as high as the number of cores safely. Eg, since my computer has 8 cores, I can do -n8 for 8 parallel processes.

Below is a sample run command, with the options you need:

pytest --uc -n4

---

Also note that UC Mode and CDP Mode aren't compatible with The Selenium Grid, and that Docker leaves a detectable fingerprint.