Ross — Backend Development freelancer in Leicester, England, United Kingdom

A multi-threaded solution employed, with a headless Selenium web driver per thread, in order to provide scalability to the solution and process batches in a reasonable amount of time. Selenium being a necessity in order to scrape the dynamic elements. Rotational proxies were also used in order to bypass twitter rate limitation. The solution was deployed on a server (DigialOcean) with a basic dashboard created using Flask & Dash to enable a user to specify the number of threads and to download\u002Fupload datasets.