Pedro S.

Data Engineer | Data Scraping | Databricks | Cloud | Python

I have almost 5 years of experience working with data, having started as a data analyst, then data scientist, and for the past 4 years, as a data engineer. I've worked in many projects related to API consumption, web scraping and automation. I work with Python, SQL, PySpark, and AWS services like Glue, S3, Redshift, and IAM. Also have got experience with Azure services like ADLS and Functions. I have hands-on experience building scalable, production-grade pipelines using the medallion architecture, with automation and orchestration through serverless services. Experience with infrastructure-as-code practices using Terraform. I'm passionate about clean engineering, automation, and creating end-to-end data solutions that drive business value. Related to web scraping these are the tools I use depending on the need: requests, selenium, bs4, playwright. To deal with anti-bot we can always use proxies (that we can gather for free and I do have a database for that already), user-agents and cookies to mimic human like behavior. We scrape the data as json, html, xml or plain text and turn them into structured data as an EXCEL file, csv, database etc.