Manikandan K.

6+ years of professional experience in data extraction and web scraping, delivering 400+ successful website scraping projects for diverse clients, including end-to-end process automation solutions. I have extensive expertise in the development of web crawlers🕷️ that are able to, 🧑‍💻 Android Mobile Application data scraping using tools such as Android Debug Bridge (ADB) and the Android emulator to interact. 🧑‍💻 Authenticate themselves and manage sessions (sign in/sign out) 🧑‍💻 Render dynamic content such as JavaScript using browser simulation 🧑‍💻 Extract various types of data including HTML, text, images, tables, API and documents 🧑‍💻 Save extracted data to a file (csv, json, xlsx, xml, pdf, html, etc) 🧑‍💻 Insert it into a database (SQL & NoSQL) 🧑‍💻 Integrate file delivery and error notification through slack or email. 🧑‍💻 Deployment to the cloud for efficient and scalable scraping. 🧑‍💻 Experienced in selecting and configuring appropriate proxy servers and implementing proxy rotation strategies. 🧑‍💻 Proficiency in using crontab and scrapyd for scheduling tasks in our project. 🧑‍💻 Redis Queue to improve the speed and efficiency of our data extraction process. 🧑‍💻 Built APIs using FastAPI that allow users to access and interact with the data extracted from websites through web scraping. Tech stack includes: 💻 Programming languages: Python, R, Shell script 💻 Visualization Tool: Microsoft PowerBI 💻 Addressing Languages: Xpath, css 💻 Scraping libraries: scrapy, aiohttp, selenium, playwright, splash, BS4, Requests 💻 API Creation: Fastapi-python 💻Web automation: Selenium, playwright, Puppeteer 💻 Queuing libraries: Redis with python 💻 Data transformation: Pandas 💻 Databases: MySQL, SQL server, Postgres, Mongodb 💻 Cloud platforms: AWS, Azure (certified). 💻 Scraper deployment: Zyte, scrapydweb (self-managed) 💻 CI/CD: Docker 💻 Project & Code management: Github, Jira 💻 Api tools: Postman