Weikang Z.

Do you need data collected from websites, APIs, documents, videos, or audio—but cannot rely on the current process to produce complete, accurate, and regularly updated results? I build reliable data pipelines that collect, clean, validate, store, and deliver business-ready data. My work covers the complete data flow: Data Sources → Extraction → Cleaning → Matching → Validation → Storage → Monitoring → Delivery I can help when: • A website is difficult to extract data from because of dynamic content, inconsistent structures, login flows, rate limits, or frequent layout changes • An existing scraper has stopped working or produces incomplete or duplicated records • Data from multiple sources must be standardized, matched, and combined • Video, audio, PDFs, or other unstructured content must be converted into structured data • A recurring data process needs scheduling, logging, retry logic, and failure alerts • Large datasets need a suitable database structure for reliable querying and analysis • Analysts or business teams do not trust the data they receive WHAT I BUILD • Web scraping and API data pipelines • Scheduled and incremental data collection systems • Data cleaning, deduplication, matching, and validation workflows • Product, price, inventory, and competitor monitoring systems • Video, audio, text, and document extraction pipelines • PostgreSQL, MySQL, MongoDB, and Hive data storage solutions • Automated data quality checks and exception reports • Pipeline monitoring, logging, retry, and email alert systems • Analysis-ready datasets for reporting, machine learning, and business decisions RELIABLE DATA EXTRACTION I have experience collecting large volumes of data from dynamic and irregular websites, including platforms such as TikTok, Reddit, Instagram, and Douyin. Depending on the source and project requirements, I work with APIs, HTTP requests, browser automation, Selenium, Scrapy, Playwright, and Beautiful Soup. I design extraction processes with maintainability in mind. This includes pagination, session handling, incremental updates, retry rules, duplicate prevention, source identifiers, and clear failure records. DATA QUALITY AND PROCESSING Collecting records is only the beginning. I use Python, Pandas, NumPy, and SQL to transform raw data into consistent and usable datasets. A typical quality workflow can include: • Schema and required-field validation • Missing-value and duplicate checks • Data type and format standardization • Cross-source matching and reconciliation • Business-rule validation • Low-confidence or unmatched record queues • Source-to-output traceability • Data quality summaries and exception reports UNSTRUCTURED DATA PROCESSING I use Gemini, GPT, and their APIs to process videos, audio, text, and documents in batches. These workflows can include transcription, classification, structured field extraction, schema validation, and human review for uncertain results. For TikTok and Douyin projects, I processed large volumes of audio and video information to support structured analysis of social trends. PIPELINE, STORAGE, AND MONITORING I select an appropriate storage design based on data volume, update frequency, access patterns, and reporting needs. My database experience includes PostgreSQL, MySQL, MongoDB, and Hive. For recurring pipelines, I can implement: • Scheduled and incremental runs • Logging and processing history • Automatic retries • Data freshness checks • Record-count and threshold monitoring • Schema-change detection • Detailed exception notifications • Email alerts when a process or quality check fails • Docker-based deployment and reproducible environments EXPERIENCE I hold a master’s degree in Computer Science and have more than six years of experience in the data industry. I spent six years at VIVO Communication Technology Co., Ltd., where I worked with large-scale data collection, data modeling, data warehousing, processing, and analysis. This experience taught me to approach data projects as production workflows rather than isolated scripts. A successful system must continue running, make failures visible, preserve data quality, and remain understandable to the people who use it. More than 50% of my Upwork business has come from long-term cooperation. Clients value my ability to respond quickly, understand changing data requirements, and maintain complex data processes over time. CORE TECHNOLOGIES • Data engineering: Python, SQL, Pandas, NumPy, Airflow, Shell • Data extraction: Scrapy, Playwright, Selenium, Beautiful Soup, APIs • Databases: PostgreSQL, MySQL, MongoDB, Hive • Automation and deployment: Docker, Git, AWS, scheduled workflows • AI-assisted processing: Gemini API, GPT API, structured extraction and classification • Analysis: LightGBM, XGBoost, KNN, Matplotlib, Power BI, Tableau HOW I APPROACH A PROJECT 1. Define the data source, fields, update