Rishi P.

🚀 Data Engineer | Snowflake | DBT | GenAI | AI Agents | Azure | AWS | GCP I’m a Data Engineer with 5+ years of experience designing scalable data platforms, modern ELT pipelines, and AI-powered data systems. I specialize in building production-grade data infrastructure, migrating legacy data warehouses to modern cloud platforms, and developing AI-driven data products. I’ve worked with enterprise clients and startups to build high-performance pipelines, automate large-scale data ingestion, and develop intelligent systems powered by LLMs. Currently, I collaborate with a world-class founding team building cutting-edge AI and data products, architecting scalable infrastructure and developing AI-powered solutions. 🔹 What I Can Help You With ✅ Modern Data Platforms & Data Warehousing Design and implement scalable architectures using Snowflake, Azure Synapse, Databricks, and Microsoft Fabric. ✅ ELT / ETL Pipelines & Data Engineering Build robust pipelines using dbt, SQL, Python, Spark, Kafka, and Azure Data Factory, supporting high-volume data ingestion and transformation. ✅ Data Warehouse Migration Led migrations of legacy Oracle and on-prem SQL Server warehouses to Snowflake, improving processing speeds and reducing operational overhead. ✅ AI Agents & Generative AI Systems Develop AI agents and intelligent applications using GPT, Claude, Vertex, and Bedrock, integrated through FastAPI APIs and scalable backend systems. ✅ Generative AI Data Systems Built RAG-based systems combining IoT and weather data with LLMs for advanced analytics and automated insights. ✅ Large-Scale Web Scraping & Data Automation Engineered distributed scraping pipelines using Python and Selenium to extract structured data across 50+ state legislature websites, enabling automated daily data updates. ✅ Streaming & IoT Data Processing Built real-time and near-real-time pipelines using Spark Structured Streaming, Kafka, and Event Hub. ✅ Data Pipeline Optimization Reduced pipeline execution times significantly by optimizing SQL transformations, dbt models, and multi-threaded ingestion frameworks. 🔹 Cloud, Data & Infrastructure Stack • Data Platforms: Snowflake, Databricks, Azure Synapse, Microsoft Fabric • Data Engineering: dbt, SQL, Python, Spark, Kafka, Data Factory • AI / LLM Systems: GPT, Claude, Vertex, Bedrock, RAG, Prompt Engineering • Infrastructure: AWS, GCP, Azure, Supabase, Vercel, Logfire, Clerk • Databases: PostgreSQL, DynamoDB, OpenSearch, MongoDB • Visualization: Power BI 🔹 Specialized Experience • Architecting scalable AI-powered data products and AI agents • Building ELT pipelines processing tens of gigabytes of data daily • Developing large-scale web scraping frameworks and automated data ingestion systems • Handling datasets exceeding billions of records with optimized ETL and storage strategies • Designing Power BI dashboards and analytics platforms for enterprise teams 🔹 Selected results • 5× platform scale-up: 3M → 15M companies and 28M → 136M people • 5+ new markets launched with 1M+ companies added in one release • 20+ Snowpipes monitored and optimized • 60%+ runtime reduction through pipeline and ingestion optimization • 80%+ manual effort reduction through automated data scraping • 50GB+ daily data movement through Azure Synapse pipelines 🏆 Certifications Microsoft Certified: DP-600 | DP-203 | DP-900 | AZ-900 💡 I focus on delivering production-ready data systems and AI-driven platforms that scale reliably and generate real business value. If you're looking for someone who can build modern data pipelines, migrate data warehouses, or develop AI-powered data applications, I'd be happy to help. 📩 Let’s discuss your project!