Pankaj M.

I am 🏆 Certified Databricks (Associate ,Professional & GenAI) and 🏆 Microsoft Fabric & Azure Data Engineer with 8+ years of experience in ETL, leveraging Microsoft Azure technologies, Databricks, BigQuery, Microsoft Fabric, GCP, PySpark, SQL, MS SQL Server, Snowflake, and MySQL to deliver top-tier data solutions. My expertise spans a wide range of technologies and tools, ensuring seamless data movement, data scraping and transformation. I am skilled in orchestrating complex data workflows, automating pipelines, and streamlining ETL processes using AWS Glue, AWS EMR, Azure Data Factory, Databricks,Apache Airflow, SSIS Pentaho, Talend, Informatica etc. I excel at integrating with top-tier data warehouse platforms such as Snowflake, Redshift, Azure Synapse Analytics, and BigQuery, building robust ETL pipelines while managing data lakes on ADLS, S3, Databricks Lakehouse Platform and GCS. 🏆 Key Areas of Expertise: 📌 Development Skills: Databricks, Unity Catalog, Lakeflow, Kafka, Airflow, Airbyte, MS-SQL Server, Power BI, Tableau, Qlik, Apache Superset, Azure Data Factory, MySQL, ETL, Job Scheduling and Automation, Query Optimization etc. Performance Tuning 📌 Programming Languages: Python, SQL, PySpark 📌 Databases: SQL Server, PostgreSQL, MySQL, DynamoDB, Amazon RDS 📌 ETL Tools: AWS Glue, AWS EMR, Azure Data Factory, SSIS, Apache Airflow, Pentaho, Talend, Informatica. 📌 Data Scraping: Expertise in Python, Selenium, Playwright, Puppeteer and BeautifulSoup. 📌 Data Warehouse Platforms: Azure Synapse Analytics, Redshift, BigQuery, Snowflake 📌 Data Storage Services: Amazon S3, Azure Data Lake Storage (ADLS), Google Cloud Storage (GCS), Databricks File System (DBFS) 📌 Big Data Technologies: Hadoop, Hive, HBase 📌 Workflow Orchestration: Apache Airflow, Azure Data Factory 📌 Data Lake Management: Delta Tables, Delta Live Tables, Data Flows, Jobs 📌 Visualization Tools: Power BI, Tableau, Superset 📌 IDE’s: Databricks, SQL Server Management Studio, DBeaver, Snowflake, Azure Data Studio 📌 Cloud Platforms: Azure, GCP, AWS, Snowflake. Our comprehensive skill set, and hands-on experience make us adept at tackling any data engineering challenge thrown our way.