Open Source Data Platform Engineer specialising in Apache Airflow & dbt orchestration.

Apache Airflow Committer · Astronomer Cosmos Committer · Conference speaker & workshop host on data orchestration and OSS platforms

Helping teams design, scale, and operate reliable data pipelines.

• Design & build production-grade data orchestration platforms
• Contribute to and help maintain the Apache Airflow ecosystem
• Help companies adopt dbt + Airflow correctly (architecture, performance, reliability)
• Advise teams on OSS strategy, integrations, and platform maturity

• Orchestration: Apache Airflow (core + providers)
• Data Engineering: dbt, SQL, analytics engineering
• Platforms: Kubernetes, Docker, Cloud (AWS / GCP / Azure)
• Languages: Python (primary), SQL
• Open Source: Design, reviews, long-term maintenance

I actively contribute to and help maintain core projects in the Apache Airflow ecosystem, with a strong focus on production reliability, performance, and developer experience.

Currently working on the Apache Airflow ecosystem at Astronomer.

• Apache Airflow
  Contributions across operators, providers, security, and performance improvements https://github.com/apache/airflow

• Astronomer Cosmos (dbt ↔ Airflow integration)
  Contributor to execution modes, asynchronous patterns, remote caching, and dbt orchestration design https://github.com/astronomer/astronomer-cosmos

• Dag Factory
  Improving DAG generation patterns, usability, and maintainability
  https://github.com/astronomer/dag-factory

• Astronomer Providers
  Enhancements and fixes across Airflow providers for real-world asynchronous production use cases
  https://github.com/astronomer/astronomer-providers

• Great Expectations Airflow Provider
  Contributions towards CI pipelines, validation workflows, and Airflow integration https://github.com/great-expectations/airflow-provider-great-expectations

• Boosting dbt-Core Workflow Performance with Airflow’s Deferrable Capabilities
  Airflow Summit 2025 — Talk
  https://airflowsummit.org/sessions/2025/boosting-dbt-core-workflows-performance-with-airflows-deferrable-capabilities/

• Productionising dbt-Core in Airflow
  Airflow Summit 2025 — Workshop
  https://airflowsummit.org/sessions/2025/productionising-dbt-core/

• Overcoming Performance Hurdles in Integrating dbt With Airflow
  Airflow Summit 2024 — Talk
  https://airflowsummit.org/sessions/2024/overcoming-performance-hurdles-in-integrating-dbt-with-airflow/

• Optimizing Airflow Performance: Strategies, Techniques & Best Practices
  Airflow Summit 2024 — Talk
  https://airflowsummit.org/sessions/2024/optimizing-airflow-performance-strategies-techniques-and-best-practices/

📩 Reach out:

Always happy to discuss architecture, open-source collaboration, and data orchestration challenges.

• LinkedIn: https://www.linkedin.com/in/pankajkoti
• Email: pankajkoti699@gmail.com