Rephonic
Artwork for The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

Astronomer
Airflow
Apache Airflow
Data Engineering
Machine Learning
Astronomer
Kubernetes
DBT
Dataops
Airflow 3
Snowflake
Data Pipelines
Data Analytics
Artificial Intelligence
Big Data
Data Quality
Medallion Architecture
Data Governance
Texas Rangers
Airflow 3.0
Large Language Models

Welcome to The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI— the podcast where we keep you up to date with insights and ideas propelling the Airflow community forward. Join us each week, as we explore the current state, future and potential of Airflow with leading thinkers in the community, and discover how best to leverage this workflow management system to meet the ever-... more

PublishesWeeklyEpisodes109Founded9 years ago
Number of ListenersCategory
Technology

Listen to this Podcast

Artwork for The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

Latest Episodes

Migrating two decades of legacy job orchestration is the kind of project most teams quietly avoid. In this episode, Marc Lamberti is joined by [Sanket Patel](linkedin.com/in/sanket-patel-a2a57b3a), Director of Engineering at Trading Technologies, to ... more

How do you orchestrate AI workflows when LLM outputs are non-deterministic and evaluation costs can quietly exceed compute costs? Shawn Feng, Head of Data at [Firework](firework.com), joins the show to walk through how his team uses Airflow to coordi... more

YouTube

Saks Global runs one of the largest retail data operations in the US, with around 8 million SKUs flowing across point-of-sale, e-commerce, catalog, and fraud detection systems into Snowflake. In this episode, [Shailesh Kadam](linkedin.com), Architect... more

YouTube

Airflow 3.3 is here, with a set of features to help with the messy realities of production pipelines: persisting state across retries, reacting intelligently to different failure types, and partitioning assets by more than just time. In this episode,... more

YouTube

Key Facts

Accepts Guests
Accepts Sponsors
Contact Information
Podcast Host
Number of Listeners
Find out how many people listen to this podcast per episode and each month.

Similar Podcasts

People also subscribe to these shows.

Recent Guests

Shawn Feng
Head of Data at Firework
Firework
Episode: Orchestrating AI video intelligence and evaluation pipelines at Firework
Shailesh Kadam
Architect at Saks Global
Saks Global
Episode: Orchestrating Retail Data Pipelines at Saks Global
Kowsy Narayan
Cloud Data Platform Lead, Data Engineering at Ontario Teachers Pension Plan (OTPP)
Ontario Teachers Pension Plan (OTPP)
Episode: Running Airflow 3 in a regulated environment at OTPP
Julian Larralde
Director of Data Engineering at Skimlinks
Skimlinks
Episode: Managing a Customer Analytics Platform with Airflow at Skimlinks
Najeeb Sulaiman
Senior Data Engineer at JLR
Jaguar Land Rover (JLR)
Episode: Building a custom Tableau provider for Airflow at JLR
Mateus Ferreira
Senior Data Engineer at Luiza Labs
Luiza Labs (Magazine Luiza)
Episode: Orchestrating 2,000 Airflow pipelines at Luiza Labs with Mateus Ferreira
William Orgertrice III
Data engineer at Cargill
Cargill
Episode: Enhancing DAGs for Data Processing with William Orgertrice III at Cargill
Shri Hegde
Data and AI engineer and Airflow champion
Astronomer Champions program
Episode: Getting Into Data Engineering with Shrividya Hegde, Data and AI Engineer
Filip Kunčar
Platform Director at ShipMonk
ShipMonk
Episode: Orchestrating DBT With Cosmos and Airflow with Filip Kunčar at ShipMonk Product Development

Host

Kenton Danis
Host of The Data Flowcast; data-focused podcast host with emphasis on Airflow and data platforms.

Reviews

4.9 out of 5 stars from 42 ratings
  • Great way to learn about what data teams are doing in 2024

    [disclaimer: review from former podcast host that has since been replaced by much better voices!]

    If you’re wondering why a data platform and team is important to everyone - from the World Series champion Texas Rangers to worldwide casinos like Wynn to financial services companies - look no further. The techniques

    Apple Podcasts
    5
    Pqdthorne
    United States2 years ago
  • Amazing!

    Loved the show, it helped me wrap my head around most of Airflow’s concepts and how to use different constructs correctly.

    The people invited were top quality and very involved in Airflow development and usage.

    Apple Podcasts
    5
    JeanPieroHM
    Germany7 years ago
  • Helped me a lot as I began exploring Airflow

    I had kept hearing folks talk about airflow, and stumbled across the astronomer podcast as I began trying to learn more. I’ve been quite impressed so far, and am hoping to add airflow to my toolkit

    Apple Podcasts
    5
    ascloyd
    United States8 years ago
  • Use cases

    Very helpful to hear how industry leaders are using Airflow!

    Apple Podcasts
    5
    Wrecklessshiv
    United States8 years ago
  • Great primer for Data Engineering

    If you're new to the field and want to learn techniques, tools, and best practices, this is a great place to start.

    Apple Podcasts
    5
    Appmagnet
    United States8 years ago

Listeners Say

Key themes from listener reviews, highlighting what works and what could be improved about the show.

Great primer for data engineering and Airflow concepts.
Sponsors and tooling discussions are seen as valuable context for production environments.
Listeners praise practical, real-world usage insights from heavy hitters.
High-quality guests and in-depth technical discussions are common positives.
Shows helpful for beginners to understand tools and workflows in data teams.

Chart Rankings

How this podcast ranks in the Apple Podcasts, Spotify and YouTube charts.

Apple Podcasts
#179
Chile/Technology

Talking Points

Recent interactions between the hosts and their guests.

Orchestrating AI video intelligence and evaluation pipelines at Firework
Q: What are the main challenges you face when productionizing AI workflows with Airflow, and how are you addressing them?
Key challenges include lack of determinism in AI outputs, the need for thorough observability and evaluation of model quality beyond task success, and managing costs. Firework addresses these with additional evaluation layers, traceability, regression testing, and cost safeguards like thresholds and smarter scheduling to keep expenses in check.
Orchestrating AI video intelligence and evaluation pipelines at Firework
Q: Why did you choose Airflow as the orchestrator for these AI workflows?
Airflow offered the right balance of flexibility and reliability, and its mature, stable platform is capable of handling diverse automation needs—from SQL transformations to Python tasks and AI-related activities. It provided clear visibility into dependencies and execution, which is critical when integrating multiple AI and data system steps.
Orchestrating AI video intelligence and evaluation pipelines at Firework
Q: What is the business use case for the AI workflows your team is managing?
The primary goal is to improve end-user experience and purchase conversion by making the shopping journey more interactive and personalized through AI. This includes conversational agents that help users discover products and extract insights from interactions, while maintaining a closed loop of data ingestion, model updates, and evaluation to ensure continual improvement.
Running Airflow 3 in a regulated environment at OTPP
Q: Tell me about OTPP's cloud migration journey and your move to Airflow 3 and remote execution.
OTPP moved from on-prem to cloud, adopted a managed Airflow service via Astronomer to reduce infrastructure overhead, and upgraded to Airflow 3 with remote execution to improve security, scalability, and operational simplicity. They planned a quick POV with product engineers and used tooling like Astro CLI to ease upgrades.
Orchestrating 2,000 Airflow pipelines at Luiza Labs with Mateus Ferreira
Q: What led you to develop the YAML wrapper to generate DAGs for non-engineers?
The wrapper is designed to simplify extraction and load processes for business users, bridging data from databases into Lake and BigQuery, with metadata-driven governance and checkpoints to ensure safe, auditable pipelines; it also enables consistent resource configuration across pipelines.

Audience Metrics

Listeners, social reach, demographics and more for this podcast.

Listeners per Episode
Gender Skew
Location
Interests
Professions
Age Range
Household Income
Social Media Reach

Frequently Asked Questions About This Podcast

What is This Podcast about and what kind of topics does it cover?

This show centers on Apache Airflow, data engineering, and AI orchestration, with episodes featuring hands-on practitioners, platform leaders, and open-source contributors. Conversations span real-world deployment in enterprises, evolving tooling (CT L, AI providers, event-driven scheduling), governance and data quality, and practical patterns for building reliable pipelines at scale. A recurring thread is applying Airflow to production-centric use cases—from securing and auditing workflows to integrating AI agents and MLOps components—often with insights on community involvement, upgrade paths, and sponsor-supported managed services. The format typically blends technical deep-dives with pragmatic lessons, making it valuable for engineers, ... more

Where can I find podcast stats for this podcast?

Rephonic provides a wide range of podcast stats for this podcast. We scanned the web and collated all of the information that we could find in our comprehensive podcast database. See how many people listen to this podcast and access YouTube viewership numbers, download stats, audience demographics, chart rankings, ratings, reviews and more.

How many listeners does this podcast get?

Rephonic provides a full set of podcast information for three million podcasts, including the number of listeners. View further listenership figures for this podcast, including podcast download numbers and subscriber numbers, so you can make better decisions about which podcasts to sponsor or be a guest on. You will need to upgrade your account to access this premium data.

What are the audience demographics for this podcast?

Rephonic provides comprehensive predictive audience data for this podcast, including gender skew, age, country, political leaning, income, professions, education level, and interests. You can access these listener demographics by upgrading your account.

How many subscribers and views does this podcast have?

To see how many followers or subscribers this podcast has on Spotify and other platforms such as Castbox and Podcast Addict, simply upgrade your account. You'll also find viewership figures for their YouTube channel if they have one.

Which podcasts are similar to this podcast?

These podcasts share a similar audience with this podcast:

1. Practical AI
2. The Real Python Podcast
3. The a16z Show
4. The Twenty Minute VC (20VC): Venture Capital | Startup Funding | The Pitch
5. All-In with Chamath, Jason, Sacks & Friedberg

How many episodes of this podcast are there?

this podcast launched 9 years ago and published 109 episodes to date. You can find more information about this podcast including rankings, audience demographics and engagement in our podcast database.

How do I contact this podcast?

Our systems regularly scour the web to find email addresses and social media links for this podcast. We scanned the web and collated all of the contact information that we could find in our podcast database. But in the unlikely event that you can't find what you're looking for, our concierge service lets you request our research team to source better contacts for you.

Where can I see ratings and reviews for this podcast?

Rephonic pulls ratings and reviews for this podcast from multiple sources, including Spotify, Apple Podcasts, Castbox, and Podcast Addict.

View all the reviews in one place instead of visiting each platform individually and use this information to decide if a show is worth pitching or not.

How do I access podcast episode transcripts for this podcast?

Rephonic provides full transcripts for episodes of this podcast. Search within each transcript for your keywords, whether they be topics, brands or people, and figure out if it's worth pitching as a guest or sponsor. You can even set-up alerts to get notified when your keywords are mentioned.

What guests have appeared on this podcast?

Recent guests on this podcast include:

1. Shawn Feng
2. Shailesh Kadam
3. Kowsy Narayan
4. Julian Larralde
5. Najeeb Sulaiman
6. Mateus Ferreira
7. William Orgertrice III
8. Shri Hegde

To view more recent guests and their details, simply upgrade your Rephonic account. You'll also get access to a typical guest profile to help you decide if the show is worth pitching.

Find and pitch the right podcasts

We help savvy brands, marketers and PR professionals to find the right podcasts for any topic or niche. Get the data and contacts you need to pitch podcasts at scale and turn listeners into customers.
Try it free for 7 days