Overview

Hello and welcome to the Apache Airflow Bootcamp: Hands-On Workflow Automation with Practical Examples! Throughout my career, I've built and managed countless workflows using Apache Airflow, and I'm excited to share my knowledge with you.This course is designed to take you from a complete beginner to a confident user of Apache Airflow. We'll cover everything from installation to advanced features, and you'll get hands-on experience through practical examples and real-world projectsWhat's included in the course ?Introduction to AirflowUnderstanding the purpose and benefits of using Apache Airflow.Pros and cons of adopting Airflow in your projects.Airflow ArchitectureA detailed look into the components that make up Airflow.Key terminology used in Airflow.Configuration and InstallationStep-by-step guide to installing Airflow.The role and configuration of the airflow.cfg file.Airflow Web UI ViewsLaunching and navigating the Airflow Web UI.DAG ViewGrid ViewGraph ViewCalendar ViewTask Duration ViewCode ViewVariable ViewGantt ViewDAGs (Directed Acyclic Graphs)What is a DAG?Creating a DAG definition file.Different methods for DAG creation.Understanding DAG Run, default_arguments, and DAG arguments.Using parameters in DAGs and passing parameters through TriggerDagRunOperator.Scheduling concepts including depends_on_past, wait_for_downstream, catchup, and backfill.Airflow CLI and CheatsheetUtilizing the Airflow CLI for various operations.Handy cheatsheet for quick reference.Tasks in AirflowWhat are tasks and task instances?The lifecycle of a task. Operators in AirflowDetailed exploration of operators including BashOperator, PostgresOperator, PythonOperator, SqliteOperator, and EmailOperator.SensorsUsing sensors like FileSensor, SQLSensor, TimeDeltaSensor, and TimeSensor.BranchingImplementing branching logic with BranchSQLOperator, BranchPythonOperator, BranchDayOfWeekOperator, BranchDateTimeOperator, and ShortCircuitOperator.DAG Dependencies and TaskGroupsManaging DAG dependencies and using TaskGroups.Using TriggerDagRunOperator and ExternalTaskSensor.HooksUnderstanding and using hooks such as PostgresHook and SHook.Resource ManagementManaging resources with pools and task priorities.Executors in AirflowDifferent types of executors: SequentialExecutor and LocalExecutor.Transitioning from SequentialExecutor to LocalExecutor.Airflow Metadata Database and RolesUnderstanding the Airflow metadata database.Managing roles: creating users with different roles, including admin, public, user, and operator roles.Creating custom roles and modifying existing ones.SLA (Service Level Agreement)Setting and managing task-level and DAG-level SLAs.Handling SLA misses.Advanced ConceptsUsing XComs for inter-task communication.Retrieving context parameters and using callback functions.Dealing with zombie tasks, SIGTERM, and SIGKILL errors.I believe that mastering workflow automation with Airflow can open up incredible opportunities in the field of data engineering. I've seen firsthand how it can transform the way we handle data, and I can't wait to see what you'll achieve with these skills.So, whether you're looking to advance your career, work on more efficient data pipelines, or just curious about Airflow, you're in the right place. Let's dive in and start creating some amazing workflows together. Are you ready? Let's get started!I wish you a great success!

Skills

Reviews