Advanced Data Engineering

Duke University via Coursera

Go to Course: https://www.coursera.org/learn/advanced-data-engineering

Introduction

Create and manage data pipelines and their lifecycle

Connect and work with message queues to manage data processing

Use vector, graph, and key/value databases for data storage at scale

Syllabus

Queues and Databases-RabbitMQ and MySQL

In this module, you will learn about databases and queues. You will find out the purpose and components of RabbitMQ including its use of queues and integration with Celery. Through hands-on exercises, they will gain experience connecting Celery to RabbitMQ within a Flask application and implementing task patterns like fire and forget and result retrieval. The course also covers core MySQL skills like interacting via the command line interface, manipulating databases, and integrating with Python web apps. By the end, students will have a foundational understanding of RabbitMQ, Celery, and MySQL that allows them to start building modern, asynchronous applications backed by a database.

Optimizing Workflow Management at Scale with Apache Airflow

Achieving Scalability with Vector, Graph, and Key/Value Databases

In this module, we explore vector and graph databases, powerful tools for managing and extracting insights from large, complex datasets. As data volumes continue to grow, scalability is crucial. We'll learn how vector and graph databases can efficiently store data while maintaining relationships, enabling more advanced analytics. Through real-world examples, you'll see how these databases unlock scalability for machine learning, fraud detection, social networks, and more.

Real-world Advanced Data Engineering Projects

In this final module, you will work on advanced real-world data engineering projects, applying everything you've learned. You'll encounter complex data challenges and devise solutions using the latest tools and techniques. This is an opportunity to bring together data engineering concepts covered throughout the course and implement them holistically to deliver impactful outcomes.