Data Engineering Capstone Project

Go to Course: https://www.coursera.org/learn/data-enginering-capstone-project

Introduction

### Course Review: Data Engineering Capstone Project on Coursera If you've been navigating the expansive realm of data engineering, you're likely aware of the pivotal role it plays in empowering organizations to leverage their data for strategic decisions. The **Data Engineering Capstone Project** offered by Coursera, in collaboration with IBM, is the perfect culmination of your journey through the IBM Data Engineering Professional Certificate. This course provides an immersive experience that allows you to apply your acquired knowledge and skills to a real-world project, simulating the responsibilities of a Junior Data Engineer. #### Overview This capstone project serves as a fantastic opportunity to demonstrate your data engineering capabilities through practical application. You will step into the shoes of a Junior Data Engineer who has just joined a company faced with the challenge of architecting and implementing a data analytics platform. The course content is well-structured, focusing on critical elements of data engineering, including data platform architecture, data warehousing, ETL processes, and analytics using modern tools and frameworks. #### Syllabus Breakdown 1. **Data Platform Architecture and OLTP Database** - You start by designing a data platform utilizing MySQL as an OLTP database. This module will enhance your understanding of transaction processing and data storage before diving deeper into more complex systems. 2. **Querying Data in NoSQL Databases** - The course then transitions to NoSQL databases, specifically MongoDB, emphasizing its use in storing e-commerce catalog data. This segment will enable you to understand the contrasts between SQL and NoSQL paradigms. 3. **Build a Data Warehouse** - This module focuses on designing and implementing a data warehouse. You will gain insights into data integration and reporting, key skills necessary for any data engineer aiming to bridge raw data and actionable insights. 4. **Data Analytics** - As an engineer in an e-commerce setting, you will create a reporting dashboard that reflects essential business metrics. This part ties together the need for data visualization and decision-making based on data-driven insights. 5. **ETL & Data Pipelines** - You will engage with ETL processes through hands-on labs, transferring data across different storage solutions while learning how to effectively manipulate data for analysis. This is critical for any data engineer in today's data landscape. 6. **Big Data Analytics with Spark** - Finally, utilizing Apache Spark, you will analyze data from webserver logs and predict sales forecasts using a pretrained model. This module will equip you with the skills to handle large datasets and apply machine learning techniques. 7. **Final Submission and Peer Review** - The course wraps up with a final project where you'll compile your lab work for peer review. This collaborative approach not only solidifies your learning but also encourages engagement with the community. #### Recommendation The **Data Engineering Capstone Project** is highly recommended for anyone looking to solidify their skills in data engineering. Whether you are a newcomer in the field or looking to pivot your career, this course provides an extensive foundation combined with hands-on experience. The structured modules ensure that you not only understand theoretical concepts but also know how to apply them effectively in a practical setting. Moreover, the opportunity to collaborate and receive feedback from peers adds an extra layer of learning that is not often found in many online courses. The peer review process holds you accountable and can inspire you to produce high-quality work. ### Conclusion Choosing to enroll in the Data Engineering Capstone Project can be a game-changer in your professional journey. By the end of the course, you will have a well-rounded portfolio project to showcase to potential employers, highlighting your expertise in data platforms, cloud programming, ETL processes, and big data analytics. If you aim to thrive in the rapidly expanding field of data engineering, this course is a decisive step in your career development. Enroll today and take the next step in your data engineering journey!

Syllabus

Data Platform Architecture and OLTP Database

In this module, you will design a data platform that uses MySQL as an OLTP database. You will be using MySQL to store the OLTP data.

In this module, you will design a data platform that uses MongoDB as a NoSQL database. You will use MongoDB to store the e-commerce catalog data.

In this module you will design and implement a data warehouse and you will then generate reports from the data in the data warehouse.

In this module, you will assume the role of a data engineer at an e-commerce company. Your company has finished setting up a data warehouse. Now you are assigned the responsibility to design a reporting dashboard that reflects the key metrics of the business.

In this module, you will use the given python script to perform various ETL operations that move data from RDBMS to NoSQL, NoSQL to RDBMS, and from RDBMS, NoSQL to the data warehouse. You will write a pipeline that analyzes the web server log file, extracts the required lines and fields, transforms and loads data.

In this module, you will use the data from a webserver to analyse search terms. You will then load a pretrained sales forecasting model and predict the sales forecast for a future year.

In this final module you will complete your submission of screenshots from the hands-on labs for your peers to review. Once you have completed your submission you will then review the submission of one of your peers and grade their submission.