Perform data science with Azure Databricks

Go to Course: https://www.coursera.org/learn/perform-data-science-with-azure-databricks

Introduction

### Course Review: Perform Data Science with Azure Databricks In the ever-evolving world of data science, gaining proficiency in cutting-edge tools and platforms is essential. One such invaluable resource is the course titled **"Perform Data Science with Azure Databricks,"** available on Coursera. This comprehensive course is the fourth installment in a five-course program that prepares learners for the **DP-100: Designing and Implementing a Data Science Solution on Azure** certification exam. Here’s a detailed review of the course and a strong recommendation for both aspiring and experienced data scientists. #### Course Overview Designed for individuals eager to leverage cloud computing for data science, this course teaches participants how to utilize **Apache Spark** and powerful clusters operating on the **Azure Databricks** platform. The platform itself allows for seamless data processing, making it a fantastic environment for executing complex data science workloads. Whether you're aiming to sharpen your skills or prepare for a professional certification, this course equips learners with the tools and knowledge to succeed. #### Syllabus Breakdown The course is constructed into clear, digestible modules: 1. **Introduction to Azure Databricks:** In this opening module, learners explore the comprehensive capabilities of Azure Databricks. The module covers the architecture of Databricks Spark Clusters and the types of tasks best suited for Apache Spark, providing a strong conceptual foundation for subsequent lessons. 2. **Working with Data in Azure Databricks:** This module focuses on fundamental data manipulation techniques in Azure Databricks. Participants get hands-on experience with data handling functions such as reading, writing, and querying large datasets from multiple sources. The practical application of the **DataFrame Column Class** for transformations like sorting and filtering enriches the learning experience. 3. **Processing Data in Azure Databricks:** Delving deeper, learners gain insight into handling custom functions through User-Defined Functions (UDFs) and explore Delta Lake for data operations. This module expertly blends theoretical knowledge with practical skills, essential for real-world data science applications. 4. **Get Started with Databricks and Machine Learning:** This segment introduces learners to PySpark's machine learning package, enabling them to perform exploratory data analysis, model training, and evaluation—all crucial components of the machine learning lifecycle. 5. **Manage Machine Learning Lifecycles and Fine-Tune Models:** Participants learn to navigate MLflow for tracking experiments and utilize Spark's machine learning library for hyperparameter tuning. This emphasis on lifecycle management and optimization prepares learners comprehensively for practical scenarios. 6. **Train a Distributed Neural Network and Serve Models with Azure Machine Learning:** In the final module, participants learn about distributed deep learning using Uber’s Horovod framework and the Petastorm library. This involves using machine learning services in Azure, providing students with relevant skills for deploying models in production environments. #### Why You Should Take This Course **1. Strong Practical Focus:** Each module is crafted to include hands-on experiences that build crucial skills required in the field of data science. The integration of theory and practice ensures that learners can apply their knowledge in real-world scenarios. **2. Accessible to All Levels:** Whether you're a novice looking to start your journey or an experienced data scientist aiming to specialize in Azure, this course offers valuable insights and practical applications tailored to various skill levels. **3. Preparation for Certification:** As part of a broader program to achieve certification, this course not only imparts essential skills but strategically positions learners for success in passing the DP-100 exam. **4. Expert Instructors:** The course is delivered by industry professionals who have vast experience working with Azure Databricks and machine learning technologies, providing learners with current and relevant knowledge. **5. Career Advancement:** Data science is a field in high demand, particularly with cloud computing capabilities. Gaining expertise in Azure Databricks positions you competitively in the job market. #### Conclusion In a nutshell, **"Perform Data Science with Azure Databricks"** is a highly recommended course for anyone interested in mastering data science in a cloud environment. With its blend of theoretical knowledge and practical application, this course stands out as an essential stepping stone for those eager to work with machine learning solutions at cloud scale. If you're serious about advancing your data science career, enrolling in this course will be a decision well worth making.

Syllabus

Introduction to Azure Databricks

In this module, you will discover the capabilities of Azure Databricks and the Apache Spark notebook for processing huge files. You will come to understand the Azure Databricks platform and identify the types of tasks well-suited for Apache Spark. You will also be introduced to the architecture of an Azure Databricks Spark Cluster and Spark Jobs.

Azure Databricks supports day-to-day data-handling functions, such as reads, writes, and queries. In this module, you will work with large amounts of data from multiple sources in different raw formats. You will also learn to use the DataFrame Column Class Azure Databricks to apply column-level transformations, such as sorts, filters and aggregations. You will also use advanced DataFrame functions operations to manipulate data, apply aggregates, and perform date and time operations in Azure Databricks.

Azure Databricks supports a range of built in SQL functions, however, sometimes you have to write custom function, known as User-Defined Function (UDF). In this module, you will learn how to register and invoke UDFs. You will also learn how to use Delta Lake to create, append, and upsert data to Apache Spark tables, taking advantage of built-in reliability and optimizations.

In this module, you will learn how to use PySpark’s machine learning package to build key components of the machine learning workflows that include exploratory data analysis, model training, and model evaluation. You will also learn how to build pipelines for common data featurization tasks.

In this module, you will learn how to use MLflow to track machine learning experiments and how to use modules from the Spark’s machine learning library for hyperparameter tuning and model selection.

In this module, you will learn how to use the Uber’s Horovod framework along with the Petastorm library to run distributed, deep learning training jobs on Spark using training datasets in the Apache Parquet format. You will also learn how to use MLflow and Azure Machine Learning service register, package, and deploy a trained model to both Azure Container Instance, and Azure Kubernetes Service as a scoring web service.