Handling Streaming Data with Azure Databricks Using Spark Structured Streaming

Modern data pipelines often contain streaming data that must be processed in real time. You would need to manage multiple streams and data sets to produce consistent results in a real-world scenario. This course, Handling Streaming Data With Azure Databricks Using Spark Structured Streaming teaches you how to use Spark Structured Streaming to create streaming pipelines. It is run on Microsoft Azure. You will first see a brief overview of Spark Structured Streaming's processing model. Next, you will learn how to implement the scenario and set up the environment. Next, you'll learn how to set up sources and sinks and build each stage of the streaming pipeline. This involves extracting data from different sources, transforming it and loading it into multiple sinks such as Azure SQL, Azure Event Hubs and Azure Data Lake. Additionally, you will learn how to aggregate data with Windows and the various timestamps that are associated with each event. Next, you'll learn how to combine streams with historical or static data. You will also learn how to combine multiple streams together. Finally, you'll learn how to create a production-ready pipeline, schedule it in Databricks and manage them with Databricks CLI. After completing this course, you will be able to create complex streaming pipelines on Azure Databricks to solve various business problems.