Windowing and Join Operations on Streaming Data with Apache Spark on Databricks

Apache Spark Structured Streaming treats real-time data like a table that is constantly being added to. This stream processing model shifts the burden of stream processing from the user to Spark, making it easy and intuitive to process streaming Data with Spark. Apache Spark supports a variety of windowing operations and join operations on streaming datasets using processing time and event times. This course on Windowing and Join Operations with Streaming Data on Databricks will explain the differences between stateless operations that only operate on one streaming entity and stateful operations which operate on multiple entities accumulated within a stream. Next, you'll explore the various types of windows that Apache Spark supports, including sliding windows, global windows, and tumbling windows. Finally, you'll learn the differences between processing time, event time, and ingestion time. You will also see how windowing operations can be performed using both processing and event times. You will also connect to an HDInsight Kafka Cluster to read records from your input stream. Watermarking will be used to handle late-arriving data. You will also learn how to use watermarks in order to limit the state of Apache Spark storage. Finally, you will explore Spark's support for stream-stream and static-stream joins. This course will show you how to connect to Azure Event Hubs in order to read records.