Beginning Apache Spark 3

Take a journey toward discovering, learning, and using Apache Spark 3.0. In this book, you will gain expertise on the powerful and efficient distributed data processing engine inside of Apache Spark; its user-friendly, comprehensive, and flexible programming model for processing data in batch and streaming; and the scalable machine learning algorithms and practical utilities to build machine learning applications.

Beginning Apache Spark 3 begins by explaining different ways of interacting with Apache Spark, such as Spark Concepts and Architecture, and Spark Unified Stack. Next, it offers an overview of Spark SQL before moving on to its advanced features. It covers tips and techniques for dealing with performance issues, followed by an overview of the structured streaming processing engine. It concludes with a demonstration of how to develop machine learning applications using Spark MLlib and how to manage the machine learning development lifecycle. This book is packed with practical examples and code snippets to help you master concepts and features immediately after they are covered in each section.

After reading this book, you will have the knowledge required to build your own big data pipelines, applications, and machine learning applications.

WHAT YOU WILL LEARN

* Master the Spark unified data analytics engine and its various components
* Work in tandem to provide a scalable, fault tolerant and performant data processing engine
* Leverage the user-friendly and flexible programming model to perform simple to complex data analytics using dataframe and Spark SQL
* Develop machine learning applications using Spark MLlib
* Manage the machine learning development lifecycle using MLflow

WHO THIS BOOK IS FOR

Data scientists, data engineers and software developers.

HIEN LUU has extensive experience in designing and building big data applications and machine learning infrastructure. He is particularly passionate about the intersection between big data and machine learning. Hien enjoys working with open source software and has contributed to Apache Pig and Azkaban. Teaching is also one of his passions, and he serves as an instructor at the UCSC Silicon Valley Extension school teaching Apache Spark. He has given presentations at various conferences such as Data+AI Summit, MLOps World, QCon SF, QCon London, Hadoop Summit, and JavaOne.

Chapter 1: Introduction to Apache Spark

Chapter Goal: Provide an overview of Apache Spark

No of pages 15

Sub -Topics

1. Overview & history

2. Spark concepts & architecture3. Spark Unified Stack

4. Apache Spark applications

Chapter 2: Working with Apache Spark

Chapter Goal: Provide details about different ways of interacting with Apache Spark

No of pages: 35

Sub - Topics

1. Downloading and Installing Apache Spark

2. Exploring Apache Spark using Spark shells

3. Exploring Apache Spark using Databricks

4. Exploring Apache Spark source code

Chapter 3: Spark SQL - Foundation

Chapter Goal: Provide an overview to Spark SQL component

No of pages: 60

Sub - Topics

1. Overview & architecture

2. Introduction to DataFrames Structured APIs

3. Reading & writing data with Spark SQL data sources

4. Introduction to datasets

Chapter 4: Spark SQL - Advance

Chapter Goal: Go over the advanced features in Spark SQL

No of pages : 50

Sub - Topics:

1. Working with aggregations

2. Joining data

3. Working with analytics functions

4. Explore Spark SQL catalyst optimizer

Chapter 5: Optimizing Apache Spark Applications

Chapter Goal: Go over tips and techniques for dealing with performance issues

No of pages: 30

Sub - Topics:

1. Common performance issues

2. Speed up performance by leveraging in-memory computation

3. Understand the different support joins in Spark4. Leverage Spark UI to diagnose performance issue

Chapter 6: Structured Streaming - Foundation

Chapter Goal: Overview of Structured Streaming processing engine

No of pages: 50

Sub - Topics:

1. General streaming processing concepts

2. Structured Streaming programming model

3. Working with streaming data sources and sinks

4. Understanding output modes and triggers

Chapter 7: Structured Streaming - Advanced

Chapter Goal: Cover complex issues in streaming processing

No of pages: 40

Sub - Topics:

1. Streaming processing with event time

2. Stateful streaming processing

3. Handling duplicate data

4. Monitoring streaming processing applications

Chapter 8: Machine Learning with Apache Spark

Chapter Goal: How to developing Machine Learning applications using Spark MLlib

No of pages: 60

Sub - Topics:

1. Machine learning overview

2. Taking a tour of supported machine learning algorithms

3. Building machine learning pipelines

4. Machine learning tasks in action

5. Parameters tuning

Chapter 9: Machine Learning Application Development w/ MLflow

Chapter Goal: Using MLflow to manage the Machine Learning development lifecycle

No of pages: 25

Sub - Topics:

1. Overview of MLflow

2. Tracking machine learning development experiments

3. Managing & deploying machine learning models

4. Leveraging Spark for batch modeling predictions