Pyspark

Credits : Following notes are learnings from Spark: The Definitive Guide by Bill Chambers and Matei Zaharia

• Overview of Big Data and Spark
  1. What is Apache Spark ?
  2. A Gentle Introduction to Spark
  3. A Tour of Spark’s Toolset
• Structured APIs - DataFrames, SQL And Datasets
  1. Structured API Overview
  2. Basic Structured Operations
  3. Working With Different Types of Data
  4. Aggregations
  5. Joins
  6. Data Sources
  7. Spark SQL
  8. Datasets

Till this its probably enough to learn 80% of pyspark and people can head dive into the pyspark easily.

• Low-Level APIs
  1. Resilient Distributed Datasets
  2. Advanced RDDs
  3. Distributed Shared Variables

This shoud be performed practically with a running spark cluster and probably book is best place to read this.

• Production Applications
  1. How Spark Runs on a Cluster
  2. Developing Spark Applications
  3. Deploying Spark
  4. Monitoring and Debugging
  5. Performance Tuning

You should take a look at Apache Flink as well for Streaming Solutions.

• Streaming
  1. Stream Processing Fundamentals
  2. Structured Streaming Basics
  3. Event Time And Stateful Processing
  4. Structured Streaming in Production

For Running Machine Learning Workloads on Spark. There is a dedicate guide for ML on the site.

• Advanced Analytics and Machine Learning
  1. Advanced. Analytics and Machine Learning Overview
  2. Preprocessing and Feature Engineering
  3. Classification
  4. Regression
  5. Recommendation
  6. Unsupervised Learning
  7. Graph Analytics
  8. Deep Learning
• Ecosystem
  1. Language Specifics: Python (pyspark) and R (SparkR and Sparklyr)
  2. Ecosystem and Community

Books

• Learning Spark 2.0