PySpark Fundamentals

 

Data Architects, Data Engineers, Data Scientists
 

• Attendants should have knowledge in:
  • Python
  • Spark fundamentals

Part I. Spark Concepts

• Unit 1. Spark Fundamentals

Spark and distributed computing. Basic Architecture. Spark Core. RDDs, DataFrames and DataSets. Spark SQL, Spark Streaming, Machine Learning and GraphFrames Part II. Spark Resilient Distributed DataSets

• Unit 2. Major Spark APIs

Low level RDD API. DataFrame API. Dataset API. SparkSQL API, Spark SQL and optimization.

• Unit 3. Advanced RDDs

Working with Key/Value Pairs, Lookups, Aggregations, Partitioning, Serialization Lab Included

• Unit 4. Advanced Spark Concepts

Join Operations, Stateful Transformations, Accumulators, Fault Tolerance Lab included Part III. Spark DataFrames

• Unit 5. Introduction to Spark Structured Programming DataFrames in PySpark. Managing columns. DataFrames transformation and filters

Lab Included

• Unit 6. Spark Structured Operations

Using Spark SQL. SQL operations in DataFrames. Managing tables. UDF implementation.

• Unit 7. Spark Aggregations

Grouping operations. Aggregations. Window Operations. Combining DataFrames.

• Unit 8. Advanced DataFrames

Broadcast and accumulator variables. Partitioning. Catalyst Optimization Part III. Spark Configuration

• Unit 9. Spark Configuration, monitoring and tuning

Obtain and modify Spark properties. Spark execution on YARN. Debugging Spark. Monitor Spark using web UIs, and external Instrumentation. Performance tuning considerations. Logging and logging adjustments. Troubleshooting Spark applications. Labs included