Sameer Farooqui from Databricks delivers a hands-on tutorial using Spark SQL and DataFrames to retrieve insights and visualizations from datasets published by the City of San Francisco. The labs are targeted for an audience with some general programming or SQL query experience, but little to no experience with Spark. Sameer will begin with some brief theory and lecture on Spark, before diving into several demos performing visualizations and analysis on calls made to the San Francsico Fire Department on July 4th.
----Topics-----
00:00:06 - Workshop Intro & Environment Setup
00:13:06 - Brief Intro to Spark
00:17:32 - Analysis Overview: SF Fire Department Calls for Service
00:23:22 - Analysis with PySpark DataFrames API
00:29:32 - Doing Date/Time Analysis
00:47:53 - Memory, Caching and Writing to Parquet
01:00:40 - SQL Queries
01:21:11 - Convert a Spark DataFrame to a Pandas DataFrame
-----Q & A-----
01:24:43 - Spark DataFrames vs. SQL: Pros and Cons?
01:26:57 - Workflow for Chaining Databricks notebooks into Pipeline?
01:30:27 - Is Spark 2.0 ready to use in production?
Follow Along:
+ Databricks Community Edition: https://databricks.com/try
+ Labs: https://bit.ly/sfopenlabs
+ Learning Material: https://bit.ly/sfopenreadalong
Recent Comments