Standard vs Dedicated Access Mode in Azure Databricks: What I Learned?

I was following an Apache Spark tutorial in Azure Databricks.

Databricks automatically creates and provides a SparkSession variable (spark) when you attach a notebook to compute. You do not need to build one manually. You can execute PySpark code right away.

I typed spark into a notebook cell and ran it.

The output looked completely different from the tutorial video.

The tutorial showed full SparkSession details, including SparkContext and Spark UI links. My notebook only printed a plain object reference:

<pyspark.sql.connect.session.SparkSession object at 0x7a4bb7307e60>

SparkSession output showing pyspark.sql.connect.session.SparkSession object in Databricks notebook

Why Was This Happening?

I checked my compute settings to see what was going on.

My all-purpose cluster was using Standard access mode.

In newer Databricks runtimes, Standard mode defaults to Spark Connect.

Spark Connect uses a thin client-server architecture. It communicates with the Spark cluster over gRPC.

Spark Classic runs directly against the driver JVM. It gives you direct access to lower-level Spark APIs.

Why Did I Switch to Dedicated?

Standard mode works fine for regular data engineering tasks. It handles DataFrames, SQL queries, transformations, and actions without problems.

I wanted to understand how Spark actually works under the hood. I did not want to stop at high-level PySpark code.

Here is what I wanted to do:

  • Explore SparkContext: Understand the main entry point to Spark’s lower-level internals.
  • Practice RDD operations: Run distributed collection methods like parallelize() and map().
  • Explore Spark UI: Inspect jobs, stages, tasks, and query execution details directly.
  • Access debugging tools: Check driver and executor logs to see how Spark executes tasks.
  • Follow classic Spark examples: Run code demonstrations that rely on sc and RDD APIs without errors.

Standard mode disables direct RDD APIs and SparkContext. Dedicated mode supports both.

How Did I Switch to Dedicated Access Mode?

I updated the cluster access settings in the Databricks UI:

  1. Go to Compute.
  2. Select your cluster.
  3. Click Edit.
  4. Scroll down and expand Advanced.
  5. Under Access mode, switch from Auto (Standard) to Manual (Dedicated).
  6. Click Confirm and restart to apply the configuration.

Databricks cluster configuration settings showing access mode options

After the cluster restarted, I ran spark again:

spark

This time, the notebook printed the complete session summary:

SparkSession - hive
SparkContext
Spark UI

Version
    v4.1.0
Master
    local[*, 4]
AppName
    Databricks Shell

SparkSession output in Databricks notebook showing SparkContext, Spark UI, version 4.1.0, and master local

What Did I Learn?

My original setup was not broken. The difference came down to the cluster access mode.

Here are my main takeaways:

  • Standard is not broken, just decoupled: Standard mode uses Spark Connect. Dedicated mode gives you a classic Spark driver session.
  • Dedicated is not automatically better: Standard mode is recommended for production workloads and shared team environments. It provides better security and workload isolation.
  • Dedicated is great for learning: It gives full access to SparkContext, RDD APIs, and classic Spark internals.
  • Understanding local[*, 4]: The master string local[*, 4] means my Spark session is running in single-node mode. It uses 4 local CPU cores on one virtual machine rather than worker nodes across a distributed cluster.

For now, I will keep using Dedicated mode while exploring Spark architecture and RDD internals. Once I finish these lower-level tutorials and build real data pipelines, I will switch back to Standard mode.

Want my posts to show up more often on Google?

One click and Google will surface this site in your Top Stories.

Add as preferred source
Suraj Katwal
Written by

Suraj Katwal

Digital marketer, data enthusiast, and developer based in Melbourne, originally from Nepal. I build web platforms, Flutter apps, and digital marketing campaigns. On the data side, I work with SQL and Python while growing into dbt and data pipeline tools.