Serverless Environment Version 5 Release

Databricks has released serverless environment version 5, making it available immediately for all users running serverless compute across notebooks and jobs. This update is available in both CPU and GPU environment variants, with automatic rollout ensuring all users receive the upgrade over a short period.

Key New Features

SQL Enhancements:

• LIMIT ALL support for recursive CTEs: You can now use the LIMIT ALL clause with recursive common table expressions to explicitly specify that no row limit should be applied to query results.
• EXECUTE IMMEDIATE with constant expressions: The EXECUTE IMMEDIATE statement now supports constant expressions in the query string, enabling more flexible dynamic SQL execution.
• Improved error handling: Attempting to append to existing files in Unity Catalog volumes now returns more descriptive error messages.

Geospatial Functions:

• st_dump function: Decompose geometry objects into their constituent parts, returning a set of simpler geometries.
• Polygon interior ring functions: Two new functions (st_numinteriorrings and st_interiorringn) for working with polygon interior rings.

Configuration and Behavior Improvements

Users can now configure the spark.sql.files.maxPartitionBytes Spark configuration parameter on serverless compute to control partition byte packing when reading files. The release also includes behavior changes for improved metadata handling in materialized views and streaming tables, along with corrections for null struct handling in Parquet and schema operations.

Automatic Deployment

As with all serverless compute releases, users automatically receive the latest version without requiring code changes. The stable client API based on Spark Connect ensures application compatibility across upgrades.