SQL Scripting Now Generally Available

Databricks Serverless version 18.0 marks the general availability of SQL scripting, enabling more complex procedural SQL workflows on the serverless platform. This feature allows developers to write multi-statement SQL scripts with procedural logic directly in their notebooks and jobs.

Enhanced SQL Features and Functions

The release expands SQL capabilities with several new features:

• Parameter markers everywhere: Named (:param) and unnamed (?) parameter markers can now be used virtually anywhere a literal value can appear, including DDL statements and column type definitions
• IDENTIFIER clause expansion: The IDENTIFIER clause for casting strings to SQL object names now works in nearly all identifier contexts, including column aliases and definitions
• Literal string coalescing: Sequential string literals like 'Hello' ' World' are automatically coalesced into 'Hello World' anywhere strings are permitted
• SQL window functions in metric views: Window functions can now be used for running totals, rankings, and other window-based calculations in metric views
• New aggregate functions: bitmap_and_agg and a new library of Theta sketch functions for approximate distinct count and set operations

Streaming and Performance Improvements

Stateless streaming queries gain significant enhancements:

• Dynamic shuffle partition adjustment: Change the number of shuffle partitions without restarting your query
• Adaptive Query Execution and auto-optimized shuffle: AQE and AOS are now supported in stateless streaming queries
• Improved UDF performance: Unity Catalog Python UDFs with the same owner now share isolation environments by default, reducing memory usage. Use STRICT ISOLATION clause when full isolation is required

Additional Changes

The Redshift JDBC driver has been upgraded to version 2.1.0.28, providing compatibility improvements for Redshift connections.