[OCTOBER NEWSLETTER] Less work per query. More in Release 0.17.0
Email at a Glance
Release 0.17.0 at a Glance —Open table efficiency, 56% smaller indexes, and more flexible JSON updates
- Stack Highlight — Optimizing Data Latency and Query Latency in Open Table Formats
Latest Resources — Query performance, agent workloads, telemetry deletes, and Kafka-to-lakehouse architecture
Up Next — November’s AI + analytics data platform roundtable and dinner at AWS re:Invent
Index builds in seconds, not minutes: meet StarTree Cloud 0.17.0
StarTree Cloud 0.17.0 makes querying your data lake leaner and faster than ever. Open Tables now use a fraction of the memory and build indexes in seconds instead of minutes, index storage drops by more than half, aggregations run up to 3.2× faster, and AI functions arrive right inside your SQL. If you run real-time analytics on lake data, this release is for you.
New Features
Leaner, faster Open Tables: Per-column heap for each loaded segment drops from about 4.3 KB to 0.9 KB, roughly 79% less. Star-tree, inverted and range index builds on Parquet-backed data are dramatically faster too: in internal benchmarks, a 300,000-row build went from about 4 minutes to about 3 seconds. And new opt-in Iceberg partition pruning skips segments your query doesn't need. Learn more
Significant storage efficiency: New clusters now write raw forward indexes in the V6 format with Zstandard compression by default. In internal tests, they came out about 56% smaller than LZ4-compressed V4 indexes, and V6 cut query CPU by 4–8%. This is the default for all new clusters going ahead. On an existing cluster? Contact StarTree Support to switch to V6. Learn more
Up to 3.2× faster aggregations: A new opt-in, index-based GROUP BY answers supported aggregations on low-cardinality columns straight from inverted indexes, with 1.8–3.2× speedups in internal benchmarks.
Smarter upserts with JSON merging: The new MERGE_JSON partial-upsert merger combines JSON objects and MAP values per column instead of replacing them, so you update only the fields that changed. Learn more
Row-level security that scales: Define one row-level access policy and apply it across every tenant or user, with filter values supplied at query time. No more maintaining a separate policy for each customer. Learn more
AI in your SQL (preview): New AI_FILTER, AI_CLASSIFY and AI_GENERATE functions bring AI-powered filtering, classification and generation right into your queries. Reach out to your account team to try them.
Fresh from Apache Pinot: UUID data types, flexible OPEN_STRUCT columns, and GROUPING SETS, ROLLUP and CUBE in the multi-stage engine. Learn more
Platform Improvements
Open Table sync now stops as soon as a segment goes into an error state, flags stalls after 5 minutes without progress, and reports unhealthy syncs and consecutive failures through new controller metrics, so problems surface before they turn into stale data. Tiered storage supports warm restarts through a disk cache, the StarTree Execution Engine runs IN and NOT IN natively, and MongoDB CDC now reports ingestion lag. This release also brings fixes for Parquet reading, S3 block caching, Iceberg snapshot loading and RocksDB upsert preloading. Learn more
Final Words
0.17.0 is available now. It moves StarTree to JDK 25 (custom BYOC plugins need a rebuild), replaces upsert compaction tasks with SegmentRefreshTask, and changes Iceberg snapshot retention defaults, so reach out to your StarTree account team to plan your upgrade. AI functions are in preview and can be enabled on request. Full release notes here.
Top of Stack Highlight
Optimizing Data Latency and Query Latency in Open Table Formats
Follow the architecture from Confluent Tableflow into Iceberg and Delta tables, then through indexed, page-level reads. This month’s webinar replay examines data arrival and query execution separately so you can evaluate latency at each stage.
Latest Resources
Query Performance: How Much Work Does Each Query Actually Require?
Two queries can finish in 50 milliseconds while consuming very different amounts of I/O, CPU, and memory. These field notes trace selective query paths, index analysis, routing, and cache behavior. Use the walkthrough to decide what to measure alongside latency in your next POC.
Query Abundance: When Machines Ask the Questions
Agent workflows can turn a single request into repeated, parallel queries. Query Abundance explores what that changes about burst capacity, predictable latency, and the cost of answering the next question.
How to Delete a Single Telemetry Event Out of Billions with Apache Pinot and StarTree
In Apache Pinot, deleting a few rows can require rewriting whole segments. Learn how StarTree’s SQL-based purge selects records, why segment count drives the work, and what to check in a dry run.
Explore our Resources Page for the latest StarTree blogs and deep dives.
Upcoming Events, Meetups, & Webinars
Where: Virtual
When: 11/19, 2PM EST
Dinner @ AWS re:Invent
Pinot, Pasta, and Peers Dinner at AWS re:Invent
Where: Carmine's in Las Vegas
When: 12/1, 6:30PM PST
Join your local Meetup group to never miss an event.
What’s The StarTree Advantage
Built and managed by the creators of Pinot, StarTree Cloud gives organizations the easiest path to scale, lower costs, and future-proof their real-time analytics platform. We take away the operational burden, add enterprise-grade features, and deliver modern integrations you won’t find in open source Pinot.
Interested in learning more? Explore StarTree Cloud today.
STAY CONNECTED
StarTree, 331 Castro St, Suite 200, Mountain View, CA 94041, USA