[FREE] A file format claims to be 100x faster than Parquet
To celebrate reaching 20,000 subscribers, I’m offering 50% off FOREVER on the annual plan.
Once you claim the offer, the 50% discount will automatically apply to all future renewals.
You have only 1 day left to claim this offer.
I couldn’t have achieved this without you guys! Thank you soooo much!
Intro
Data engineering workloads have evolved.
From historical tabular data for humans to vector embeddings, images, or documents for AI.
From sequential scans, we now also need random access- the one that used to only belong to the “transactional“ world- now is required in our field.
When an AI model needs to perform a semantic search (e.g., find a document or an image), the system performs a random-access read to retrieve a small number of rows.
Parquet wasn't designed for this kind of operation: it works best on sequential data scans, but random access is another story.
—
To fill this gap, new file formats have been introduced.
One of them is Lance, which claims to be 100x faster than Parquet for random access.
In this article, I will discuss Lance's main technical decisions that enable robust random-access performance, as well as other improvements compared to Parquet.
Note: This article is not sponsored by or affiliated with any organization.
Parquet
Revisit
This article will cover some low-level Parquet concepts. So I think it's helpful to revisit Parquet internals here so you don’t have to switch between tabs to search for those concepts.
—
Parquet organizes data in the Partition Attributes Across (PAX) layout, commonly called the hybrid format. It first divides data into “row groups,” each containing a subset of rows (horizontal partition).
Within each group, it stores values from a column together. Each column is stored in chunks. Each chunk consists of pages, the unit for encoding/decoding and compression/decompression.
Keep in mind this detail: page is the smallest unit you can read in Parquet.
Each page has a header that includes information such as value, definition, and repetition encoding (the latter two are used to handle nested data). The application uses the header to decode and read the actual data.
Every Parquet file has a FileMetadata object in the footer. It has information such as the number of rows, the data schema, and row group metadata. FileMetadata is the entry for any Parquet read operations.
The problem with random access of data
In Parquet, a page is the smallest unit you can read. For data scans, reading and decoding a whole page at a time is ideal. But when you need only a few rows in a page, that is a different story.
To celebrate reaching 20,000 subscribers, I’m offering 50% off FOREVER on the annual plan.
Once you claim the offer, the 50% discount will automatically apply to all future renewals.
You have only 1 day left to claim this offer.
I couldn’t have achieved this without you guys! Thank you soooo much!
—
To read a row in a page, the engine must fetch the page from disk, then decompress (if it’s compressed) and decode the whole page to reach the target row. Spending time on the whole page when you only need a single row will increase random-access latency.
The larger the page size, the higher the latency.
The problem with random access of column schema
Parquet FileMetadata is encoded using Apache Thrift or Protocol Buffers. However, Thrift and Protocol Buffers don't support random access; the reader must deserialize the entire FileMetadata object to get the schema, even if it needs only a single column.
AI workloads such as training machine learning models require many descriptive attributes, or features. Each is stored as a column. It’s no surprise that a table used in machine learning has 10,000 columns.
The more columns, the higher the latency to retrieve a single column schema from FileMetadata.
The row-group constraint
The row group in Parquet is tied directly to the physical I/O unit. The write engine cannot flush a partial column chunk, so it must buffer the entire row group in memory before writing. This is a common cause of out-of-memory (OOM) failures, especially with large column values like AI vector embeddings.
Even without OOM, you may still run into other problems when storing a very large column. Imagine one column with 4KiB values (e.g., 4,096 bytes); a 512 MB Row Group would contain only 125,000 rows (512,000,000/4,096).
Because all columns in the row group must have the same number of rows, all other columns must also be capped at 125,000 rows, although an Int32 column could have 128 million rows (512,000,000/8).
Because each row group will be small, the Parquet file must contain many row groups. This could be inefficient:
More metadata to manage
More, smaller reads: within a row group, columns are stored one after another. Reading one column across many row groups means many separate, non-adjacent reads. Reading data from disk is best when it's as sequential as possible, but more row groups don’t encourage that. More reads also mean more GET object storage requests.
Ineffective compression: large row groups provide a larger sample size of data for each column, allowing compression and encoding algorithms to be more effective.
Lance
Lance is a modern, open-source columnar data format designed for machine learning and multimodal AI, launched around 2022 by LanceDB (a YC W22 startup) to address Apache Parquet’s performance limitations in AI workloads.
In the Lance spec, there are roughly 4 sections:
The data pages: each page holds some or all of one column’s data. Every page in the file is located by an absolute offset.
The column metadata: it provides information about the data pages and is located near the end of the file. It includes a set of protobuf messages; each column has its own message, including information such as pointers to the data pages or the encoding scheme.
A column metadata offset tableprovides the position and size of each column’s metadata message. It’s stored right next to the column metadata.
The offset array and the footer live at the end of the file. The array contains the offset to the column metadata, and the footer contains the offset to the array.
Page Layout
Lance handles random access differently depending on value size.
For small values like integers and short strings, Lance uses mini-block encoding. The data page is organized into chunks; each is around 4–8 KiB, with at most 4,096 values.
Like Parquet’s page, the engine still reads and decodes a chunk to get one value. However, because it's much smaller, the decoding is faster.
For large values (vector embeddings, images, large text, 256 bytes per value as threshold), Lance uses “full-zip” encoding, which interleaves each value’s metadata directly with its data and stitches these metadata-data buffers right next to each other.
The compression used is restricted to “transparent” encodings, meaning you can decompress a single value without decoding neighboring values. So if you need embedding #48, you jump directly to it and decompress just that one value.
—
With transparent encoding, you can decode each value on its own, as long as you know where it is. Example:
Dictionary:
["red","blue","red","green"]becomes indices[0, 1, 0, 2]. Index #3 is2, which maps to"green".
With opaque encoding, decoding one value requires decoding other values too. Example:
Delta:
[100, 102, 105, 106]stored as base100plus differences[2, 3, 1]. Value #3 = 100 + 2 + 3 + 1, so you need every difference before it.
Opaque encodings could save more space, but the values depend on each other, so you can’t decode just one.
Mini-block can use opaque encodings, while full-zip is strictly limited to transparentencodings.
Abandon Row Group
The Lance file structure abandons the row group concept completely. The column’s data is stored across multiple disk pages. Each page defaults to 8MB.
Different columns can have different numbers of pages. Because a page is referenced by an absolute offset, Lance doesn’t require the pages in the same column to be contiguous.
This gives a freedom Parquet can’t offer: the writer can use separate buffers for each column instead of buffering the whole row-group.
Each column's page count simply follows its data size. Take a file with 1 million rows. A boolean column is about 128 KiB, so it fits in a single page. A 3 KiB embedding column is about 3 GiB, which becomes roughly 384 pages of 8 MiB each.
No longer does every column have to share the same row group size.
—
The same freedom applies on the read side.
In Parquet, the row group is usually the unit of parallelism. Ten threads usually means ten row groups. If a file has only one big row group, there’s not much to split.
Lance lets you split the file at any row boundary.
Say two threads share a 2 million-row file. One reads rows 0 to 999,999; the other reads from 1,000,000. Row 1,000,000 might sit in the middle of a page, but that’s not the problem.
Each page records the row it starts at, and its internal layout (mini-block chunks, or full-zip byte offsets) maps any range of rows to a range of bytes.
So each thread reads only its own page slice by locating the required byte.
Schema Random-access
Because each column is described by a separate protobuf message, the reader can retrieve the metadata for the required columns without decoding other column metadata as in Parquet.
Outro
In this article, I revisit Parquet's current limitations and then discuss Lance. From its specification to its page layout, why it abandons row groups, and how it can now access column metadata in O(1).
Thank you for reading this far.
See you in my next article.
Reference
[1] Weston Pace, Chang She, Lei Xu, Will Jones, Albert Lockett, Jun Wang, Raunak Shah, Lance: Efficient Random Access in Columnar Storage through Adaptive Structural Encodings, 2025
[2] Lance v2: A columnar container format for modern data, 2024
[2] Julien Le Dem, Column Storage for the AI Era, 2025
© 2026 Vu Trinh
548 Market Street PMB 72296, San Francisco, CA 94104