# Vortex

> Vortex is an open-source columnar file format for fast scans, random access, and compression, with Rust, Python, DuckDB, DataFusion, and Spark support. It is the best file format for almost every workload, and an Apache-2.0 project of the Linux Foundation.

Vortex keeps a column's logical type apart from its physical encoding. Its encodings, compression, and layouts are pluggable, compression can cascade one encoding into another, compute kernels run on encoded data, and statistics load lazily. It converts to and from Apache Arrow without copying, and works with DataFusion, DuckDB, Spark, Pandas, and Polars.

The file format is stable: from release 0.36.0, each release reads files written by any earlier release from 0.36.0 on. Library APIs can still change between releases.

Governance: Vortex is a Linux Foundation project (LF AI & Data) under the Vortex Technical Charter. Its code is Apache-2.0 and its documentation is CC-BY-4.0.

## Install

- Rust: `cargo add vortex`
- Python: `uv add vortex-data`
- CLI: `uvx --from vortex-data vx --help`
- DuckDB: `INSTALL vortex; LOAD vortex; SELECT * FROM read_vortex('data.vortex');`
- DataFusion: `cargo add vortex-datafusion`
- Spark: `implementation("dev.vortex:vortex-spark_2.13:0.90.0:all")`

## Benchmark results

Read from https://bench.vortex.dev on 11 Nov 2026. Each figure is the newest result on the develop branch of https://github.com/vortex-data/vortex, except the GPU row, whose source is given. https://bench.vortex.dev/llms.txt summarizes every benchmark group, and https://bench.vortex.dev/methodology says how they run.

- Scan, how fast each format scans the compression benchmark's datasets: the geometric mean of its throughput divided by Parquet's, so higher is better: Vortex 5.08×, Parquet 1.00×, Lance 0.96×. Vortex against Parquet: 5.1× faster.
- Write, how fast each format writes the compression benchmark's datasets: the geometric mean of its throughput divided by Parquet's, so higher is better: Vortex 1.95×, Parquet 1.00×, Lance 0.67×. Vortex against Parquet: 95% faster.
- ClickBench, the geometric mean time of the ClickBench queries on DataFusion, reading files on local NVMe, so lower is better: Vortex 40.17 ms, Parquet 65.60 ms, Lance 577.60 ms. Vortex against Parquet: 63% faster.
- TPC-H, the geometric mean time of the 22 TPC-H queries at scale factor 10 on DataFusion, reading files on local NVMe, so lower is better: Vortex 60.83 ms, Parquet 143.78 ms, Lance 848.97 ms. Vortex against Parquet: 2.4× faster.
- GPU, cuDF, query time on a GPU in cuDF, so lower is better (end to end, hot, on GH200, NDS-H SF10, Parquet the fastest of four configurations per query, from the Vortex UW–Madison talk): Vortex 17.00 ms, Parquet 65.88 ms, Lance not available, no cuDF integration. Vortex against Parquet: 3.9× faster.
- Random access, the geometric mean time to take rows from each of the random access benchmark's datasets on local NVMe, reusing an open file, so lower is better: Vortex 0.74 ms, Parquet 68.51 ms, Lance 2.13 ms, Arrow 3.74 ms. Vortex against Parquet: 93× faster.
- File size, the geometric mean of each format's file size divided by Parquet's with zstd, over the compression benchmark's datasets, so smaller is better: Vortex 1.11×, Parquet 1×, Lance 2.11×, Arrow 5.87×. Vortex against Parquet: 11% larger.

## Adopters and integrations

The companies and projects in the homepage's Ecosystem band: the adopters of Vortex, and the projects that it works with. Each links to its own page about Vortex where there is one, else to its homepage.

- [Microsoft](https://www.microsoft.com)
- [Palantir](https://www.palantir.com)
- [Spiral](https://spiraldb.com)
- [Firebolt](https://www.firebolt.io)
- [Spice AI](https://spice.ai/blog/vortex-at-spice-ai-the-columnar-format-for-data-intensive-workloads)
- [DatologyAI](https://www.datologyai.com)
- [Dash0](https://www.polarsignals.com/blog/posts/2025/11/25/interface-parquet-vortex)
- [LangChain](https://www.langchain.com/blog/introducing-smithdb)
- [Milvus](https://milvus.io/docs/storage-v3.md)
- [DuckDB](https://duckdb.org/docs/current/core_extensions/vortex)
- [Apache DataFusion](https://datafusion.apache.org/user-guide/introduction.html)
- [Apache Arrow](https://arrow.apache.org)
- [Apache Spark](https://spark.apache.org)
- [ClickHouse](https://clickhouse.com/docs/reference/formats/Vortex)
- [Apache Iceberg](https://iceberg.apache.org/blog/apache-iceberg-file-format-api/): Vortex support is proposed, not shipped
- [Apache Hudi](https://hudi.apache.org)
- [Apache Paimon](https://paimon.apache.org/docs/master/concepts/spec/fileformat/)

## Docs

- [Questions and answers](https://vortex.dev/faq): how Vortex compares to Parquet, why it is fast, who uses it, and how to install it
- [Documentation](https://docs.vortex.dev/): guides, concepts, and the API reference
- [Documentation index for language models](https://docs.vortex.dev/llms.txt)
- [Benchmarks](https://bench.vortex.dev/): every result, charted over recent commits
- [Benchmark summaries as text](https://bench.vortex.dev/llms.txt): every group's rankings, with links to each chart
- [Source code](https://github.com/vortex-data/vortex)

## Blog

- [Have your cake and decompress it too](https://vortex.dev/blog/btrblocks-compressor): How Vortex uses BtrBlocks-style codec selection to beat Parquet+ZSTD on both size and speed.
- [December & January Bulletin](https://vortex.dev/blog/december-2025): Lazy array evaluation shipped in 0.57.0, CUDA kernels for decoding and filtering landed, and DataFusion 52 support merged.
- [November Bulletin](https://vortex.dev/blog/november-2025): Spice AI launched Cayenne on Vortex, Polar Signals replaced Parquet with Vortex, and Vortex became a DuckDB core extension.
- [October Bulletin](https://vortex.dev/blog/october-2025): An experimental GPU scan landed, the DataFusion integration pushed filters into struct fields, and DuckDB 1.4.2 made Vortex a core extension.
- [September Bulletin #2](https://vortex.dev/blog/september-2025): We released 0.54.0, published the Operators RFC, added FastLanes-based RLE, and shipped the extension for DuckDB 1.4.
- [September Bulletin](https://vortex.dev/blog/september): We added a Ray datasource, the FixedSizeList type, list support for DuckDB, and the TPC-DS and StatPopGen benchmarks.
- [Welcome to the Vortex Community Blog](https://vortex.dev/blog/welcome): Introducing a new blog to share work happening with and in the Vortex ecosystem

## Posts on other sites

- [Faster DuckDB Queries with Vortex](https://spiraldb.com/blog/duckdb-and-vortex) (spiraldb.com, 2026-10-21): With the release of DuckDB 2.0, we look at how the Vortex core extension speeds up queries by computing over compressed data, and how to read and write Vortex files from DuckDB.
- [What Vortex Has to Prove Before Iceberg Adds a Fourth File Format](https://datalakehousehub.com/blog/iceberg-new-file-format-criteria-vortex/) (datalakehousehub.com, 2026-10-21): The Vortex proposal split into criteria and candidate. What field IDs, statistics, governance, and ecosystem support a new Iceberg file format must prove.
- [Statistics are aggregates](https://spiraldb.com/blog/statistics-are-aggregates) (spiraldb.com, 2026-10-14): Min-max statistics are aggregates computed at write time. Treated that way, they can answer queries, not just skip data.
- [Fusing GPU Kernels with Dynamic Dispatch](https://spiraldb.com/blog/fusing-gpu-kernels-with-dynamic-dispatch) (spiraldb.com, 2026-10-07): GPU decompression at 2 TB/s with Fused Dynamic Dispatch. How Vortex doubles throughput by fusing CUDA kernels.
- [From S3 to GPU in One Copy](https://spiraldb.com/blog/from-s3-to-gpu-in-one-copy) (spiraldb.com, 2026-09-30): We optimized Vortex so it can stream data from object storage into GPU memory fast enough to saturate a 100 Gbps network link. The first step to get this performance is to make sure each byte is only copied once between the kernel and the GPU.
- [Vortex: One Format for Any Shape](https://spiraldb.com/blog/vortex-one-format-for-any-shape) (spiraldb.com, 2026-09-16): How Vortex keeps logical types, physical encodings, and storage layouts open to extension so one columnar format can fit any shape of data.
- [ParquetReader Now Supports Vortex Files](https://parquetreader.com/blog/parquetreader-vortex-support) (parquetreader.com, 2026-08-31): ParquetReader now supports Vortex files. Learn what the Vortex columnar format is and how to open, inspect, and convert Vortex data to CSV, JSON, or Parquet.
- [The Storage Layer for the Composable Data Stack](https://spiraldb.com/blog/vortex-the-storage-layer-for-the-composable-data-stack) (spiraldb.com, 2026-08-14): A composable query engine deserves a composable storage layer. Build the format your workload needs without rebuilding the storage stack.
- [Compile or Prefilter? Matching LIKE over Compressed Strings](https://spiraldb.com/blog/compile-or-prefilter-like-over-strings) (spiraldb.com, 2026-07-14): Two teams independently built SQL LIKE over FSST-compressed strings: Vortex prefilters candidates, TU Munich compiles the pattern. One experiment across seven x86 generations maps where each wins, where they cross, and why the two hot loops belong in one system.
- [Pattern Matching on Compressed Strings without Decompression](https://spiraldb.com/blog/pattern-matching-strings-without-decompression) (spiraldb.com, 2026-07-14): Evaluating SQL LIKE directly on FSST-compressed strings: a symbol-aware automaton and a SIMD prefilter let Vortex match without decompressing the column first.
- [My Journey Optimizing a CUDA Kernel with Polar Signals](https://www.polarsignals.com/blog/posts/2026/06/24/optimizing-fsst-cuda) (polarsignals.com, 2026-06-24): A storage engineer's descent into warp stalls, memory throttles, and aligned stores
- [AI Datasets Are Never Done. So We Built Loon.](https://zilliz.com/blog/why-we-built-loon-a-storage-engine-for-ai-data-that-never-stops-changing) (zilliz.com, 2026-06-05): Loon is a new storage engine for Milvus 3.0 and Zilliz Vector Lakebase, built to manage evolving vector datasets with ColumnGroups, row ID alignment, and Manifests.
- [We built SmithDB, the data layer for agent observability](https://www.langchain.com/blog/introducing-smithdb) (langchain.com, 2026-05-13): Introducing SmithDB: LangSmith's purpose-built distributed database for agent observability, delivering up to 12x faster performance with full portability.
- [Vortex: The Columnar Format for Data Workloads](https://spice.ai/blog/vortex-at-spice-ai-the-columnar-format-for-data-intensive-workloads) (spice.ai, 2026-04-07): Learn how Spice AI uses the Vortex columnar format in Cayenne to speed data-intensive workloads, reduce memory use, and scale concurrent ingestion and queries.
- [Introducing the Apache Iceberg File Format API](https://iceberg.apache.org/blog/apache-iceberg-file-format-api/) (iceberg.apache.org, 2026-02-20): The Apache Iceberg community is excited to announce the finalization of the File Format API, a major architectural milestone that makes file formats pluggable, consistent, and engine‑agnostic across the Iceberg Java codebase.
- [Announcing Vortex Support in DuckDB](https://duckdb.org/2026/01/23/duckdb-vortex-extension) (duckdb.org, 2026-01-23): Vortex is a new columnar file format with a very promising design. SpiralDB and DuckLabs have partnered to give you a very fast experience while reading and writing Vortex files!
- [Databases in 2025: A Year in Review](https://www.cs.cmu.edu/~pavlo/blog/2026/01/2025-databases-retrospective.html) (cs.cmu.edu, 2026-01-04): The world tried to kill Andy off but he had to stay alive to talk about what happened with databases in 2025.
- [Introducing the Spice Cayenne Data Accelerator](https://spice.ai/blog/introducing-spice-cayenne-data-accelerator) (spice.ai, 2025-12-17): Spice Cayenne is the next-generation Spice.ai data accelerator built on the Vortex columnar format for high-scale, low-latency data lake acceleration workloads.
- [Great Lakes: A Delta Lake Fork](https://www.polarsignals.com/blog/posts/2025/12/04/great-lakes) (polarsignals.com, 2025-12-04): Our updates to Delta Lake Table Format
- [Questioning an Interface: From Parquet to Vortex](https://www.polarsignals.com/blog/posts/2025/11/25/interface-parquet-vortex) (polarsignals.com, 2025-11-25): Breaking free from the shackles of interface-imposed performance limitations
- [Exploring the Evolving File Format Landscape in AI Era: Parquet, Lance, Nimble and Vortex And What It Means for Apache Iceberg](https://www.dremio.com/blog/exploring-the-evolving-file-format-landscape-in-ai-era-parquet-lance-nimble-and-vortex-and-what-it-means-for-apache-iceberg/) (dremio.com, 2025-10-20): Formats like Lance, Nimble, and Vortex show that innovation is accelerating. Each one introduces new ideas about how to structure, index, and access data. But none of them exist in isolation. They need to integrate with engines, catalogs, and governance layers to be truly useful.
- [Is Parquet becoming the bottleneck? Why new storage formats are emerging in 2025 (Lance, Vortex, and more)](https://www.databend.com/blog/category-engineering/2025-09-15-storage-format/) (databend.com, 2025-09-15): Parquet gave data lakes a common language: columnar layout, good compression, and fast scans.
- [Vortex: a Linux Foundation Project](https://spiraldb.com/blog/vortex-a-linux-foundation-project) (spiraldb.com, 2025-08-06): Building the Future of Open Source, Columnar Storage
- [Vortex on Ice](https://spiraldb.com/blog/vortex-on-ice) (spiraldb.com, 2025-04-30): Using Vortex to accelerate Apache Iceberg queries up to 4x
- [Towards Vortex 1.0](https://spiraldb.com/blog/towards-vortex-10) (spiraldb.com, 2025-04-24): Preparing to launch Vortex 1.0 and the 2025.05 Edition
- [What if we just didn’t decompress it?](https://spiraldb.com/blog/what-if-we-just-didnt-decompress-it) (spiraldb.com, 2025-03-05): How push-down compute helps Vortex run faster and decompress less
- [Zone maps, or “queries go brrr”](https://spiraldb.com/blog/zone-maps-or-queries-go-brrr) (spiraldb.com, 2025-02-25): Waiting sucks. Let's use zone maps to move fast and read less.
- [Data Layouts: Where Bytes Find Their Forever Home](https://spiraldb.com/blog/data-layouts-where-bytes-find-their-forever-home) (spiraldb.com, 2025-02-10): How Vortex tackles data layout challenges across analytics and machine learning workflows.
- [Logical vs Physical data types](https://spiraldb.com/blog/logical-vs-physical-data-types) (spiraldb.com, 2025-02-08): The rationale behind the Vortex logical type system.
- [Compressing strings with FSST](https://spiraldb.com/blog/compressing-strings-with-fsst) (spiraldb.com, 2024-11-09): Random access string compression with FSST and Rust.
- [Trick-or-treating with Vortex](https://spiraldb.com/blog/trick-or-treating-with-vortex) (spiraldb.com, 2024-10-31): On a journey to make your data more fun sized.
- [Life in the fastlanes](https://spiraldb.com/blog/life-in-the-fastlanes) (spiraldb.com, 2024-06-19): Decoding > 100 billion integers per second with scalar Rust.

## Optional

- [Full text of this file and of every post](https://vortex.dev/llms-full.txt)
- [Vortex Technical Charter](https://vortex.dev/charter.pdf)
- [Community Slack](https://vortex.dev/slack)