FAQ
Common questions about Vortex, the open-source columnar file format.
What is Vortex?
Vortex is an open-source columnar file format. Like Apache Parquet, it stores a table column by column, so a query reads only the columns it needs. Its goals are fast scans, fast random access, and good compression.
Vortex has libraries for Rust, Python, Java, C, and C++. DuckDB, Apache DataFusion, Apache Spark, and Ray can query Vortex files, and Pandas, Polars, and PyArrow can read them through the Python library.
The quickstart · Guides for each engine
How does Vortex compare to Apache Parquet?
bench.vortex.dev runs these public benchmarks on each commit to the develop branch of Vortex. Its newest results against Parquet, read on 11 Nov 2026:
- Scan throughput: 5.1× faster
- Write throughput: 91% faster
- ClickBench on DataFusion: 62% faster
- TPC-H SF10 on DataFusion: 2.4× faster
- GPU analytics with cuDF: 3.9× faster (not from bench: see the note below)
- Random access, NVMe, hot: 90× faster
- Size vs Parquet+Zstd: 11% larger
The GPU figure is an internal measurement, not a bench result. The Vortex maintainers measured query time on an NVIDIA GH200, a data-center GPU, with cuDF and a hot cache. The workload is NDS-H, a benchmark derived from TPC-H, at scale factor 10: the five queries that cuDF runs. Each Parquet time is the fastest of four Parquet writer settings.
Every benchmark on bench.vortex.dev · How the benchmarks run
Should I use Vortex instead of Parquet?
For almost every workload, yes. In the benchmarks above, Vortex is faster than Parquet at scans, writes, ClickBench queries, TPC-H queries, queries on GPUs, and single-row reads.
Parquet still has two advantages. Its files are smaller: on bench's datasets, Vortex files are 11% larger than Parquet files with zstd. More tools also read Parquet today.
Where storage cost or tool support matters most, Parquet can still be the better choice.
How does Vortex compare to Lance?
Lance is a table format and a file format with the same name. The Lance table format keeps versions of a table and supports secondary indexes, vector indexes among them. It is an alternative to lakehouse table formats such as Apache Iceberg and Delta Lake.
Vortex, like Parquet, is only a file format. A Vortex file can hold its own indexes. By default it keeps zone maps (statistics on blocks of rows), so a scan skips blocks that cannot match its filter. A custom layout can add other indexes. Indexes outside the file, such as a vector index over a whole table, and table versions are a table format's job. Apache Hudi and Apache Paimon can store tables in Vortex files. Apache Iceberg has a proposal to store tables in Vortex files.
On bench.vortex.dev's newest results, read on 11 Nov 2026, Vortex is faster than Lance in every benchmark that runs both, and its files are smaller:
- Scan throughput: 5.3× faster
- Write throughput: 2.9× faster
- ClickBench on DataFusion: 14× faster
- TPC-H SF10 on DataFusion: 14× faster
- Random access, NVMe, hot: 2.8× faster
- File size: 48% smaller
Both formats are under the Apache License 2.0. Vortex is a Linux Foundation project. LF Projects, part of the Linux Foundation, holds the Vortex trademark, and the charter keeps the project's code under the Apache License. Changing the charter takes a two-thirds vote of the Technical Steering Committee and the approval of LF Projects. Lance has its own governance, but the name and code are held by a private company (LanceDB) instead of an independent foundation.
Every benchmark on bench.vortex.dev · The Vortex Technical Charter
Why is Vortex fast?
Vortex separates a column's logical type, such as a string or a 64-bit integer, from the encoding that stores it. Each column gets the encoding that suits its data. Vortex can also apply one encoding on top of another (cascading compression).
Vortex runs filters and other compute on encoded data, often without decompressing the column. It loads a column's statistics only when a query needs them. It converts to and from Apache Arrow without copying.
Vortex's concepts · How statistics prune a scan
Is the Vortex file format stable?
Yes. From release 0.36.0, each release reads the files that any earlier release from 0.36.0 on wrote. For an older release to read a newer release's files, the writer targets an edition (a named set of wire formats) that the older release supports. The library APIs can still change between releases.
How do I install Vortex?
Run the command for your language or engine:
- Rust:
cargo add vortex - Python:
uv add vortex-data - CLI:
uvx --from vortex-data vx --help - DuckDB:
INSTALL vortex; LOAD vortex; SELECT * FROM read_vortex('data.vortex'); - DataFusion:
cargo add vortex-datafusion - Spark:
implementation("dev.vortex:vortex-spark_2.13:0.90.0:all")
Install the vx CLI · Quickstarts for each language
Who uses Vortex, and what does it work with?
The adopters of Vortex, and the projects that it works with, include:
- Microsoft
- Palantir
- Spiral
- Firebolt
- Spice AI
- DatologyAI
- Dash0
- LangChain
- Milvus
- DuckDB
- Apache DataFusion
- Apache Arrow
- Apache Spark
- ClickHouse
- Apache Hudi
- Apache Paimon
Apache Iceberg has a proposal to add Vortex support.
Who maintains Vortex, and how is it licensed?
Vortex is a project of LF AI & Data, the part of the Linux Foundation for AI and data projects. The Vortex Technical Charter governs it. Spiral donated Vortex to the Linux Foundation.
Vortex has committers and maintainers who are affiliated with organizations that include Microsoft, Dash0, Firebolt, ClickHouse, Kenn Software, Carnegie Mellon University, CWI, and Tsinghua University.
The code is under the Apache License 2.0, and the documentation is under CC BY 4.0.
The Vortex Technical Charter · The LF AI & Data announcement · Spiral's post on donating Vortex
How do I get help or contribute?
Ask in the community Slack, or open an issue or a pull request on GitHub.