Polars

Polars is a DataFrame library written in Rust for analytical workloads on a single machine, used mainly from Python and also available as a Rust crate. It stores data in the Apache Arrow columnar format and runs queries on all CPU cores in parallel, without the Python GIL getting in the way. On common operations such as filtering, grouping, joining, and sorting it is typically several times faster than pandas and uses less memory.

Polars has two modes. The eager API runs each step immediately, like pandas. The lazy API builds a query plan first and optimises it (predicate and projection pushdown, common subexpression elimination) before anything runs, and its streaming engine processes datasets larger than RAM in batches. Transformations are written as composable expressions rather than index-based operations, and SQL queries run through pl.SQLContext. An optional GPU engine, in open beta and built on NVIDIA RAPIDS cuDF, runs lazy queries on NVIDIA GPUs with collect(engine="gpu").

The library is free and open source under the MIT licence. Its company, Polars Inc., sells Polars Cloud, which runs the same Polars code distributed across machines inside the customer's own AWS account, billed per vCPU-hour of query execution with a 30-day free trial.

The main friction is the ecosystem: many plotting and machine learning libraries still expect pandas DataFrames, so pipelines often convert at the edges (cheaply, through Arrow).