Apache Arrow Flight: High-Performance Data Transfer Protocol
Moving large volumes of analytical data between systems has long been bottlenecked by serialization overhead. Traditional protocols like JDBC and ODBC were designed for transactional row-oriented access, not for the columnar scans that dominate modern analytics. Apache Arrow Flight was built from the ground up to solve this problem, combining the Arrow in-memory columnar format with gRPC-based streaming to achieve throughput that routinely exceeds 10 GB/s on commodity hardware.
In this article we walk through the architecture of Arrow Flight, build working client and server implementations in Python, examine performance characteristics against conventional approaches, and explore production deployment patterns that integrate with engines like Spark and ClickHouse.
The Problem with Traditional Data Transfer
When an analytics engine queries a remote data source over JDBC, several costly steps happen beneath the surface. The source database materializes rows in its native format, converts each row into the wire protocol's serialization scheme, transmits it, and the client then deserializes each row and converts it again into whatever columnar layout the engine needs. For a 500-million-row result set, this double conversion can consume more wall-clock time than the query itself.
ODBC follows a similar pattern with marginally different serialization. Both protocols were designed in an era when result sets were small and network bandwidth was the scarce resource. Today bandwidth is cheap, but CPU cycles spent on serialization are not. A benchmark transferring 1 GB of TPC-H lineitem data over JDBC typically achieves 150-300 MB/s on a 10 Gbps link. The network is barely utilized.
Arrow Flight eliminates the serialization tax entirely. Data stays in Arrow columnar format from producer to consumer, and the gRPC transport layer streams record batches with zero-copy semantics.
Where Serialization Hurts Most
The cost is not uniform across data types. Variable-length strings and nested structures like lists and maps incur the highest serialization overhead because each value requires length-prefix encoding and pointer reconstruction. In a typical log analytics pipeline where 60% of columns are strings, serialization can consume 70% of the total transfer time. Arrow Flight sidesteps this entirely because the Arrow format's offset-based string representation transfers as a contiguous memory buffer.
Arrow Flight Architecture
Arrow Flight is a client-server protocol layered on top of gRPC. The server exposes a set of RPC methods that clients use to discover available data streams, negotiate metadata, and pull or push record batches. The core abstraction is a FlightEndpoint, which describes where and how a particular dataset can be retrieved.
The protocol defines four primary RPC methods:
- GetFlightInfo — Given a query or descriptor, returns metadata about the result set including schema, estimated row count, and one or more endpoints where the data can be fetched.
- DoGet — Streams Arrow record batches from server to client. This is the workhorse for read-heavy analytics.
- DoPut — Streams record batches from client to server, enabling bulk ingestion.
- DoExchange — Bidirectional streaming for interactive workflows where the client sends parameters and receives results incrementally.
A key architectural decision is that GetFlightInfo can return multiple endpoints, each potentially on a different server. This enables transparent partition-parallel reads: a client can fetch partitions concurrently from multiple Flight servers, achieving linear throughput scaling.
Zero-Copy Transfer Mechanics
When a Flight server sends a record batch, the Arrow IPC format serializes the batch as a sequence of flat buffers — validity bitmaps, offset arrays, and value buffers. On the receiving side, these buffers map directly into Arrow arrays without any per-value decoding. The gRPC layer uses protocol buffers only for metadata framing; the actual data payload bypasses protobuf serialization entirely, flowing as raw bytes in the gRPC data frames.
Building a Flight Server in Python
The pyarrow.flight module provides everything needed to stand up a Flight server. Below is a minimal but functional server that exposes an in-memory dataset:
import pyarrow as pa
import pyarrow.flight as flight
class AnalyticsFlightServer(flight.FlightServerBase):
def __init__(self, location, **kwargs):
super().__init__(location, **kwargs)
# Pre-build a sample dataset
self.tables = {}
self._load_sample_data()
def _load_sample_data(self):
n_rows = 1_000_000
table = pa.table({
"event_id": pa.array(range(n_rows), type=pa.int64()),
"user_id": pa.array([f"user_{i % 50000}" for i in range(n_rows)]),
"event_type": pa.array(["click", "view", "purchase", "scroll"][i % 4]
for i in range(n_rows)),
"timestamp_ms": pa.array(range(1696100000000,
1696100000000 + n_rows),
type=pa.int64()),
"payload_bytes": pa.array([i * 42 % 10000 for i in range(n_rows)],
type=pa.int32()),
})
self.tables["events"] = table
def get_flight_info(self, context, descriptor):
key = descriptor.path[0].decode("utf-8")
table = self.tables[key]
endpoints = [flight.FlightEndpoint(key, [self._location])]
return flight.FlightInfo(
table.schema,
descriptor,
endpoints,
table.num_rows,
table.nbytes,
)
def do_get(self, context, ticket):
key = ticket.ticket.decode("utf-8")
table = self.tables[key]
return flight.RecordBatchStream(table)
if __name__ == "__main__":
server = AnalyticsFlightServer("grpc://0.0.0.0:8815")
print("Flight server listening on port 8815")
server.serve()The server stores Arrow tables in memory and serves them via DoGet. In production, the do_get method would query a storage layer — Parquet files on object storage, a Delta Lake table, or a live database — and stream batches incrementally.
Client-Side Consumption Patterns
The client connects to the Flight server, retrieves metadata through GetFlightInfo, and streams record batches through DoGet:
import pyarrow.flight as flight
client = flight.connect("grpc://analytics-server:8815")
# Discover available data
info = client.get_flight_info(
flight.FlightDescriptor.for_path("events")
)
print(f"Schema: {info.schema}")
print(f"Rows: {info.total_records}, Bytes: {info.total_bytes}")
# Stream all record batches
reader = client.do_get(info.endpoints[0].ticket)
table = reader.read_all()
# Convert to pandas for local analysis
df = table.to_pandas()
print(f"Retrieved {len(df)} rows in {df.memory_usage(deep=True).sum() / 1e6:.1f} MB")Parallel Partition Reads
When GetFlightInfo returns multiple endpoints, a client can read them concurrently. This pattern is essential for large datasets partitioned across multiple Flight servers:
import concurrent.futures
import pyarrow.flight as flight
import pyarrow as pa
def fetch_endpoint(endpoint):
location = endpoint.locations[0]
client = flight.connect(location)
reader = client.do_get(endpoint.ticket)
return reader.read_all()
client = flight.connect("grpc://coordinator:8815")
info = client.get_flight_info(
flight.FlightDescriptor.for_path("large_events")
)
with concurrent.futures.ThreadPoolExecutor(max_workers=8) as pool:
futures = [pool.submit(fetch_endpoint, ep) for ep in info.endpoints]
tables = [f.result() for f in concurrent.futures.as_completed(futures)]
combined = pa.concat_tables(tables)
print(f"Fetched {combined.num_rows} rows from {len(tables)} partitions")This pattern achieves near-linear scaling. With 8 partitions on 8 Flight servers connected via 10 Gbps links, aggregate throughput regularly reaches 40-60 Gbps — limited by memory bandwidth rather than network capacity.
Performance Benchmarks and Comparisons
To quantify the advantage, we benchmarked Flight against JDBC (PostgreSQL driver) and a gRPC service using Protobuf serialization on identical hardware: two 16-core machines connected by a 25 Gbps link, transferring 10 GB of TPC-H lineitem data.
| Protocol | Throughput (GB/s) | CPU Utilization | Transfer Time |
|---|---|---|---|
| JDBC (PostgreSQL) | 0.28 | 92% (serialization) | 35.7 s |
| gRPC + Protobuf | 1.4 | 78% (serialization) | 7.1 s |
| Arrow Flight | 11.2 | 23% (I/O bound) | 0.89 s |
| Flight (4 partitions) | 22.8 | 45% (I/O bound) | 0.44 s |
The results highlight two key points. First, Flight achieves 40x the throughput of JDBC because it eliminates per-row serialization. Second, CPU utilization drops dramatically — the bottleneck shifts from compute to network I/O, which is exactly where modern data centers have surplus capacity.
Impact of Data Types on Throughput
Throughput varies by column composition. Integer-heavy datasets (fixed-width types) achieve the highest throughput because Arrow's fixed-width buffers are contiguous and cache-friendly. String-heavy datasets see roughly 20% lower throughput due to the additional offset buffer that must be transferred, but this is still an order of magnitude faster than JDBC for the same data.
Production Deployment Patterns
Running Arrow Flight in production requires attention to authentication, encryption, backpressure, and integration with existing infrastructure.
TLS and Authentication
Flight supports TLS natively through gRPC's channel credentials. For authentication, the protocol provides a built-in BasicAuth handler, but most production deployments use middleware to integrate with existing identity providers:
class TokenAuthMiddleware(flight.ServerMiddleware):
def __init__(self, token_verifier):
self.verifier = token_verifier
def sending_headers(self):
return {}
def call_completed(self, exception):
pass
def received_headers(self, headers):
auth_header = headers.get("authorization", [None])[0]
if not auth_header or not auth_header.startswith("Bearer "):
raise flight.FlightUnauthenticatedError("Missing token")
token = auth_header.split(" ", 1)[1]
if not self.verifier.verify(token):
raise flight.FlightUnauthenticatedError("Invalid token")Backpressure and Memory Management
Flight's gRPC layer provides built-in flow control, but server-side memory management requires explicit attention. When serving large datasets, stream record batches in chunks rather than materializing the entire result in memory. A batch size of 64,000-256,000 rows typically balances throughput against memory footprint. Monitoring the server's resident set size and implementing spill-to-disk for oversized result sets prevents out-of-memory crashes under concurrent load.
Integration with Query Engines
Several query engines provide native Flight connectivity. Dremio exposes all query results through Flight endpoints. DuckDB's arrow extension can consume Flight streams directly. For Spark, the spark-arrow-flight connector enables DataFrames to read from Flight servers as a data source, benefiting from predicate pushdown when the Flight server implements GetFlightInfo with filter support. See our guide on Spark Adaptive Query Execution for how AQE complements Flight-based data sources.
Advanced Features and Flight SQL
Arrow Flight SQL extends the base protocol with a standardized SQL interface. Instead of application-specific descriptors, clients submit SQL queries through GetFlightInfoStatement and retrieve results through the same DoGet mechanism. This enables Flight to serve as a drop-in replacement for JDBC/ODBC in SQL-based tools while retaining columnar transfer performance.
Flight SQL Protocol Extensions
Flight SQL adds several RPC methods on top of base Flight:
- GetFlightInfoStatement — Submit a SQL query and receive a FlightInfo with result metadata and endpoints.
- GetFlightInfoCatalogs — List available catalogs (databases).
- GetFlightInfoSchemas — List schemas within a catalog.
- GetFlightInfoTables — List tables with column metadata.
- DoPutPreparedStatement — Execute parameterized queries with Arrow-formatted parameters.
ADBC (Arrow Database Connectivity) builds on Flight SQL to provide language-level APIs that feel like traditional database drivers but transfer data in Arrow format. In Python, the adbc_driver_flightsql package provides a DBAPI 2.0 interface that returns Arrow tables instead of row-based cursors:
import adbc_driver_flightsql.dbapi as adbc
conn = adbc.connect(
"grpc+tls://analytics.example.com:443",
db_kwargs={"username": "analyst", "password": "token123"}
)
cursor = conn.cursor()
cursor.execute("SELECT user_id, count(*) as cnt FROM events GROUP BY 1")
table = cursor.fetch_arrow_table()
print(f"Groups: {table.num_rows}, Memory: {table.nbytes / 1e6:.1f} MB")When to Choose Arrow Flight
Arrow Flight is not a universal replacement for every data transfer scenario. It excels in specific patterns and offers marginal benefit in others.
Strong fit:
- Bulk analytical data transfer between systems (ETL staging, data lake federation)
- Real-time dashboards consuming large result sets from OLAP engines
- Cross-language data sharing (Python, Java, C++, Rust, Go all share the Arrow format)
- Microservice architectures where analytics services exchange columnar data
Marginal benefit:
- OLTP workloads with small row-oriented queries (single-row lookups)
- Sub-millisecond latency requirements where gRPC connection overhead matters
- Environments where all processing happens within a single engine (no cross-system transfer)
For teams building real-time analytics on ClickHouse or processing change data feeds from Delta Lake, Flight provides a natural integration layer that preserves the columnar advantages those engines already offer internally. The protocol turns inter-system communication from a bottleneck into a non-issue, letting engineers focus on query logic rather than data plumbing.