Update: The plan is to use Cap’n Proto instead of Apache Arrow, which offers true zero-copy, and create a communication protocol similar to Apache Flight or even more straightforward to use directly QUIC or HTTP/3. Maybe we can do both implementations with Apache Arrow and Cap’n Proto.

https://docs.google.com/document/d/1Ru5UlOz-4dz9ors3FKEg2Ac-KxKVqI_wR0EitECp-mo/edit?usp=drivesdk

Idea: Make Apache Flight (gRPC) work over HTTP/3 (QUIC) and make use of RDMA (fast memory transfer). All made in Rust

Motivation
- challenge: there are no well-known production-ready implementations
- faster transfers
- faster connection time
- reduced latency
- Reduced Downtime
- High Throughput
- multiplexing without head-of-line blocking
- particularly beneficial for short-lived connections or high-frequency RPC calls
- QUIC retransmits lost packets faster than TCP, reducing delays
- QUIC allows connections to migrate between network interfaces (e.g., Wi-Fi to cellular) without disconnecting
- QUIC includes TLS 1.3 at the transport layer, so all HTTP/3 connections are encrypted by design
- HTTP/3 allows prioritizing certain streams, optimizing resource utilization for high-performance data pipelines
- QUIC supports an unlimited number of concurrent streams without degrading performance

Using HTTP/3 and QUIC for Arrow Flight and Tonic brings several potential advantages, especially in scenarios requiring high-performance data exchange and improved reliability over modern networks. Below are the key advantages:

This would involve
- extend Quinn crate for HTTP/3
- change the transport layer in tonic crate to use the above, hence we'll have gRPC over HTTP/3
- have Apache Flight with Arrow working with the above, this will give us Flight over HTTP/3
- make use of RDMA for fast memory transfer

here are some details https://chatgpt.com/share/673ec4ef-ccdc-8003-8dab-d1a4ecd5005d

Does anyone care to get involved? all this will be open source