Polars AWS S3 support (#14648)

# Description

Provides Amazon S3 support.

- Utilizes your existing AWS cli configuration. 
- Supports AWS SSO
- Supports
[gimme-aws-creds](https://github.com/Nike-Inc/gimme-aws-creds).
- respects the settings of AWS_PROFILE environment variable for
selecting profile config
- AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_REGION environment
variables for configuring without an AWS config

Usage:
```nushell
polars open s3://bucket/and/path.parquet
```

Supports:
- CSV
- Parquet
- NDJSON / json lines
- Arrow

Doesn't support:
- eager dataframes
-  Avro
- JSON
This commit is contained in:
Jack Wright
2024-12-25 04:15:50 -08:00
committed by GitHub
parent f2dcae570c
commit 23ba613b00
11 changed files with 843 additions and 103 deletions

View File

@@ -6,5 +6,12 @@ static GLOBAL: mimalloc::MiMalloc = mimalloc::MiMalloc;
fn main() {
env_logger::init();
serve_plugin(&PolarsPlugin::default(), MsgPackSerializer {})
match PolarsPlugin::new() {
Ok(ref plugin) => serve_plugin(plugin, MsgPackSerializer {}),
Err(e) => {
eprintln!("{}", e);
std::process::exit(1);
}
}
}