126 releases (46 breaking)
0.46.0 | May 6, 2024 |
---|---|
0.45.1 | Mar 2, 2024 |
0.45.0 | Feb 5, 2024 |
0.44.0 | Dec 31, 2023 |
0.3.0 | Mar 25, 2022 |
#17 in Filesystem
54,097 downloads per month
Used in 45 crates
(31 directly)
3MB
57K
SLoC
Apache OpenDAL™
OpenDAL is a data access layer that allows users to easily and efficiently retrieve data from various storage services in a unified way.
- Documentation: stable | main
- Release notes
Services
Standard Storage Protocols (like ftp, webdav)
Object Storage Services (like s3, gcs, azblob)
- azblob: Azure Storage Blob services
- cos: Tencent Cloud Object Storage services
- gcs: Google Cloud Storage Service
- obs: Huawei Cloud Object Storage Service (OBS)
- oss: Aliyun Object Storage Service (OSS)
- s3: AWS S3 alike services
- supabase: Supabase Storage Service being worked on
- wasabi: Wasabi Cloud Storage
File Storage Services (like fs, azdls, hdfs)
- fs: POSIX alike file system
- azdls: Azure Data Lake Storage Gen2 services (As known as abfs)
- hdfs: Hadoop Distributed File System(HDFS)
- ipfs: InterPlanetary File System HTTP Gateway
- ipmfs: InterPlanetary File System MFS API being worked on
- webhdfs: WebHDFS Service
Consumer Cloud Storage Service (like gdrive, onedrive, icloud drive)
- gdrive: Google Drive being worked on
- onedrive: OneDrive being worked on
- icloud: Icloud Drive being worked on
Key-Value Storage Service (like rocksdb, sled)
Cache Storage Service (like memcached, moka)
- ghac: GitHub Action Cache Service
- memcached: Memcached service
- mini_moka: Mini Moka backend
- moka: Moka backend
- vercel_artifacts: Vercel Remote Caching Service being worked on
Welcome to add any services that are not currently supported here.
Features
Access data freely
- Access different storage services in the same way
- Behavior tests for all services
Access data painlessly
- 100% documents covered
- Powerful
Layers
- Automatic retry support
- Full observability: logging, tracing, metrics.
- Native chaos testing
Access data efficiently
- Zero cost: Maps to API calls directly
- Best effort: Automatically selects best read/seek/next based on services
- Avoid extra calls: Reuses metadata when possible
Quickstart
use opendal::Result;
use opendal::layers::LoggingLayer;
use opendal::services;
use opendal::Operator;
#[tokio::main]
async fn main() -> Result<()> {
// Pick a builder and configure it.
let mut builder = services::S3::default();
builder.bucket("test");
// Init an operator
let op = Operator::new(builder)?
// Init with logging layer enabled.
.layer(LoggingLayer::default())
.finish();
// Write data
op.write("hello.txt", "Hello, World!").await?;
// Read data
let bs = op.read("hello.txt").await?;
// Fetch metadata
let meta = op.stat("hello.txt").await?;
let mode = meta.mode();
let length = meta.content_length();
// Delete
op.delete("hello.txt").await?;
Ok(())
}
Examples
The examples are available at here.
Contributing
Check out the CONTRIBUTING guide for more details on getting started with contributing to this project.
License and Trademarks
Licensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0
Apache OpenDAL, OpenDAL, and Apache are either registered trademarks or trademarks of the Apache Software Foundation.
Dependencies
~9–63MB
~1M SLoC