Overview

SeaweedFS is a distributed storage system for object storage, files and data-lakehouse tables, built to hold billions of files and serve them fast. A single weed binary runs an S3-compatible object store, a POSIX-style file system through its filer, and S3 Table Buckets for Apache Iceberg and Lance data over the same on-disk data. The project has been developed in the open since 2014, is written in Go and is released under the Apache License 2.0.

How the storage works

• Master nodes — one server, or a Raft group of three, tracks which volume lives on which volume server and hands out file ids. It is not in the read path: clients cache the volume-to-server mapping and talk to volume servers directly.
• Volume servers — blobs are written into append-only volume files, each with a 16-byte in-memory index entry, so a read is a single disk seek. Small files are packed together with roughly 40 bytes of metadata per file and no per-file inode.
• Filer — adds directories and files on top and keeps that metadata in a store you already run: LevelDB, RocksDB, SQLite, MySQL, PostgreSQL, Cassandra, HBase, MongoDB, Redis, Elasticsearch, etcd, TiKV, FoundationDB, YDB, ArangoDB, Tarantool, or MySQL/PostgreSQL-compatible systems such as TiDB and CockroachDB.
• Scaling out — capacity is one more volume server pointed at the master, and filers and S3 gateways are stateless, so throughput grows by running more of them. Nothing is rebalanced until you ask for it.

Object storage and the S3 API

• The S3 gateway implements the object and bucket APIs with 73 operations, S3 Tables with 36, IAM with 39 and STS with 5, so the AWS CLI and SDKs, rclone, restic, Spark and Trino work unchanged.
• Versioning, Object Lock with retention and legal hold, lifecycle rules, tagging, CORS, checksums, presigned URLs, browser POST uploads, multipart uploads and an atomic RenameObject.
• Bucket policies with conditions and variables, IAM users, groups and policies, and STS through OIDC, LDAP or Kubernetes service accounts.
• Server-side encryption with SSE-S3, SSE-KMS or SSE-C against OpenBao/Vault, AWS KMS, Azure Key Vault or GCP KMS, plus audit logging, bucket quotas and rate limiting.

File access and the lakehouse

• The same data is reachable as a FUSE mount on Linux, macOS and Windows, over WebDAV, SFTP, HDFS, plain HTTP and TUS resumable uploads, or from Kubernetes through the CSI driver, the Operator and a Helm chart.
• S3 Table Buckets hold Apache Iceberg tables by default, or Lance tables for vector and multimodal data, served by a built-in Iceberg REST catalog and Lance namespace — no Hive Metastore or Glue to run alongside it.
• Cloud Drive mounts an S3, Google Cloud Storage, Azure, Backblaze B2, Wasabi or Storj bucket and caches content with the capacity of the whole cluster, while Cloud Tier moves whole warm volumes out to cloud storage.
• Active-active or active-passive replication between clusters, filer-store replication for metadata high availability, change data capture with webhooks, metadata backup and AES-256-GCM encryption at rest.

Operating it day to day

• A web Admin UI covers the dashboard, cluster and volume inventory, bucket and user management, policies and a file browser; the master and filer keep their own smaller status pages.
• Prometheus metrics, per-file or per-volume TTL, automatic compression and compaction, maintenance workers that balance, vacuum and repair volumes, and weed shell for erasure-coding jobs.
• Hot data is replicated for speed and warm data is erasure-coded in the background, so writes never pay the encoding cost; a Rust volume server can replace the Go one on the same on-disk format.

Why choose SeaweedFS

Most distributed file systems are more complicated than they need to be, and SeaweedFS is deliberately flat. The master tracks volumes rather than files, so a cluster with billions of files still has only a few thousand volumes and a small master; a file’s volume id is a cacheable lookup instead of a hash or metadata traversal; and adding a server adds capacity without a data reshuffle — the two costs that make Ceph’s CRUSH placement and an in-memory HDFS namenode painful at scale. The project’s own benchmark on a MacBook SSD with 1 KB files at concurrency 16 reports 15,708 writes and 47,019 random reads per second, and a mixed S3 warp run totalling about 3.3 GiB/s, with throughput growing per volume server added.

Common uses

• Object storage for applications that would otherwise rent it, with an S3 endpoint that existing SDKs and backup tools already speak.
• Image, video and document repositories where millions of small files break inode-bound or hash-based designs.
• A backup target for restic, rclone or Velero, with versioning and Object Lock for retention.
• A self-hosted lakehouse for Iceberg or Lance tables queried by Spark, Trino, DuckDB or ClickHouse.
• A fast cache in front of cloud buckets, so repeated reads never leave the local network.

Deployment and requirements

Release builds cover Linux on amd64, arm and arm64, macOS on Intel and Apple silicon, Windows on amd64, and FreeBSD and OpenBSD; container images, a Helm chart, the Kubernetes Operator and a CSI driver cover the container side. A ready-to-use single-node stack is one command — ./weed mini -dir=./data starts the master, a volume server, the filer, the S3 gateway and WebDAV together — and it is auto-tuned for one node, so it is usable for real single-node work such as an S3 gateway issuing presigned URLs. Without an admin password or mTLS the Admin UI and its gRPC worker control plane bind to loopback instead of the public interface; set WEED_ADMIN_PASSWORD or configure https.admin mTLS before exposing it. Volumes up to 8 TB need the large-disk build, and single files range from a byte to tens of terabytes. A commercial Enterprise Edition sold from seaweedfs.com adds point-in-time recovery, sealed directories, a kernel mount and support, but the Apache-2.0 core runs without a licence.

Related tools

• Rclone — speaks the S3 and WebDAV endpoints directly, for copying data in and out of a cluster.
• Seafile — a user-facing sync and share application with block deduplication, where SeaweedFS is the storage layer such an application would sit on.
• Nextcloud — a full collaboration suite that can use S3-compatible primary storage instead of local disks.
• Syncthing — peer-to-peer device sync with no central store, a different answer to the same file-distribution problem.