git-lfs-walrus
Git LFS adapter for storing large files on the Walrus decentralized storage network
Project Description
Git LFS replaces large files with text pointers in Git repositories while storing the actual files on remote servers, keeping repos lightweight. Traditional Git LFS relies on centralized storage (GitHub LFS, etc.) with vendor lock-in and single points of failure. git-lfs-walrus provides a decentralized alternative using Walrus distributed storage with erasure coding for redundancy, censorship resistance, and true data ownership.
How it's Made
git-lfs-walrus is built in Rust using the Tokio async runtime for high-performance I/O operations. The project leverages Git LFS's custom transfer protocol to intercept file operations and redirect them to Walrus storage.
The architecture consists of three main components:
- Smudge operation: Retrieves original files from Walrus when checking out, using blob IDs from LFS pointers
- Transfer protocol: Handles upload/download operations for Git LFS custom transfers with progress reporting
We used Walrus's JSON API for programmatic interaction with the storage network, sending commands through the Walrus CLI and parsing responses to extract blob IDs and handle errors. The implementation uses Serde for JSON serialization/deserialization and StructOpt for command-line interface.
Key technical innovations include async streaming for large file handling, robust error handling for network operations, and a mapping system between Git LFS SHA256 hashes and Walrus blob IDs embedded in LFS pointer files as comments.
