Streaming engine innovation: cloud-native, high-performance video file system

Versatile storage support

Introducing Callaba's File System: a versatile and pragmatic solution tailored for the intricate demands of managing video files and data-intensive applications in today's diverse cloud environments. 

• Elastic throughput & high performance
Callaba’s caching system efficiently handles video files by smoothly managing data flow, particularly in areas of high access or concentration.

The metadata service quickly processes millions of requests per second, responding with microsecond-level latency. This speed is essential for tasks like searching, sorting, or retrieving specific details about your video recordings/media assets. 

• Budget-friendly storage
Callaba’s design combines budget-friendly object storage and scalable high-throughput performance, enabling independent scaling. This way, storage that remains cost-effective while delivering great performance. 

• Wide storage compatibility
Callaba’s architecture supports open-source storage engines and over 40 object storage systems, effortlessly meeting varying storage needs for very different projects.

• Cloud-native architecture
Operating as a high-performance POSIX file system, Callaba is specifically tailored for the dynamic cloud-native environment. Callaba can serve different cloud setups, including public, hybrid, and multi-cloud configurations. An important feature is its automatic replication of data across different clouds and regions, providing redundancy and creating a sturdy foundation for multi-cloud systems.

 When you use Callaba to store data, it resides in Object Storage (e.g Amazon S3). The associated metadata can be stored in compatible databases such as Redis, MySQL, and TiKV, depending on specific needs.

Highlighted Features

Fully POSIX-compatible 
Use as a local file system, seamlessly docking with existing applications without breaking business workflow.

Fully Hadoop-compatible
Callaba is compatible with Hadoop 2.x and Hadoop 3.x as well as a variety of components in the Hadoop ecosystems.

S3-compatible
Callaba provides an S3-compatible interface.

Cloud Native
A Kubernetes CSI driver is provided for easily using Callaba in Kubernetes.

Shareable
Callaba is a shared file storage that can be read and written by thousands of clients.

Lightning fast
The latency can be as low as a few milliseconds, and the throughput can be expanded nearly unlimitedly (depending on the size of the Object Storage).

Data Encryption
Supports data encryption in transit and at rest.

Global File Locks
Callaba supports both BSD locks (flock) and POSIX record locks (fcntl).

Data Compression
Callaba supports LZ4 or Zstandard to compress all your data.

Callaba’s File System consists of three parts

1. Client

Coordinates Object Storage and metadata storage engine as well as implementation of file system interfaces such as POSIX, Hadoop, Kubernetes, and S3 gateway.

2.  Storage

Stores data, with supports of a variety of data storage media, e.g., local disk, public or private cloud Object Storage, and HDFS.

3. Metadata Engine

Stores the corresponding metadata that contains information of file name, file size, permission group, creation and modification time and directory structure, etc., with supports of different metadata engines, e.g., Redis, MySQL, SQLite and TiKV.

File storage technology

In Callaba, when you store a file, it undergoes a structured process. The file is divided into “Chunks,” each set at a fixed size, typically capped at 64 MiB. Each Chunk, in turn, is made up of one or more “Slices,” where the length of each Slice varies based on how the file is written. Within a Slice, you find size-fixed “Blocks,” defaulting to 4 MiB.

Ultimately, these Blocks are stored in Object Storage. Simultaneously, Callaba handles the metadata — information about the file, its Chunks, Slices, and Blocks. This metadata is stored in dedicated engines through Callaba. This meticulous organization ensures efficient and systematic storage of files, which can be particularly advantageous when dealing with video recordings or assets, where large file sizes and structured data are common.

Throughput

A sequential read/write benchmark has also been performed on Callaba, EFS and S3FS by fio.

Above result figure shows that Callaba can provide 10X more throughput than the other two.

Metadata IOPS

A simple mdtest benchmark has been performed on Callaba, EFS and S3FS by mdtest.

The result shows that Callaba can provide significantly more metadata IOPS than the other two.

Supported Object Storage

• Amazon S3 (and other S3 compatible Object Storage services)
• Google Cloud Storage
• Azure Blob Storage
• Alibaba Cloud Object Storage Service (OSS)
• Tencent Cloud Object Storage (COS)
• Qiniu Cloud Object Storage (Kodo)
• QingStor Object Storage
• Ceph RGW
• MinIO
• Local disk
• Redis
• and more

Summary

In summary, Callaba is a reliable solution for managing video files and data-intensive applications across diverse cloud setups. Its adaptability, automatic data replication, and meticulous file organization ensure efficiency. Seamless integration with massive cloud storage and streamlined metadata handling make it a cost-effective choice for those seeking a practical and scalable data management solution in modern cloud-native environments.