Object Subsystem (ObjectNode)

The object subsystem provides S3-compatible object storage interface. It makes possible for CubeFS becomes a fusion storage that expose two type interface (POSIX and S3-compatible). So that user can operate files stored in CubeFS by using native Amazon S3 SDKs.

Structure

../_images/cfs-object-subsystem-structure.png

The ObjectNode is a functional subsystem node. It fetch volume view (volume topology) on demand from Resource Manager (Master). Each ObjectNode communicate with metadata subsystem (MetaNode) and replica-subsystem (DataNode) directly.

ObjectNode is stateless design with high scalability and it have ability to operate all files stored in the CubeFS cluster which it belong to directly without any volume-mount operation. not support ec-volume temporarily

Features

  • Provides object storage interface compatible with native Amazon S3 SDKs.

  • Fusion storage expose two type interface (POSIX and S3-compatible).

  • Stateless and high scalability

Semantic Transform

Based on our POSIX-compatible design. Every file operate request comes from object storage interface need to be made semantic transform to POSIX.

POSIX

Object Storage

Cluster

Region

Volume

Bucket

Path

Key

Example:

../_images/cfs-object-subsystem-semantic.png

Put object ‘example/a/b.txt’ will be create and write data to file ‘/a/b.txt’ in volume ‘example’.

User

Before using the object storage service, you need to create a user through the Master. While creating users, AccessKey and SecretKey will be generated for each user.

CubeFS uses the field Owner of the volume as the user ID. There are two ways to create users:

  • When creating a volume through the API of the Master, if there is no user with the same name as the owner of the volume in the cluster, a user with the user ID of Owner will be automatically created.

  • Create a user by calling the user management API of the Master.

Authentication

The signature validation algorithm in object storage interface is fully compatible with native Amazon S3 service. The authentication consisting of AccessKey and SecretKey generated by Resource Manager (Master) with user creation, which can be obtained through the Master API. The AccessKey is a 16-character string unique in the entire CubeFS cluster.

The user has all access permissions to the volume owned by him. Users can grant other users specified permissions to access volumes under their own names. The permissions are divided into the following three categories:

  • Readonly or readwrite permission.

  • Permission for a single operation, such as GetObject, PutObject, etc.

  • Custom permission.

When a user uses the object storage service to execute a certain operation, CubeFS will identify whether the user has the corresponding permission.

Invisible Temporary Data

In order to make write operation in object storage interface atomically. Every write operation will create and write data to an invisible temporary. The volume operator in ObjectNode puts file data into temporary which only have ‘inode’ without ‘dentry’ in metadata. When all the file data stored successfully, the volume operator create or update ‘dentry’ in metadata makes it visible to users.

Object Mode Conflict (Important)

The POSIX and object storage are two different types of storage product, and the object storage is a Key-Value pair storage service. So that the object with key ‘a/b/c’ and the object with key ‘a/b’ are different object without any conflict.

But CubeFS is based on POSIX design. According to semantic transformation rule, the ‘b’ part in key ‘a/b/c’ transform to folder ‘b’ under the folder ‘a’ , and in key ‘a/b’ transform to file ‘b’ under the folder ‘a’.

The object key like this is conflict in CubeFS.

When the object being operated has conflicts with the existing object due to the above mode inconsistency, the ObjectNode will return a 409 status code to the client.

HTTP/1.1 409 Conflict
x-amz-request-id: 79e92f6c1a134a7895b60acb018f63cc
Date: Tue, 12 May 2020 07:32:31 GMT
Content-Type: application/xml
<Error>
    <Code>ObjectModeConflict</Code>
    <Message>Object already exists but file mode conflicts.</Message>
    <Resource>backup</Resource>
    <RequestID>79e92f6c1a134a7895b60acb018f63cc</RequestID>
</Error>

Supported S3 Features

  • File object operations.

  • Directory object operations.

  • Multipart upload.

  • Parallel download for high-level SDK APIs.

  • Tagging for bucket and object.

  • User-defined metadata for object.

  • IP address and network segment black and white list for bucket ACL.

  • Signature Algorithm V2 and V4.

  • Cross-Origin Resource Sharing (CORS).

Unsupported S3 Features

  • Version

  • Restore deleted objects

  • Locking objects

  • Lifecycle configuration for bucket and object.

  • Hosting Websites

  • Encryption

  • BitTorrent

Supported APIs

API

Reference

AbortMultipartUpload

https://docs.aws.amazon.com/AmazonS3/latest/API/API_AbortMultipartUpload.html

CompleteMultipartUpload

https://docs.aws.amazon.com/AmazonS3/latest/API/API_CompleteMultipartUpload.html

CopyObject

https://docs.aws.amazon.com/AmazonS3/latest/API/API_CopyObject.html

CreateBucket

https://docs.aws.amazon.com/AmazonS3/latest/API/API_CreateBucket.html

CreateMultipartUpload

https://docs.aws.amazon.com/AmazonS3/latest/API/API_CreateMultipartUpload.html

DeleteBucket

https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteBucket.html

DeleteBucketCors

https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteBucketCors.html

DeleteBucketPolicy

https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteBucketPolicy.html

DeleteBucketTagging

https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteBucketTagging.html

DeleteObject

https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObject.html

DeleteObjects

https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObjects.html

DeleteObjectTagging

https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObjectTagging.html

GetBucketAcl

https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetBucketAcl.html

GetBucketCors

https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetBucketCors.html

GetBucketLocation

https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetBucketLocation.html

GetBucketPolicy

https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetBucketPolicy.html

GetBucketTagging

https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetBucketTagging.html

GetObject

https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html

GetObjectAcl

https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObjectAcl.html

GetObjectTagging

https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObjectTagging.html

HeadBucket

https://docs.aws.amazon.com/AmazonS3/latest/API/API_HeadBucket.html

HeadObject

https://docs.aws.amazon.com/AmazonS3/latest/API/API_HeadObject.html

ListBuckets

https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListBuckets.html

ListMultipartUploads

https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListMultipartUploads.html

ListObjects

https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjects.html

ListObjectsV2

https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html

ListParts

https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListParts.html

PutBucketAcl

https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutBucketAcl.html

PutBucketCors

https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutBucketCors.html

PutBucketPolicy

https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutBucketPolicy.html

PutBucketTagging

https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutBucketTagging.html

PutObject

https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObject.html

PutObjectAcl

https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObjectAcl.html

PutObjectTagging

https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObjectTagging.html

UploadPart

https://docs.aws.amazon.com/AmazonS3/latest/API/API_UploadPart.html

Supported SDKs

Object Node provides S3-compatible object storage interface, so that you can operate files by using native Amazon S3 SDKs.

Name

Language

Link

AWS SDK for Java

Java

https://aws.amazon.com/sdk-for-java/

AWS SDK for JavaScript

JavaScript

https://aws.amazon.com/sdk-for-browser/

AWS SDK for JavaScript in Node.js

JavaScript

https://aws.amazon.com/sdk-for-node-js/

AWS SDK for Go

Go

https://docs.aws.amazon.com/sdk-for-go/

AWS SDK for PHP

PHP

https://aws.amazon.com/sdk-for-php/

AWS SDK for Ruby

Ruby

https://aws.amazon.com/sdk-for-ruby/

AWS SDK for .NET

.NET

https://aws.amazon.com/sdk-for-net/

AWS SDK for C++

C++

https://aws.amazon.com/sdk-for-cpp/

Boto3

Python

http://boto.cloudhackers.com