Skip to main content

Specs

specs is core of our services:

  • fields.toml defines all fields that we can use in our APIs.
  • operations.toml defines our APIs.
  • pairs.toml defines all global pairs that we support.
  • info_object_meta.toml defines all object metadata.
  • info_storage_meta.toml defines all storage metadata.
  • service.toml defines a service's name, ability and limitations.

go-storage provides a code generator definitions to generate code using specs.

Operation#

definitions uses fields.toml and operations.toml to generate our public APIs.

[storager.op.read]description = "will read the file's data."params      = ["path", "w"]pairs       = ["size", "offset", "io_callback"]results     = ["n"]

will become

type Storager interface {    ...        // Read will read the file's data.    Read(path string, w io.Writer, pairs ...Pair) (n int64, err error)    // ReadWithContext will read the file's data.    ReadWithContext(ctx context.Context, path string, w io.Writer, pairs ...Pair) (n int64, err error)        ...        mustEmbedUnimplementedStorager()}

All supported fields are following:

  • description: This operation's description.
  • params: This operation's params
  • results: This operation's returning value
  • pairs: Required pair for this operation, all service should support.
  • local: Mark this operation as a local function, definitions will not add context and err for this function.
  • object_mode: Object mode check for this operation (Introduced in GSP-61)

Pair#

Pair is the input argument for operations:

func Read(path string, w io.Writer, pairs ...Pair) (n int64, err error) {}

go-storage provides pairs.WithXxx to support strong typed pair:

func WithContentType(v string) Pair {    return Pair{        Key:   "content_type",        Value: v,    }}

We have two kind of pairs:

  • Global Pair: defined in specs, could be used in all services.
  • System Pair: defined in service.toml, could only be used in current service.

All supported fields are following:

  • type: corresponding type of this pair
  • description: description for this pair, supports multiple lines.

Example of adding global pair:

[size]type = "int64"description = "specify size for this request, storage will only read limited content data"

Example of adding system pair:

[pairs.force_path_style]type = "bool"description = "see http://docs.aws.amazon.com/AmazonS3/latest/dev/VirtualHosting.html for Amazon S3: Virtual Hosting of Buckets"

Info#

Info is the returning information from services:

  • Storage related info carried in StorageMeta
  • Object related info carried in Object

go-storage provides GetXxx and MustGetXxxx to support strong typed info:

func (o *Object) GetContentLength() (int64, bool) {}
func (o *Object) MustGetContentLength() int64 {}

We have two kind of info:

  • Global Info: defined in specs, could be used in all services.
  • System Info: defined in service.toml, could only be used in current service.

All supported fields are following:

  • type: corresponding type of this pair.
  • export: whether this info should be exported.
  • description: description for this pair, supports multiple lines.

Example of adding global info for storage metadata:

[name]type = "string"export = true

Example of adding system pair for object metadata:

[infos.object.meta.storage-class]type = "string"

Service#

definitions will use service.toml to generate service code. The full example of service.toml looks like following:

name = "s3"
[namespace.service.new]required = ["credential"]
[namespace.service.op.create]required = ["location"]
[namespace.storage]implement = ["multiparter"]
[namespace.storage.new]required = ["location", "name"]
[namespace.storage.op.read]optional = ["offset", "io_callback", "size"]
[pairs.force_path_style]type = "bool"description = "see http://docs.aws.amazon.com/AmazonS3/latest/dev/VirtualHosting.html for Amazon S3: Virtual Hosting of Buckets"
[infos.object.meta.storage-class]type = "string"

name#

name = "s3"

The name of current service, should be unique.

namespace#

go-storage has two namespace: service and storage.

namespace COULD implement extra interfaces:

[namespace.storage]implement = ["multiparter"]

We should also add related UnimplementedMultiparter stub into our Storage struct.

namespace MUST add new section to describe how to initiate this namespace:

[namespace.service.new]required = ["credential"]optional = ["endpoint"]

This section contains two fields:

  • required: Required pairs for initialization
  • optional: Optional pairs for initialization (could have default value)

namespace MAY declare pairs for specific operations:

[namespace.storage.op.read]optional = ["offset", "io_callback", "size"]

This section contains two fields:

  • required: Required pairs for initialization
  • optional: Optional pairs for initialization (could have default value)
info

Operations could have required pairs, all service should support.

For example, Read operation has ["size", "offset", "io_callback"], all service should support those pairs, or their need to add convert functions:

func (s *Storage) convertWriteStorageClass(v string) (string, bool) {    return "", true}

pairs#

[pairs.force_path_style]type = "bool"description = "see http://docs.aws.amazon.com/AmazonS3/latest/dev/VirtualHosting.html for Amazon S3: Virtual Hosting of Buckets"

System pairs, name should be snack_case

infos#

[infos.object.meta.storage-class]type = "string"

System infos, name should be kebab-case