package storage import ( "context" "io" ) // --- basics ---> // Storage is one of the base interfaces in the storage APIs. // This type is rarely seen by itself alone (and never useful to implement alone), // but is included in both ReadableStorage and WritableStorage. // Because it's included in both the of the other two useful base interfaces, // you can define functions that work on either one of them // by using this type to describe your function's parameters. // // Library functions that work with storage systems should take either // ReadableStorage, or WritableStorage, or Storage, as a parameter, // depending on whether the function deals with the reading of data, // or the writing of data, or may be found on either, respectively. // // An implementation of Storage may also support many other methods. // At the very least, it should also support one of either ReadableStorage or WritableStorage. // It may support even more interfaces beyond that for additional feature detection. // See the package-wide docs for more discussion of this design. // // The Storage interface does not include much of use in itself alone, // because ReadableStorage and WritableStorage are meant to be the most used types in declarations. // However, it does include the Has function, because that function is reasonable to require ubiquitously from all implementations, // and it serves as a reasonable marker to make sure the Storage interface is not trivially satisfied. type Storage interface { Has(ctx context.Context, key string) (bool, error) } // ReadableStorage is one of the base interfaces in the storage APIs; // a storage system should implement at minimum either this, or WritableStorage, // depending on whether it supports reading or writing. // (One type may also implement both.) // // ReadableStorage implementations must at minimum provide // a way to ask the store whether it contains a key, // and a way to ask it to return the value. // // Library functions that work with storage systems should take either // ReadableStorage, or WritableStorage, or Storage, as a parameter, // depending on whether the function deals with the reading of data, // or the writing of data, or may be found on either, respectively. // // An implementation of ReadableStorage may also support many other methods -- // for example, it may additionally match StreamingReadableStorage, or yet more interfaces. // Usually, you should not need to check for this yourself; instead, // you should use the storage package's functions to ask for the desired mode of interaction. // Those functions will will accept any ReadableStorage as an argument, // detect the additional interfaces automatically and use them if present, // or, fall back to synthesizing equivalent behaviors from the basics. // See the package-wide docs for more discussion of this design. type ReadableStorage interface { Storage Get(ctx context.Context, key string) ([]byte, error) } // WritableStorage is one of the base interfaces in the storage APIs; // a storage system should implement at minimum either this, or ReadableStorage, // depending on whether it supports reading or writing. // (One type may also implement both.) // // WritableStorage implementations must at minimum provide // a way to ask the store whether it contains a key, // and a way to put a value into storage indexed by some key. // // Library functions that work with storage systems should take either // ReadableStorage, or WritableStorage, or Storage, as a parameter, // depending on whether the function deals with the reading of data, // or the writing of data, or may be found on either, respectively. // // An implementation of WritableStorage may also support many other methods -- // for example, it may additionally match StreamingWritableStorage, or yet more interfaces. // Usually, you should not need to check for this yourself; instead, // you should use the storage package's functions to ask for the desired mode of interaction. // Those functions will will accept any WritableStorage as an argument, // detect the additional interfaces automatically and use them if present, // or, fall back to synthesizing equivalent behaviors from the basics. // See the package-wide docs for more discussion of this design. type WritableStorage interface { Storage Put(ctx context.Context, key string, content []byte) error } // --- streaming ---> type StreamingReadableStorage interface { GetStream(ctx context.Context, key string) (io.ReadCloser, error) } // StreamingWritableStorage is a feature-detection interface that advertises support for streaming writes. // It is normal for APIs to use WritableStorage in their exported API surface, // and then internally check if that value implements StreamingWritableStorage if they wish to use streaming operations. // // Streaming writes can be preferable to the all-in-one style of writing of WritableStorage.Put, // because with streaming writes, the high water mark for memory usage can be kept lower. // On the other hand, streaming writes can incur slightly higher allocation counts, // which may cause some performance overhead when handling many small writes in sequence. // // The PutStream function returns three parameters: an io.Writer (as you'd expect), another function, and an error. // The function returned is called a "WriteCommitter". // The final error value is as usual: it will contain an error value if the write could not be begun. // ("WriteCommitter" will be refered to as such throughout the docs, but we don't give it a named type -- // unfortunately, this is important, because we don't want to force implementers of storage systems to import this package just for a type name.) // // The WriteCommitter function should be called when you're done writing, // at which time you give it the key you want to commit the data as. // It will close and flush any streams, and commit the data to its final location under this key. // (If the io.Writer is also an io.WriteCloser, it is not necessary to call Close on it, // because using the WriteCommiter will do this for you.) // // Because these storage APIs are meant to work well for content-addressed systems, // the key argument is not provided at the start of the write -- it's provided at the end. // (This gives the opportunity to be computing a hash of the contents as they're written to the stream.) // // As a special case, giving a key of the zero string to the WriteCommiter will // instead close and remove any temp files, and store nothing. // An error may still be returned from the WriteCommitter if there is an error cleaning up // any temporary storage buffers that were created. // // Continuing to write to the io.Writer after calling the WriteCommitter function will result in errors. // Calling the WriteCommitter function more than once will result in errors. type StreamingWritableStorage interface { PutStream(ctx context.Context) (io.Writer, func(key string) error, error) } // --- other specializations ---> // VectorWritableStorage is an API for writing several slices of bytes at once into storage. // It's meant a feature-detection interface; not all storage implementations need to provide this feature. // This kind of API can be useful for maximizing performance in scenarios where // data is already loaded completely into memory, but scattered across several non-contiguous regions. type VectorWritableStorage interface { PutVec(ctx context.Context, key string, blobVec [][]byte) error } // PeekableStorage is a feature-detection interface which a storage implementation can use to advertise // the ability to look at a piece of data, and return it in shared memory. // The PeekableStorage.Peek method is essentially the same as ReadableStorage.Get -- // but by contrast, ReadableStorage is expected to return a safe copy. // PeekableStorage can be used when the caller knows they will not mutate the returned slice. // // An io.Closer is returned along with the byte slice. // The Close method on the Closer must be called when the caller is done with the byte slice; // otherwise, memory leaks may result. // (Implementers of this interface may be expecting to reuse the byte slice after Close is called.) // // Note that Peek does not imply that the caller can use the byte slice freely; // doing so may result in storage corruption or other undefined behavior. type PeekableStorage interface { Peek(ctx context.Context, key string) ([]byte, io.Closer, error) } // the following are all hypothetical additional future interfaces (in varying degress of speculativeness): // FUTURE: an EnumerableStorage API, that lets you list all keys present? // FUTURE: a cleanup API (for getting rid of tmp files that might've been left behind on rough shutdown)? // FUTURE: a sync-forcing API? // FUTURE: a delete API? sure. (just document carefully what its consistency model is -- i.e. basically none.) // (hunch: if you do want some sort of consistency model -- consider offering a whole family of methods that have some sort of generation or sequencing number on them.) // FUTURE: a force-overwrite API? (not useful for a content-address system. but maybe a gesture towards wider reusability is acceptable to have on offer.) // FUTURE: a size estimation API? (unclear if we need to standardize this, but we could. an offer, anyway.) // FUTURE: a GC API? (dubious -- doing it well probably crosses logical domains, and should not be tied down here.)