Add support for S3 storage providers

[mikeseese]

Longtail supports both Google Cloud Storage and S3 buckets, so I believe adding S3 support could open doors to more than one S3 storage provider (not just AWS, assuming DanEngelbrecht/golongtail#209 gets implemented). I personally use DigitalOcean as my cloud provider, so it'd be great to not be locked in to using Google also.

I've started working on an implementation in a fork

[Golle]

Hi,

Thanks for showing interest in Gamecure! This is as you stated in the other Issue an early alpha or an MVP for Content Creators at GOALS and the way we work here (plastic, single repo for engine+game, jira, GCS).
The plan from the start was to make it possible to extend Gamecure with other cloud providers, VCS etc. and it's something we definitely want to do.

Documentation is not there yet, but it's something that we keep improving.

Regarding S3 support I've got some news for you.
This library is currently in development and we plan to replace the current go-longtail client with it.
https://github.com/Golle/longtail

This lib will make it easy to create your own BlockStore implementations and support any cloud provider (or local storage) you have.

Here's a simplified example of how a Remote store could be implemented once we have have integrated this lib.

public class CustomRemoteBlockStore : IBlockStore
{
    private static readonly HttpClient Client = new();

    private readonly string _baseUri;
    private readonly string _fileExtension;

    public CDNRemoteBlockStore(string baseUri, string fileExtension)
    {
        _baseUri = baseUri;
        _fileExtension = fileExtension;
    }

    public Task PreflightGet(ReadOnlyMemory<ulong> blockHashes)
    {
        // Preload any hashes if needed
        return Task.CompletedTask;
    }

    public async Task<StoredBlock?> GetStoredBlock(ulong blockHash)
    {
        // Get the stored block from any storage (Azure, S3, GCS, network drive, local filesystem)
        var blockname = $"0x{blockHash:x16}";
        var result = await ReadBytesFromStore($"{_baseUri}/{blockname.Substring(2, 4)}/{blockname}{_fileExtension}");
        return result != null ? new StoredBlock(result) : null;
    }
    
    public async Task<StoreIndex?> GetExistingContent(ulong[] chunkHashes, uint minBlockUsagePercent)
    {
        // Read the store index file
        var result = await ReadBytesFromStore($"{_baseUri}/storage/store.lsi");
        return result != null ? new StoreIndex(result) : null;
    }

    private async Task<byte[]?> ReadBytesFromStore(string uri)
    {
        var result = await Client.SendAsync(new HttpRequestMessage(HttpMethod.Get, uri));
        if (result.IsSuccessStatusCode)
        {
            var bytes = await result.Content.ReadAsByteArrayAsync();
            Logger.Trace($"Read from CDN url: {uri} statusCode: {result.StatusCode} elapsed: {timer.Elapsed.TotalMilliseconds} ms Total bytes: {bytes.Length}");
            return bytes;
        }
        return null;
    }
}

It would be awesome if you could join our public discord https://discord.com/invite/goals and ping me and we can have a chat about your use case and how your team would benefit from using Gamecure.

[mikeseese]

Regarding S3 support I've got some news for you.
This library is currently in development and we plan to replace the current go-longtail client with it.
Golle/longtail

Cool I guess; the golongtail CLI tool already supports AWS S3 and is a minor revision away from supporting non-AWS S3. I think this move to using the C bindings directly and reimplement the cloud provider side of thing is the right path, but it's not the shortest path for S3 🤷

Anyway, thanks for the response and diligent effort. I'm not really a fan of the opinionated structure and tight coupling the structure the app has and would require a bit of a overhaul to rethink interfaces to be more open to be flexible for other use cases. C# isn't my forte, so rapid prototyping just isn't as quick for me as NodeJS and the npm dependency ecosystem. As a typical software engineer, it's tempting for me to just start my own project from scratch 😅 I definitely don't like reinventing the wheel and spending time writing something from scratch however