S3-as-a-Service is now in alpha!¶
We've been talking about it for a while in TUadmins and other places, and it's finally here. You can now order an S3 object storage bucket for storing large amounts of data and receive it within a few seconds.
Give it a try and let us know what you think!
S3-whatnow?¶
S3 is different from traditional file systems. Its main feature is also its main tradeoff: you have no consistency guarantees. With traditional file systems two programs can read and write the same file at the same time. This is not true for S3, as applications need to upload the entire file to create or update it. If two programs upload the same file, there are little to no guarantees which one will win.
However, this consistency model allows S3 to scale really well and gives us the option to use cheap disks for storage. Since the S3 protocol works over HTTPS, your applications can scale out and still access the data as well, unlike a traditional mounted storage volume.
How to order¶
As with all other Datafactory services, you can order this one through the OpenShift playground. Create a project and drop your request via a simple YAML file. The bucket will be created in a few seconds.
Some limitations¶
As this service is still in alpha, there are a few limitations in place:
- Maximum 1 TB of data and 100,000 files per bucket.
The S3 endpoint is only reachable from the Datalab network.The service is now fully reachable from the TUnet.- The underlying storage system is present only in one datacenter, there may be outages.
- No backups (yet)! If bad things™ happen, you may lose your data.
Of course, the hardware is already on its way to alleviate these limitations. Until then, please help us test this service.
Use cases¶
Here are a few ideas what you may want to store on S3:
- Temporary files for batch processing (video conversion, etc.)
- Large AI models
- Backups from your databases
As the platform matures, more use cases will become feasible.
Let us know what you think!