Using S3 buckets (experimental)¶
Experimental feature
This feature is experimental. The syntax and the service may change in the future.
Note
S3 object storage buckets are only reachable from the TUnet.
Datafactory offers S3 object storage for cheap, large scale data storage. A large number of applications supports S3 out of the box.
Requesting an S3 bucket¶
You can request an S3 bucket by using the following custom object:
apiVersion: tuwien.ac.at/v1alpha1
kind: S3BucketRequest
metadata:
name: mybucket
spec:
secretName: target-secret
Once the S3 bucket is created, the connection details will be exposed in the specified secret.
Warning
This resource will also create a Secret and a ConfigMap named datafactory-s3-<NAME>. These are for internal
purposes only. All details you need are in the named secret.
Differences between S3 and a file system¶
Although you can mount S3 as a filesystem in Kubernetes, you shouldn't. A filesystem offers consistency guarantees when writing, file locking and the ability to simultaneously read and write a file from multiple processes. S3 does not offer any of these guarantees and file writes are all-or-nothing, meaning that a partial file upload will be dropped with no changes made to the file.
Due to the lack of consistency guarantees S3 scales really well, which means that storage can be very cheap. S3 also offers the option to group objects (files) not only by filename, but also by tags, allowing you to implement custom logic around file handling.
Another feature of S3 is object versioning and object locking, which means you can protect your files even from yourself, aiding in tasks such as log retention or backups.
Integrating existing applications with S3¶
Should your existing application already support S3, you will need to provide the settings and credentials to the application. You can find these settings in the ConfigMap and Secret mentioned above.
Taking the AWS CLI as an example, you could create the following pod, mounting the Secret as environment variables:
---
apiVersion: v1
kind: Pod
metadata:
name: temp
spec:
containers:
- name: awscli
image: amazon/aws-cli
stdin: true
tty: true
resources:
requests:
memory: 128Mi
limits:
memory: 2Gi
securityContext:
seccompProfile:
type: RuntimeDefault
runAsNonRoot: true
allowPrivilegeEscalation: false
capabilities:
drop: ["ALL"]
command:
- /bin/sh
envFrom:
- secretRef:
name: target-secret
You can now run oc attach -ti temp and then run AWS commands, such as:
aws s3api list-buckets
Customizing the secret fields¶
By default, the created secret contains fields specific to the boto3 and
will work with most applications out of the box. However, for some applications you may want to customize the names
of the fields in the secret to better fit your application. You can do this by specifying the desired field names
in the fields section of your request:
apiVersion: tuwien.ac.at/v1alpha1
kind: S3BucketRequest
metadata:
name: test
namespace: project-test2
spec:
secretName: target-secret
fields:
accessKey: MY_ACCESS_KEY
The fields object has the following options:
| Field | Default | Notes |
|---|---|---|
accessKey |
AWS_ACCESS_KEY_ID |
|
secretKey |
AWS_SECRET_ACCESS_KEY |
|
region |
AWS_DEFAULT_REGION |
|
endpointUrl |
AWS_ENDPOINT_URL |
|
bucketName |
AWS_BUCKET |
Not a boto3 field, default name for consistency. |
host |
AWS_ENDPOINT_HOST |
Not a boto3 field, default name for consistency. |
port |
AWS_ENDPOINT_PORT |
Not a boto3 field, default name for consistency. |
s3_addressing_style |
AWS_S3_ADDRESSING_STYLE |
Can contain the auto, virtual or path values. |
Programming against the S3 API¶
When programming against the S3 API, you should always use the AWS SDK for your language unless you have a very specific reason not to do that. AWS is the originator of the S3 protocol and their SDK is the most complete. It also exists for a large number of languages.
For convenience, we are linking the SDK documentation for the more popular languages: