Zum Inhalt

Using S3 buckets (experimental)

Experimental feature

This feature is experimental. The syntax and the service may change in the future.

Note

S3 object storage buckets are only reachable from the TUnet.

Datafactory offers S3 object storage for cheap, large scale data storage. A large number of applications supports S3 out of the box.

Requesting an S3 bucket

You can request an S3 bucket by using the following custom object:

apiVersion: tuwien.ac.at/v1alpha1
kind: S3BucketRequest
metadata:
  name: mybucket
spec:
  secretName: target-secret

Once the S3 bucket is created, the connection details will be exposed in the specified secret.

Warning

This resource will also create a Secret and a ConfigMap named datafactory-s3-<NAME>. These are for internal purposes only. All details you need are in the named secret.

Differences between S3 and a file system

Although you can mount S3 as a filesystem in Kubernetes, you shouldn't. A filesystem offers consistency guarantees when writing, file locking and the ability to simultaneously read and write a file from multiple processes. S3 does not offer any of these guarantees and file writes are all-or-nothing, meaning that a partial file upload will be dropped with no changes made to the file.

Due to the lack of consistency guarantees S3 scales really well, which means that storage can be very cheap. S3 also offers the option to group objects (files) not only by filename, but also by tags, allowing you to implement custom logic around file handling.

Another feature of S3 is object versioning and object locking, which means you can protect your files even from yourself, aiding in tasks such as log retention or backups.

Integrating existing applications with S3

Should your existing application already support S3, you will need to provide the settings and credentials to the application. You can find these settings in the ConfigMap and Secret mentioned above.

Taking the AWS CLI as an example, you could create the following pod, mounting the Secret as environment variables:

---
apiVersion: v1
kind: Pod
metadata:
  name: temp
spec:
  containers:
    - name: awscli
      image: amazon/aws-cli
      stdin: true
      tty: true
      resources:
        requests:
          memory: 128Mi
        limits:
          memory: 2Gi
      securityContext:
        seccompProfile:
          type: RuntimeDefault
        runAsNonRoot: true
        allowPrivilegeEscalation: false
        capabilities:
          drop: ["ALL"]
      command:
        - /bin/sh
      envFrom:
        - secretRef:
            name: target-secret

You can now run oc attach -ti temp and then run AWS commands, such as:

aws s3api list-buckets

Customizing the secret fields

By default, the created secret contains fields specific to the boto3 and will work with most applications out of the box. However, for some applications you may want to customize the names of the fields in the secret to better fit your application. You can do this by specifying the desired field names in the fields section of your request:

apiVersion: tuwien.ac.at/v1alpha1
kind: S3BucketRequest
metadata:
  name: test
  namespace: project-test2
spec:
  secretName: target-secret
  fields:
    accessKey: MY_ACCESS_KEY

The fields object has the following options:

Field Default Notes
accessKey AWS_ACCESS_KEY_ID
secretKey AWS_SECRET_ACCESS_KEY
region AWS_DEFAULT_REGION
endpointUrl AWS_ENDPOINT_URL
bucketName AWS_BUCKET Not a boto3 field, default name for consistency.
host AWS_ENDPOINT_HOST Not a boto3 field, default name for consistency.
port AWS_ENDPOINT_PORT Not a boto3 field, default name for consistency.
s3_addressing_style AWS_S3_ADDRESSING_STYLE Can contain the auto, virtual or path values.

Programming against the S3 API

When programming against the S3 API, you should always use the AWS SDK for your language unless you have a very specific reason not to do that. AWS is the originator of the S3 protocol and their SDK is the most complete. It also exists for a large number of languages.

For convenience, we are linking the SDK documentation for the more popular languages: