# Using S3 buckets (experimental)

!!! warning "Experimental feature"
    This feature is experimental. The syntax and the service may change in the future.

!!! note
    S3 object storage buckets are only reachable from the TUnet.

Datafactory offers S3 object storage for cheap, large scale data storage. A large number of applications
supports S3 out of the box.

## Requesting an S3 bucket

You can request an S3 bucket by using the following custom object:

```yaml
apiVersion: tuwien.ac.at/v1alpha1
kind: S3BucketRequest
metadata:
  name: mybucket
spec:
  secretName: target-secret
```

Once the S3 bucket is created, the connection details will be exposed in the specified secret.

!!! warning
    This resource will also create a Secret and a ConfigMap named `datafactory-s3-<NAME>`. These are for internal
    purposes only. All details you need are in the named secret.

## Differences between S3 and a file system

Although you *can* mount S3 as a filesystem in Kubernetes, you shouldn't. A filesystem offers consistency
guarantees when writing, file locking and the ability to simultaneously read and write a file from multiple
processes. S3 does not offer any of these guarantees and file writes are all-or-nothing, meaning that a partial
file upload will be dropped with no changes made to the file.

Due to the lack of consistency guarantees S3 scales really well, which means that storage can be very cheap.
S3 also offers the option to group objects (files) not only by filename, but also by tags, allowing you to
implement custom logic around file handling.

Another feature of S3 is object versioning and object locking, which means you can protect your files even
from yourself, aiding in tasks such as log retention or backups.

## Integrating existing applications with S3

Should your existing application already support S3, you will need to provide the settings and credentials
to the application. You can find these settings in the ConfigMap and Secret mentioned above.

Taking the AWS CLI as an example, you could create the following pod, mounting the Secret as
environment variables:

```yaml
---
apiVersion: v1
kind: Pod
metadata:
  name: temp
spec:
  containers:
    - name: awscli
      image: amazon/aws-cli
      stdin: true
      tty: true
      resources:
        requests:
          memory: 128Mi
        limits:
          memory: 2Gi
      securityContext:
        seccompProfile:
          type: RuntimeDefault
        runAsNonRoot: true
        allowPrivilegeEscalation: false
        capabilities:
          drop: ["ALL"]
      command:
        - /bin/sh
      envFrom:
        - secretRef:
            name: target-secret
```

You can now run `oc attach -ti temp` and then run AWS commands, such as:

```
aws s3api list-buckets
```

## Customizing the secret fields

By default, the created secret contains fields specific to the [boto3](https://docs.aws.amazon.com/boto3/latest/) and
will work with most applications out of the box. However, for some applications you may want to customize the names
of the fields in the secret to better fit your application. You can do this by specifying the desired field names
in the `fields` section of your request:

```yaml
apiVersion: tuwien.ac.at/v1alpha1
kind: S3BucketRequest
metadata:
  name: test
  namespace: project-test2
spec:
  secretName: target-secret
  fields:
    accessKey: MY_ACCESS_KEY
```

The `fields` object has the following options:

| Field                 | Default                   | Notes                                               |
|-----------------------|---------------------------|-----------------------------------------------------|
| `accessKey`           | `AWS_ACCESS_KEY_ID`       |                                                     |
| `secretKey`           | `AWS_SECRET_ACCESS_KEY`   |                                                     |
| `region`              | `AWS_DEFAULT_REGION`      |                                                     |
| `endpointUrl`         | `AWS_ENDPOINT_URL`        |                                                     |
| `bucketName`          | `AWS_BUCKET`              | Not a boto3 field, default name for consistency.    |
| `host`                | `AWS_ENDPOINT_HOST`       | Not a boto3 field, default name for consistency.    |
| `port`                | `AWS_ENDPOINT_PORT`       | Not a boto3 field, default name for consistency.    |
| `s3_addressing_style` | `AWS_S3_ADDRESSING_STYLE` | Can contain the `auto`, `virtual` or `path` values. |

## Programming against the S3 API

When programming against the S3 API, you should always use the AWS SDK for your language unless you have a very
specific reason not to do that. AWS is the originator of the S3 protocol and their SDK is the most complete. It also
exists for a large number of languages.

For convenience, we are linking the SDK documentation for the more popular languages:

- [JavaScript/Typescript](https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/welcome.html)
- [Java](https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/home.html)
- [Python (boto3)](https://docs.aws.amazon.com/boto3/latest/guide/quickstart.html)
- [Go](https://docs.aws.amazon.com/sdk-for-go/v2/developer-guide/welcome.html)
- [Ruby](https://docs.aws.amazon.com/sdk-for-ruby/v3/developer-guide/welcome.html)