S3 compatible storage backend¶

InvenioRDM provides seamless integration with S3-compatible object storage systems. This allows institutions to leverage scalable, cost-effective, and highly durable cloud storage solutions for their research data, ensuring secure and efficient management of digital assets.

S3 via RustFS¶

RustFS is an open-source, S3-compatible high-performance object storage system supporting migration and coexistence with other S3-compatible platforms such as Ceph.

After having setup your RustFS application, you can review the settings below.

Bucket and data location¶

A default bucket (default) is created automatically by invenio-cli in your RustFS container. This bucket will be used as the default files' location for your InvenioRDM instance if you choose s3 as file storage during project initialization.

If you want to change the bucket name, open a shell window and type invenio files location s3-default s3://new-bucket --default (preceded by uv run or pipenv run as appropriate). You will also need to manually create the new bucket in RustFS or your S3 storage system. The new bucket will only be used for new records or drafts (existing drafts and records will use default).

Credentials¶

The last step is to change the default credentials to improve the security of your instance. By default, the RUSTFS_ACCESS_KEY and the RUSTFS_SECRET_KEY are "CHANGE_ME". To change these to something more secure you will need to:

• Stop your services by using the CLI: invenio-cli services stop (Or stop only your RustFS container).
• In the docker-services.yaml file change the RUSTFS_ACCESS_KEY and RUSTFS_SECRET_KEY values.
• Update the credentials:
  • You should be using environment variables to inject secrets. Define INVENIO_S3_ACCESS_KEY_ID for the user and INVENIO_S3_SECRET_ACCESS_KEY for the password.
  • If you are developing locally, you can change your invenio.cfg and set the value of S3_ACCESS_KEY_ID to the desired username and the value of S3_SECRET_ACCESS_KEY to the password.
• Start your services: invenio-cli services start.

It's done! Ready to roll!

Amazon S3¶

If you're using Amazon S3, you'll want to provide the endpoint URL and set the region (if you're not using the default of us-east-1):

S3_ENDPOINT_URL='https://s3.us-west-2.amazonaws.com/'
S3_REGION_NAME='us-west-2'

To enable file previews in InvenioRDM when using Amazon S3, you must configure CORS (Cross-Origin Resource Sharing) for your S3 bucket. In the AWS console, navigate to your bucket's permissions tab and add:

[
    {
        "AllowedHeaders": [],
        "AllowedMethods": [
            "GET"
        ],
        "AllowedOrigins": [
            "*"
        ],
        "ExposeHeaders": []
    }
]

## Non-Amazon S3 Providers

If you're using a Non-Amazon S3 provider, you may have to configure the checksum behaviour with: