Deploy to production¶

This guide explains how to deploy InvenioRDM in a production environment.

Size your infrastructure¶

The following are baseline minimum requirements for an InvenioRDM production deployment. On their own, they correspond to a small instance. Adjust the resources based on your expected traffic, data volume, and performance requirements.

Lightweight services:

• HAProxy (if used at all): 0.1 CPU, 100 MB memory
• Nginx: 0.1 CPU, 100 MB memory
• Redis: 0.1 CPU, 250 MB memory (use RabbitMQ recommendations below if also used as queue)
• RabbitMQ (if Redis not used for queue as well): 0.5 CPU, 1 GB memory

Application nodes:

• Web nodes: 0.1 CPU, 300-400 MB memory per Python process
• Worker nodes: 1+ CPU, 300-400 MB memory per Python process

Data services:

• PostgreSQL: 1 CPU, 1-2 GB memory
• OpenSearch (master/data nodes): 1-2 CPUs, 2-4 GB memory

Storage:

• A few GB are sufficient for PostgreSQL, OpenSearch, Redis, and RabbitMQ
• For InvenioRDM application files, estimate based on your expected upload volume

See each service's official documentation for detailed requirements.

Deployment strategies¶

Choose a deployment strategy based on your needs, experience, and available resources. Adapt this reference guide to your specific requirements.

Docker Compose¶

The simplest deployment method uses Docker Compose with a configuration similar to the reference file in your InvenioRDM project.

Suitable for small instances:

• Thousands of records
• Tens of concurrent users

A single server with 4 vCPUs and 16 GB memory is typically sufficient if all services are co-located. If multiple servers are used (via docker-compose's network_mode: host for example), they can each be scaled down versions i.e., 1-2 vCPUs and 2-8 GB.

Key considerations:

• Mount volumes or use bind mounts for all services (database, search, cache, queue) to persist data across restarts
• For local file storage, backup your volumes or bind mount to prevent data loss
• Customize configuration by injecting environment variables (see Configure everything)
• A single-server deployment may become insufficient as records and users grow; scale up the server first, then consider horizontal scaling
• Review critical network security aspects to avoid exposing your infrastructure to the internet

Virtual Machines (Infrastructure as a Service - IaaS)¶

At CERN, Invenio v3 instances run on OpenStack. For InvenioRDM, you can either: - Install all services on a single large VM, or - Distribute services across multiple VMs (similar to Docker Compose distribution)

Suitable from small to large instances, scalable.

Kubernetes/OpenShift (Platform as a Service - PaaS)¶

For containerized deployments, at CERN we use OpenShift with the Invenio Helm Charts. These charts also work with Kubernetes clusters.

Suitable from small to large instances, scalable.

Checklist¶

Services¶

Start by installing and configuring all required services. Follow official documentation for each.

1. Database (PostgreSQL):

• Install and configure securely
• Create an unprivileged user with database creation and management permissions
• This user should only have database deletion rights if you need to use InvenioRDM wipe scripts
• Ensure remote connectivity from application servers
• Restrict external network access
• Implement backup and recovery strategies

2. Search engine (OpenSearch):

• Install and configure securely
• Set up a master node and at least one data node
• Restrict access to a dedicated user
• Ensure remote connectivity from application servers
• Restrict external network access
• Implement backup and recovery strategies at least for statistics
• Except for statistics, all important data in OpenSearch can be reindexed from the database
• Views and downloads statistics are only stored in OpenSearch (not the database)
• You can follow the instructions here to back them and/or other indices

3. Cache (Redis):

• Install and configure securely
• Ensure remote connectivity from application servers
• Restrict external network access

4. Message queue (RabbitMQ):

• Install and configure securely
• Enable queue persistence to prevent task loss on service failure
• Ensure remote connectivity from application servers
• Restrict external network access

InvenioRDM application¶

Install all requirements, then install the InvenioRDM Python application. You can spy the Dockerfile of the official InvenioRDM Docker images for what packages to install if not using those images.

Also look at the Compose file of the latest cookiecutter template to see the most up-to-date commands that the different services should run. The ones transcribed below in this documentation page may become out-of-date, but the ones at that link will always be the most up-to-date.

Configure the application:

1. Open the invenio.cfg file
2. Adjust configuration for production, including service connections
3. Test the web app and worker for errors:

$ invenio run --extra-files invenio.cfg
$ celery -A invenio_app.celery -l INFO

Build and lock assets:

Build assets first:

$ invenio assets build

If everything is working, it is recommended to lock Javascript assets

invenio-cli assets lock

and commit to your project repository the resulting:

• pnpm-lock.yaml and package.json if using pnpm
• package-lock.json and package.json if using npm

Then have your Dockerfile or deployment process use the lock file to install assets.

# Example if building assets in a Dockerfile
# For best layer-cache reuse, place above any layer
# that copies frequently-changing source.
# for pnpm
COPY package.json pnpm-lock.yaml ./
RUN pnpm install --frozen-lockfile --shamefully-hoist
# for npm
COPY package.json package-lock.json ./
RUN npm ci

Re-run invenio-cli assets lock whenever your JavaScript dependencies change.

Start the application server:

Review the application server configuration (uWSGI at time of writing) and run the server.

Start Celery schedulers:

$ celery -A invenio_app.celery beat -l INFO
$ celery -A invenio_app.celery beat -l INFO --scheduler invenio_jobs.services.scheduler:RunScheduler

Serve HTTP traffic¶

Install and configure Nginx reverse proxy:

• Serve static assets directly from the build directory
• Forward backend requests to the uWSGI server

Configure SSL termination:

• If using HAProxy as load balancer, configure HTTPS certificates and SSL termination on HAProxy
• Otherwise, configure SSL on Nginx

Expose your services:

• Expose only HAProxy or Nginx IPs
• Configure DNS to route traffic to your reverse proxy/load balancer

Security: Only the load balancer/reverse proxy should be internet-facing. All other infrastructure components must be isolated from external network access.

Pre-launch checklist¶

Before making your instance publicly available, verify everything works:

1. Open the website on your configured domain

   • Verify HTTPS is working properly with valid certificates
   • Confirm the homepage loads without errors

2. Create an admin user

   $ invenio users create admin@your-institution.org --password ... --active --confirm
   

   • Replace admin@your-institution.org with your admin email
   • The --active and --confirm flags activate and confirm the account immediately

3. Login with the admin user

   • Use the credentials you just created
   • Verify you can access the administration interface

4. Try to upload a new draft

   • Create a new draft record
   • Upload a test file
   • Verify the file uploads successfully
   • Confirm that the draft appears in your uploads

5. Check the Jobs in the admin panel and run them

   • Navigate to the admin panel's "Jobs" section
   • Verify background tasks are processing correctly
   • Run any pending jobs if needed

Versioning¶

As your project evolves, you will add custom features or upgrade InvenioRDM versions. Versioning helps control which code is deployed.

At CERN, we use GitHub tags/releases (e.g., v1.0.0) for version management.

Containerized deployments:

• Automate image builds using GitHub Actions or other CI tools
• Some PaaS platforms can auto-detect new images for specific tags (e.g., production) and trigger automatic redeployment

Monitoring¶

After deployment, comprehensive monitoring provides observability into your infrastructure.

Log aggregation:

• Configure services to output logs to a central location
• Aggregate logs for dashboard visualization using tools like:
• Loki
• OpenSearch Dashboards

System monitoring (IaaS):

• Aggregate system logs to monitor resource consumption
• Collect metrics with Prometheus
• Visualize with Grafana

Application-specific monitoring:

• Monitor Celery workers with Flower
• Many tools can integrate with Prometheus

For IaaS deployments, store logs on servers with log rotation regardless of your chosen monitoring tool.

Alerting¶

Once monitoring is configured, set up alerts for code exceptions and when metrics reach configured thresholds.

For the InvenioRDM Python application:

Many services exist that can catch and provide detailed exception information (variables, stack traces, etc.). Sentry is a popular one. Registering a Python logger to send emails with stacktraces is also a rugged possibility.

Security¶

To secure your instance:

• Correctly configure your InvenioRDM instance
• Use valid HTTPS certificates
• Keep your instance and all services up-to-date

Secure file uploads: Be extremely careful with user-uploaded content, as served files could contain malicious code. Effective protection methods:

• Whitelist MIME types
• Sanitize MIME types to prevent browser execution (e.g., convert HTML to plain text)
• Serve files from a separate domain using a static server without sessions

Load testing¶

Understand your system's load capacity to properly size infrastructure. Test with tools like:

• Locust
• k6

Troubleshooting¶

First step: Identify which layer or service has the problem using:

• Monitoring and dashboard tools (see above)
• Direct log file inspection

For InvenioRDM Python application issues:

• Get detailed error information from logs or alerting solutions
• Reproduce and debug in a local installation. See How to debug for more information.
• Checkout the troubleshooting documentation and reach out on Discord.