Skip to content

Upgrade from v13 to v14

Background and prerequisites

This article details the low-level steps to follow to upgrade your InvenioRDM v13 instance to v14.0.

Version 14 introduces a number of default tooling changes (pipenv -> uv, npm -> pnpm, ...) and requires using Python 3.14. In particular, this is the last version where we document usage of pipenv. Future releases will assume usage of uv in all documentation. Although using pipenv will still work for this version, it will not be actively supported in the next version (at that point it may or may not work).

As usual, these steps do assume an existing installation of InvenioRDM v13, the previous version. If your InvenioRDM installation is older than v13, you must first upgrade to v13 before proceeding with the steps in this guide. However, it doesn't assume you are necessarily on v13.1. The instructions will work whether you are on v13.0 or v13.1.

The throughline of this document is a sequential series of steps to execute. We highly recommend running the steps in a local development environment first where experience with the particularities of your instance can be gained without data loss worry. Then we recommend you run the steps into a staging environment mirroring your production deployment and accrue further insight into specificities of your environment (or missing details in these update steps!). Equipped with that knowledge, running the steps on your production environment should be smooth.

As always, reach out on Discord if you need help! There are more details in this one, so don't hesitate.

Backup

Always backup your database, statistics indices and files before you try to perform an upgrade.

If your deposited files are stored in <venv folder>/var/instance/data, you should back them up outside this location before you continue. This command checks the location of your deposited files:

invenio files location list

This is usually only the case in development, as a containerized production deployment will store the deposited files elsewhere.

Switch to Python 3.14

Required for upgrade: Yes. Python 3.14 is now the required Python version for InvenioRDM v14. Support for lower versions is not maintained moving forward. Maybe they work, but there is no guarantee.

If you need to switch, here are the steps:

  1. Install Python 3.14 locally

    uv python install 3.14

    # With the pyenv tool (https://github.com/pyenv/pyenv)
pyenv install 3.14

  2. Change the required Python version in pyproject.toml if using uv or Pipfile if still using pipenv.

    # Set this line
requires-python = "~=3.14.0"

    [requires]
# Set this line
python_version = "3.14"

  3. Change the FROM line in your Dockerfile to a base image using Python 3.14

    # This image also provides all other libraries & tools for InvenioRDM v14
FROM ghcr.io/inveniosoftware/invenio:14-debian

    At time of writing the image above is the only official one, but you may use your own. As long as your image provides Python 3.14 (and the tools you want).

Upgrade to InvenioRDM v14 proper

Required for upgrade: Yes! This is the main upgrade section.

Here are the core sequential steps to upgrade to InvenioRDM v14.

Virtual environments

All commands below assume you are running them according to their installation environments. Typically it means invenio commands should be executed:

  • inside the application's virtual environment OR
  • via pipenv run or uv run in case you are not inside a virtual environment OR
  • environment with executables installed globally

Check your database for the alembic_version table

In order to proceed with the database migration later on, the alembic_version table must exist in your database. Run the following to ensure it exists:

wget -nv https://raw.githubusercontent.com/inveniosoftware/docs-invenio-rdm/master/docs/releases/v14/ensure_alembic_version_table_exists.py -O /tmp/ensure_alembic_version_table_exists.py
invenio shell /tmp/ensure_alembic_version_table_exists.py

Why was this necessary

InvenioRDM v13 contained a race condition that could have prevented the alembic_version table from being created. Alembic uses this table to track database schema migrations, and you cannot upgrade to v14 without it.

If you encounter issues, please ask for help on our Discord server.

Upgrade invenio-cli

Make sure you have the latest invenio-cli installed. For InvenioRDM v14, it should be v1.12.0 or higher. See Install the command line tool for how to install it.

$ invenio-cli --version
invenio-cli, version 1.12.0

Upgrade packages

Change the version of invenio-app-rdm to 14.0 in <my-site>/pyproject.toml if using uv or <my-site>/Pipfile if using pipenv:

dependencies = [
-    "invenio-app-rdm[opensearch2]~=13.0.0",
+    "invenio-app-rdm[opensearch2]~=14.0.0",

[packages]
-invenio-app-rdm = {extras = [opensearch2], version = "~=13.0.0"}
+invenio-app-rdm = {extras = [opensearch2], version = "~=14.0.0"}

Install InvenioRDM v14:

invenio-cli install

invenio-cli install reuses the existing virtualenv and Pipfile.lock, so recreate both:

pipenv --rm
invenio-cli packages lock
invenio-cli install

Update database schemas and content

Run the pre-migration step

Run one of the following commands, depending on whether your installation uses uv or pipenv:

invenio shell $(find $(dirname $(dirname $(uv python find)))/lib/*/site-packages/invenio_app_rdm -name prepare_migration_13_0_to_14_0.py)

invenio shell $(find $(pipenv --venv)/lib/*/site-packages/invenio_app_rdm -name prepare_migration_13_0_to_14_0.py)

If it exits with an error, please contact us on Discord, and we will help you find an individual solution for the issue.

Run the schema migration step

Run the Alembic migration to update the database schema:

invenio alembic upgrade

If this exits with errors, please have a look to the Troubleshooting section and if you don't find a solution there contact us on Discord.

Run the content migration

Execute the data migration script to update the content of the DB:

invenio shell $(find $(dirname $(dirname $(uv python find)))/lib/*/site-packages/invenio_app_rdm -name migrate_13_0_to_14_0.py)

invenio shell $(find $(pipenv --venv)/lib/*/site-packages/invenio_app_rdm -name migrate_13_0_to_14_0.py)

Update search engine mappings and content

# precede by uv run or pipenv run as appropriate
invenio index destroy --yes-i-know
invenio index init
# if you have records custom fields
invenio rdm-records custom-fields init
# if you have communities custom fields
invenio communities custom-fields init
invenio rdm rebuild-all-indices

Update OAI-PMH percolator mapping and Job Logs Index

Percolators and job datastreams get their new mapping from the invenio index init step, but are not affected by the index rebuild from the step before. They have to be updated by running the following script.

wget -nv https://raw.githubusercontent.com/inveniosoftware/docs-invenio-rdm/master/docs/releases/v14/migrate_percolator_and_jobs_datastream.py -O /tmp/migrate_percolator_and_jobs_datastream.py
invenio shell /tmp/migrate_percolator_and_jobs_datastream.py

Update vocabularies

The following out-of-the-box vocabularies have been enhanced with terms from the DataCite 4.4-4.7 releases and/or mapping improvements. They all benefited from added translations:

  • datetypes (new entry, translations)
  • descriptiontypes (translations)
  • licenses (new entry, translations)
  • relationtypes (new entries, translations)
  • resourcetypes (new entries, mapping improvements, translations)
  • contributorsroles (new entry, translations)
  • creatorsroles (new entry, translations)
  • titletypes (translations)
  • removalreasons (translations)

In order to update these in your repository, you'll need for each vocabulary to:

  1. Assess if you've customized the vocabulary for your instance. Check in your instance's app_data/vocabularies/ directory if you have a corresponding customized vocabulary file.

  2. If you've customized the vocabulary for your instance, you will need to merge changes from the source files in invenio-rdm-records into the custom vocabulary file in your instance according to what you and/or your stakeholders think makes sense in your context. If you have not customized the vocabulary, you are probably fine with adopting the changes, but you can always double-check what those are and decide if you adopt them.

  3. If you've decided to adopt the changes (and have merged the changes per step 2.), run the vocabulary update command: invenio rdm-records add-to-fixture <vocabulary fixture>. For example: invenio rdm-records add-to-fixture datetypes.

If you plan on adopting all those, you can run all the add-to-fixture commands:

invenio rdm-records add-to-fixture datetypes
invenio rdm-records add-to-fixture descriptiontypes
invenio rdm-records add-to-fixture licenses
invenio rdm-records add-to-fixture relationtypes
invenio rdm-records add-to-fixture resourcetypes
invenio rdm-records add-to-fixture contributorsroles
invenio rdm-records add-to-fixture creatorsroles
invenio rdm-records add-to-fixture titletypes
invenio rdm-records add-to-fixture removalreasons

Info

The resourcetypes vocabulary was also subject to another cleanup operation upstream that removed an entry from it. The commands above only add or update entries. Steps to follow to replicate this removal in your instance can be found in the section about aligning "thesis" and "dissertation" resource types below.

Update your configuration or infrastructure

Required for upgrade: Assess on a case by case basis. Typically optional.

This last section highlights the changes to your configuration or infrastructure that you should assess. Determine if each applies to your instance, and perform the appropriate changes.

Cache backend configuration

Required for upgrade: Yes, if your instance explicitly sets CACHE_TYPE.

Flask-Caching 2.5.0 removed the deprecated legacy backend names such as redis, simple, and memcached. If your existing configuration still uses one of those names, update it to the corresponding backend class name before running the upgraded instance or its test suite.

For example, in docker-services.yml:

- INVENIO_CACHE_TYPE=redis
+ INVENIO_CACHE_TYPE=RedisCache

If you set the value directly in invenio.cfg, update it there instead:

- CACHE_TYPE = "redis"
+ CACHE_TYPE = "RedisCache"

Common replacements are redis -> RedisCache, simple -> SimpleCache, and memcached -> MemcachedCache. Also check any CI/test-specific environment files, because stale values there can cause unit tests to fail even when the running service configuration has already been updated.

Switch tools

Switch from pipenv to uv

Please have a look at the dedicated Switch from pipenv to uv page.

Switch from npm to pnpm

The new JavaScript dependencies manager pnpm makes installations faster and more secure.

Please have a look at the dedicated Switch from npm to pnpm page.

Switch from webpack to rspack

The new assets builder drastically reduces installations time.

Please have a look at the dedicated Switch from webpack to rspack page.

invenio-cli run --host ... --port ... / invenio-cli containers start

invenio-cli run no longer overrides SITE_API_URL and SITE_UI_URL. Passing --host and --port only defines the port and host the development server is listening on. As such, you should change the following lines in invenio.cfg for your non-containerized development environment:

# in invenio.cfg
-SITE_UI_URL = "https://127.0.0.1"
+SITE_UI_URL = "https://127.0.0.1:5000"

-SITE_API_URL = "https://127.0.0.1/api"
+SITE_API_URL = "https://127.0.0.1:5000/api"

Consequently, to match your non-local setup, set INVENIO_SITE_UI_URL and INVENIO_SITE_API_URL in docker-services.yml (for app) or your custom environment setup.

Overridable IDs in the deposit form

If you are not overriding any of these components, you do not need to change anything, else you will need to change the ID(s) in your mapping file to reflect these modifications.

The full list of ID changes can be found here.

Custom field widget prop names

Many custom field widgets used the icon and description props, which have now been deprecated and replaced with labelIcon and helpText respectively.

If you have developed React components in your instance, or overridden existing UI components, look for any declaration of icon or description and change them to:

import { parametrize } from "react-overridable"
import { TitlesField } from "@js/invenio_rdm_records"

export const overriddenComponents = {
  "InvenioRdmRecords.DepositForm.TitlesField": parametrize(
    TitlesField,
    {
        ...
-      description: "Describe your resource in a few words",
+      helpText: "Describe your resource in a few words",
        ...
-      icon: "barcode",
+      labelIcon: "barcode",
        ...
    }
  )
}

The old names are deprecated and will be removed in a future release. Please update to the new names.

Deprecated GitHub integration

The invenio-github module has been deprecated and its support will be removed with InvenioRDM v15. Please use invenio-vcs instead.

Please see this detailed guide for more information on how to upgrade. This is only necessary if your instance was actively using invenio-github (with at least one user having connected their GitHub account) and you want to keep the existing data. See also the documentation on how to configure the new module.

Align "Thesis" and "Dissertation" resource types

Required for upgrade: No.

Your upgrade is complete. This last section describes an entirely optional change that is not part of the upgrade. Nothing here affects your instance unless you choose to run it, and you can do so at any later time. Because resource types are a highly visible and commonly customized vocabulary, we suggest rather than impose this change. Decide together with your instance's stakeholders (librarians, curators) whether it fits your data before applying it.

What changed and why

With InvenioRDM v14, the default resource types "Publication / Thesis" (id publication-thesis) and "Publication / Dissertation" (id publication-dissertation) were merged into one: "Publication / Thesis" (id publication-dissertation). This resource type maps to Datacite's resourceTypeGeneral "Dissertation" (datacite_general: Dissertation in the InvenioRDM's default resource type YAML file). The separate publication-thesis entry is dropped.

InvenioRDM interprets Datacite's Dissertation as covering both former entries, so a single entry mapping to "Dissertation" was more accurate. Datacite's resourceTypeGeneral "Text" is not as precise in this context. Staying close to the DataCite schema as a default is a core goal of InvenioRDM.

If you deliberately want to keep both types (for example publication-dissertation for PhD work and publication-thesis for the rest), or you have customized their DataCite mappings, you may prefer to keep your current setup or apply only part of this change. Skipping this section does not affect your InvenioRDM installation.

Applying the change

If you decide to go ahead, follow whichever of the two options below matches your instance.

Both rely on the migrate_thesis_to_dissertation.py helper. This is a set of reference functions, not a ready-to-run tool: read it, and copy or adapt the parts that fit your case. One convenient way to use it as-is is to download it and load its functions into an invenio shell:

wget -nv https://raw.githubusercontent.com/inveniosoftware/docs-invenio-rdm/master/docs/releases/v14/migrate_thesis_to_dissertation.py -O /tmp/migrate_thesis_to_dissertation.py
invenio shell -i -- /tmp/migrate_thesis_to_dissertation.py

Always test against a copy of your data first.

Option A: adopt the new default publication-dissertation

Use this if you want to switch your publication-thesis records over to the new default publication-dissertation resource type, id and all. It rewrites your records and drafts, re-registers their DataCite DOIs, and re-indexes them.

  1. If you are using a customized list of resource types in <my_instance>/app_data/vocabularies/resource_types.yaml, then:

    1. Set title.<lang> to "Thesis" (in the appropriate language) for the entry with id equal or equivalent to publication-dissertation.
    2. Remove the entry with id equal or equivalent to publication-thesis.

  2. Apply the resource types change:

    • invenio rdm-records add-to-fixture resourcetypes
    • Note that this changes the vocabulary, but does not delete publication-thesis from your data stores. Deletion is done in step 4.

  3. Rewrite every existing record and draft from publication-thesis to publication-dissertation. The run_update_for_resource_type function rewrites the resource type (including in related identifiers), re-registers DataCite DOI metadata, and re-indexes the affected records:

    # in `invenio shell`, with the helper loaded (see above)
run_update_for_resource_type()

  4. Delete the publication-thesis vocabulary entry via invenio shell:

    from invenio_vocabularies.proxies import current_service as vocabulary_service
vocabulary_service.delete(system_identity, ('resourcetypes', 'publication-thesis'))
Option B: keep your resource type, map it to DataCite's "Dissertation"

Use this if you want to keep your own resource type (for example your existing publication-thesis, or any custom type) but have its DataCite DOIs use Dissertation as a value instead of Text. Your records are not changed, only the DataCite metadata of their DOIs is updated.

  1. In <my_instance>/app_data/vocabularies/resource_types.yaml, set props.datacite_general to Dissertation for your resource type entry (and adjust props.datacite_type if you set one).

  2. Apply the vocabulary change:

    • invenio rdm-records add-to-fixture resourcetypes

  3. Re-register the DataCite DOI metadata of every published record of that resource type, so DataCite reflects the new value. The run_update_doi_metadata_for_resource_type function does this without changing the records (replace publication-thesis with your resource type id):

    # in `invenio shell`, with the helper loaded (see above)
run_update_doi_metadata_for_resource_type("publication-thesis")

Troubleshooting

uv cache permission errors

If a uv command fails with a permission error for its cache directory, the configured cache may not be writable by the user running the command. For example:

$ uv python find
error: Failed to initialize cache at `/.cache/uv`
  Caused by: failed to create directory `/.cache/uv`: Permission denied (os error 13)

Set a writable temporary cache directory and retry the command:

export UV_CACHE_DIR=/tmp/uv-cache

Invenio alembic upgrade could cause problems

Unique constraint violation errors can be solved with database cleanups

In some rare cases, instances have been observed to run into unique constraint violations (e.g. sqlalchemy.exc.IntegrityError: (psycopg2.errors.UniqueViolation) ...) during the Alembic upgrade. This can happen due to leftover artifacts that were never properly cleaned up, likely due to old bugs. Removing the offending leftover rows from the database will resolve these errors.

The necessary steps depend on the concrete situation, but generally what's needed in such cases is to clean up the tables mentioned in the error message. For example, the following error would tell you that you need to delete old entries from the rdm_records_files table, and which (record_id, key) combination to look out for.

The following is a real-life example with altered identifiers: 

sqlalchemy.exc.IntegrityError: (psycopg2.errors.UniqueViolation) could not create unique index "uidx_rdm_records_files_record_id_key"
DETAIL:  Key (record_id, key)=(imagine-this-were-a-valid-uuid, research_NMR.zip) is duplicated.

[SQL: CREATE UNIQUE INDEX IF NOT EXISTS uidx_rdm_records_files_record_id_key ON rdm_records_files (record_id, key)]

In this example, we can see that there are two entries for the reported record_id, and the older one has had its object_version_id set to null: 

inveniordm=> SELECT * FROM rdm_records_files WHERE record_id = 'imagine-this-were-a-valid-uuid';
          created           |          updated           |                  id                  | json | version_id |       key        |              record_id               |          object_version_id
----------------------------+----------------------------+--------------------------------------+------+------------+------------------+--------------------------------------+--------------------------------------
 2024-12-03 18:09:12.125274 | 2024-12-03 18:09:12.125283 | 20a5a54a-5c6a-400d-aa3e-5db99323768c | {}   |          1 | research_NMR.zip | imagine-this-were-a-valid-uuid       |
 2024-12-03 18:09:33.503182 | 2024-12-03 18:09:33.546971 | 17235935-1e95-4529-b189-82a250113e93 | {}   |          3 | research_NMR.zip | imagine-this-were-a-valid-uuid       | 2e218ef5-ba92-4fa4-a58a-0c204fc771a1
(2 rows)

The duplication was resolved with DELETE FROM rdm_records_files WHERE record_id = 'imagine-this-were-a-valid-uuid' AND object_version_id IS null.

After that cleanup, invenio alembic upgrade went through successfully.

⚠️ Be careful to only clean up the leftover data, though! If you are unsure which entries are the leftovers, feel free to ask for help in the Discord server.

Issues due to Role.id not equal to Role.name

If you encounter issues with your custom code (e.g., relied on RoleNeed holding the Role.name as its value, but it now holds Role.id), then you should run the following script to set Role.id to Role.name for all your Roles (make sure you have invenio-app-rdm 14.0.1 installed beforehand):

invenio shell $(find $(dirname $(dirname $(uv python find)))/lib/*/site-packages/invenio_app_rdm -name migrate_role_ids_to_names.py)

invenio shell $(find $(pipenv --venv)/lib/*/site-packages/invenio_app_rdm -name migrate_role_ids_to_names.py)