Jobs/Automated tasks¶

Introduced in v13

A job is a wrapper around a Celery task that can be scheduled, run on demand, and managed via the administration interface. You can also monitor the execution status of each job, run history, and logs.

Scheduler¶

The Invenio job system uses a custom Celery task scheduler which requires a separate Celery beat. You can run the custom beat in a separate container or shell like so:

celery -A invenio_app.celery beat -l INFO --scheduler invenio_jobs.services.scheduler:RunScheduler

The -l INFO flag sets the logging level to INFO. You can adjust the logging level (e.g., DEBUG, WARNING, ERROR) based on your needs, and the logs will reflect the level you set.

How to create a new job¶

This guide walks developers through implementing a new job using the engine provided by the invenio-jobs module. Jobs are asynchronous tasks that can be triggered from the admin UI or REST API. They run using Celery and support logging, argument validation, and result tracking.

Prerequisites¶

• Python development environment set up with InvenioRDM.
• Understanding of Celery tasks and entry points.
• Access to an InvenioRDM instance with administrator rights.
• A custom Celery scheduler beat process configured and running, which is required for executing both scheduled and manual jobs.

1. Define Your Celery Task¶

Create your actual business logic as a Celery task:

from celery import shared_task
from your_package import get_records_with_expired_embargoes, lift_embargo, reindex_records

@shared_task
def update_expired_embargoes(since=None, **kwargs):
    """Update and reindex records with expired embargoes."""
    records = get_records_with_expired_embargoes(since)
    for record in records:
        lift_embargo(record)
    reindex_records(records)

Error Handling and Logging¶

• Use current_app.logger for logs visible in the UI.
• Raise TaskExecutionPartialError to mark a job as partially successful.

from invenio_jobs.errors import TaskExecutionPartialError
raise TaskExecutionPartialError("Processed 80%, 10 records failed.")

2. Create a Job Class¶

Use JobType to define a job that wraps your task. This crucial step ensures that your task appears as an selectable option when you choose what job to run within the UI. You can inherit directly or use the create() helper.

from invenio_jobs.jobs import JobType

class UpdateEmbargoesJob(JobType):
    id = "update_expired_embargoes"
    title = "Update expired embargoes"
    description = "Lift and reindex expired embargoes."
    task = update_expired_embargoes

    @classmethod
    def build_task_arguments(cls, job_obj, since=None, **kwargs):
        return {"since": since}

To include custom fields in the admin form, define a Marshmallow schema:

from marshmallow import fields, validate
from invenio_jobs.jobs import PredefinedArgsSchema

class ExportEmbargoSchema(PredefinedArgsSchema):
    dry_run = fields.Boolean(
        required=False,
        missing=False,
        metadata={"description": "If true, no changes will be persisted."}
    )

class UpdateEmbargoesJob(JobType):
    id = "update_expired_embargoes"
    title = "Update expired embargoes"
    description = "Lift and reindex expired embargoes."
    task = update_expired_embargoes
    arguments_schema = ExportEmbargoSchema

    @classmethod
    def build_task_arguments(cls, job_obj, since=None, dry_run=False, **kwargs):
        return {
            "since": since,
            "dry_run": dry_run
        }

3. Register Your Job and Task¶

In your project file, register the task and job using entry points:

[project.entry-points."invenio_celery.tasks"]
mysite_tasks = "mysite.tasks"

[project.entry-points."invenio_jobs.jobs"]
update_expired_embargoes = "mysite.jobs:UpdateEmbargoesJob"

[options.entry_points]
invenio_celery.tasks =
    mysite_tasks = mysite.tasks

invenio_jobs.jobs =
    update_expired_embargoes = mysite.jobs:UpdateEmbargoesJob

4. Run and Monitor Jobs¶

Once your instance is restarted:

• Navigate to the Admin Panel → Jobs.
• Click Create Job and fill in the fields:
  • Title and description.
  • Select the task (e.g. Update Expired Embargoes).
  • Choose the queue (a default queue should exist if jobs are enabled).
• After saving, you can trigger the job immediately or schedule it.
• You can also inspect previous runs and access detailed logs.

Tips¶

• Make your tasks idempotent if they might be run concurrently.
• Jobs and their runs are stored in the database, with status, arguments, and logs recorded for traceability.
• Admins only: only superusers can run jobs.
• See Job system design for an explanation of how the system works internally.

Email Notification Templates¶

Job notification emails use Jinja2 templates that can be customized to match your instance's branding and requirements. The templates are designed to be user-friendly and provide clear information about job execution outcomes.

Available Templates¶

The notification system uses three templates:

• invenio_jobs/emails/base.html - Base email template with header and footer
• invenio_jobs/emails/run_notification.html - HTML email body for job run notifications
• invenio_jobs/emails/run_notification.txt - Plain text email body for job run notifications

Customizing Templates¶

To customize the email templates for your instance, override them in your instance's templates directory:

1. Create the templates directory structure in your instance:

   mkdir -p templates/semantic-ui/invenio_jobs/emails
   

2. Copy the templates you want to customize from the invenio-jobs module to your instance templates directory.

3. Modify the templates to match your instance's branding, adding custom styling, logos, or additional information as needed.

Template Context¶

The notification templates receive the following context variables:

• job - The Job object containing job metadata (name, description, etc.)
• run - The JobRun object with execution details (status, started_at, finished_at, etc.)
• config - Application configuration for accessing site settings
• admin_url - Direct link to view the run details in the admin panel

Example Customization¶

Here's an example of how to customize the HTML notification template:

{% extends "invenio_jobs/emails/base.html" %}

{% block content %}
<div style="padding: 20px;">
  <h2>Job Run Notification: {{ job.title }}</h2>

  <p>The job "{{ job.title }}" has completed with status:
    <strong style="color: {% if run.status == 'SUCCESS' %}green{% else %}red{% endif %};">
      {{ run.status }}
    </strong>
  </p>

  <p>Started: {{ run.started_at }}</p>
  <p>Finished: {{ run.finished_at }}</p>

  <p><a href="{{ admin_url }}">View full details in admin panel</a></p>
</div>
{% endblock %}