Application settings

HERITRACE is configured through Docker environment variables.

Configuration method

Configure settings in your docker-compose.yml file:

environment:
  # Application settings
  - FLASK_ENV=production
  - APP_TITLE=HERITRACE
  - APP_SUBTITLE=A semantic editor for heritage data

  # Security (REQUIRED: Change this!)
  - SECRET_KEY=your-secure-secret-key-here

  # Query Configuration
  - COUNT_LIMIT=10000

  # Database endpoints
  - DATASET_DB_URL=http://host.docker.internal:8999/sparql
  - PROVENANCE_DB_URL=http://host.docker.internal:8998/sparql
  - DATASET_DB_TRIPLESTORE=virtuoso
  - PROVENANCE_DB_TRIPLESTORE=virtuoso

  # ORCID authentication
  - ORCID_CLIENT_ID=your-client-id
  - ORCID_CLIENT_SECRET=your-client-secret
  - ORCID_SAFELIST=0000-0000-0000-0000,0000-0000-0000-0001

Tip

Keep sensitive credentials secure. Use Docker secrets or external secret management for production.

Core application settings

Basic configuration

Environment Variable	Type	Description	Required	Default
APP_TITLE	String	Main application title shown in UI	No	HERITRACE
APP_SUBTITLE	String	Subtitle displayed in interface	No	Heritage Enhanced Repository Interface
SECRET_KEY	String	Flask secret key for security	Yes	generate-a-secure-random-key
CACHE_VALIDITY_DAYS	Integer	Days until the performance cache expires (internal Redis only)	No	7

Caution

Set a unique SECRET_KEY in production.

Note

The performance cache prevents slow initialization queries from running on every startup. After CACHE_VALIDITY_DAYS, the application will reinitialize counters from the database. This setting only applies when using the internal Redis instance.

Redis configuration

HERITRACE uses Redis for counter management and caching. You can use either an internal Redis instance (default) or connect to an external Redis server.

Internal Redis (default)

By default, HERITRACE runs a Redis instance inside the application container. No additional configuration is needed.

# No REDIS_URL specified - uses internal Redis
environment:
  - CACHE_VALIDITY_DAYS=7

External Redis

To use an external Redis instance, set the REDIS_URL environment variable:

environment:
  # Redis Configuration
  - REDIS_URL=redis://heritrace-redis:6379/0

  # Counter initialization is skipped when using external Redis
  # Make sure counters are already populated in the external instance

Environment Variable	Type	Description	Required	Default
REDIS_URL	String	Connection URL for external Redis server	No	redis://localhost:6379/0 (internal)

Database configuration

Compatible with SPARQL 1.1 triplestores. Tested with Virtuoso and Blazegraph.

When DATASET_DB_TEXT_INDEX_ENABLED is enabled, uses database text indexing to optimize SPARQL queries on literals. Available for Virtuoso and Blazegraph only.

Triplestore settings

environment:
  # Triplestore Types
  - DATASET_DB_TRIPLESTORE=virtuoso      # 'virtuoso' or 'blazegraph'
  - PROVENANCE_DB_TRIPLESTORE=virtuoso   # 'virtuoso' or 'blazegraph'

  # Database URLs
  - DATASET_DB_URL=http://localhost:8999/sparql
  - PROVENANCE_DB_URL=http://localhost:8998/sparql

  # Storage Types
  - DATASET_IS_QUADSTORE=true
  - PROVENANCE_IS_QUADSTORE=true

  # Features
  - DATASET_DB_TEXT_INDEX_ENABLED=true

Database options

Virtuoso (Recommended)

environment:
  - DATASET_DB_TRIPLESTORE=virtuoso
  - PROVENANCE_DB_TRIPLESTORE=virtuoso
  - DATASET_DB_TEXT_INDEX_ENABLED=true

Pros:

• Mature and battle-tested database
• Open source and SPARQL 1.1 compliant
• Excellent performance even with large datasets
• Full-text indexing and named graph support

Cons:

• Complex configuration
• Poor documentation

Visit the Virtuoso GitHub Repository

Blazegraph

environment:
  - DATASET_DB_TRIPLESTORE=blazegraph
  - PROVENANCE_DB_TRIPLESTORE=blazegraph
  - DATASET_DB_TEXT_INDEX_ENABLED=true

Pros:

• Simple to configure
• Open source and SPARQL 1.1 compliant
• Full-text indexing and named graph support

Cons:

• No longer actively maintained
• Performance degrades and may stop accepting triples with large datasets
• Poor documentation

Visit the Blazegraph GitHub Repository

Database Deployment Recommendations

For optimal performance, it is recommended to deploy your databases using Docker containers on the heritrace-network Docker network. This reduces latency between the application and database containers.

While the system functions correctly with external databases, using Docker containers on the same network provides better performance regardless of which triplestore you choose (Virtuoso, Blazegraph, or others).

Pro Tip: Managing Virtuoso with `virtuoso-utilities`

For comprehensive management of your Virtuoso instance—including launching the server, bulk-loading data, exporting dumps, and rebuilding the full-text index—we highly recommend using virtuoso-utilities.

This Python package automates many complex tasks that can be challenging to configure and execute correctly in Virtuoso, significantly simplifying database administration.

Storage configuration

Environment Variable	Options	Description
DATASET_IS_QUADSTORE	true/false	Enable named graph support
PROVENANCE_IS_QUADSTORE	true/false	Enable provenance named graphs
DATASET_DB_TEXT_INDEX_ENABLED	true/false	Enable internal query optimization using the database’s full-text search index

Schema and display configuration

These settings control the data model’s constraints and its presentation in the user interface.

Environment Variable	Description	Default
SHACL_PATH	Path to the SHACL schema file that defines data validation rules	./shacl.ttl
DISPLAY_RULES_PATH	Path to the YAML display rules file that controls how entities are rendered	./display_rules.yaml

File Customization

You can expose SHACL and display rules files as Docker volumes to customize them directly.

Data provenance and versioning

These settings configure how the application handles data provenance and versioning, aligning with the W3C PROV Ontology.

environment:
  # Provenance and Versioning
  - DATASET_GENERATION_TIME=2024-09-16T00:00:00+02:00
  - PRIMARY_SOURCE=https://doi.org/your-doi

Environment Variable	Description	Default
PRIMARY_SOURCE	Defines the default primary source (DOI/URL). This value is used for the prov:hadPrimarySource property. It’s used for the initial dataset import and proposed as the default when creating or modifying entities	Required
DATASET_GENERATION_TIME	Specifies the creation timestamp for a pre-existing dataset. This value is used for the prov:generatedAtTime property in the initial historical snapshot	Current time

URI generation and counter handling

HERITRACE uses a pluggable architecture for URI generation and counter management, allowing you to customize how unique identifiers are created for entities.

environment:
  # Component classes - specify custom implementations via these variables
  - COUNTER_HANDLER_CLASS=default_components.meta_counter_handler.MetaCounterHandler
  - URI_GENERATOR_CLASS=default_components.meta_uri_generator.MetaURIGenerator

Custom components

You can create custom URI generators and counter handlers by:

1. Creating custom components: Write your own Python classes in a custom_components/ directory
2. Mounting the volume: Add - ./custom_components:/app/custom_components to your docker-compose volumes
3. Updating environment variables: Set the class paths to your custom implementations

Example:

environment:
  - COUNTER_HANDLER_CLASS=custom_components.my_counter.FileCounterHandler
  - URI_GENERATOR_CLASS=custom_components.my_uri.UUIDURIGenerator
volumes:
  - ./custom_components:/app/custom_components

Default components configuration

The default components are configured through environment variables within their respective Python files:

• MetaCounterHandler: Configure Redis connection settings directly in default_components/meta_counter_handler.py
• MetaURIGenerator: Configure base IRI, supplier prefix, and regex patterns directly in default_components/meta_uri_generator.py

URI_GENERATOR

The URI_GENERATOR component is responsible for creating unique URIs for new entities. A custom URI generator should implement the following methods:

• generate_uri(entity_type: str | None = None) -> str: Generates a new URI for the given entity type
• initialize_counters(sparql) -> None: Initializes any required state from existing data in the database

Example: UUID-Based URI Generator

Here is an example of a custom generator that uses UUIDs instead of counters:

import uuid
from heritrace.uri_generator.uri_generator import URIGenerator

class UUIDURIGenerator(URIGenerator):
    """
    A simple URI generator that creates unique URIs using UUIDs.
    This generator does not rely on sequential counters.
    """
    def __init__(self, base_iri: str):
        self.base_iri = base_iri.rstrip('/')

    def generate_uri(self, entity_type: str | None = None) -> str:
        """
        Generates a new URI by appending a new UUID to the base IRI.
        The entity_type is ignored in this implementation.
        """
        return f"{self.base_iri}/{uuid.uuid4()}"

    def initialize_counters(self, sparql) -> None:
        """
        This method is not needed for a UUID-based generator,
        so it has an empty implementation.
        """
        pass

COUNTER_HANDLER

Caution

The COUNTER_HANDLER is only required if your URI_GENERATOR uses a counter-based strategy. If you use a different approach (like the UUID example above), this component is not needed and can be omitted from the configuration.

The COUNTER_HANDLER is a component designed to work with counter-based URI generators. It manages the persistent state of the numerical counters. While optional, it is crucial for ensuring URI uniqueness across application restarts when using a counter-based strategy.

A custom counter handler should implement the following methods:

• read_counter(entity_name: str) -> int: Returns the current counter value for a given entity type. Should return 0 if the counter doesn’t exist.
• set_counter(new_value: int, entity_name: str): Sets the counter for an entity type to a specific value.
• increment_counter(entity_name: str) -> int: Atomically increments the counter for an entity type by one and returns the new value.
• close(): Closes any open connections (e.g., to a database).

The default implementation, MetaCounterHandler, uses a Redis database to store these counters. This provides a fast and reliable way to persist the counter state.

Example: A Simple In-Memory Counter Handler

Here is an example of a minimal, non-persistent counter handler. Note: This is for demonstration only and is not suitable for production, as counter values will be lost on application restart.

import threading

class InMemoryCounterHandler:
    """
    A simple, non-persistent counter handler that stores counters in memory.

    This implementation is for demonstration purposes only. It is NOT
    suitable for production as all counter states will be lost when the
    application restarts.
    """
    def __init__(self):
        self._counters = {}
        self._lock = threading.Lock()

    def set_counter(self, new_value: int, entity_name: str) -> None:
        """Sets the counter for a given entity type to a specific value."""
        with self._lock:
            self._counters[entity_name] = new_value

    def read_counter(self, entity_name: str) -> int:
        """Reads the current value of a counter, returning 0 if not found."""
        with self._lock:
            return self._counters.get(entity_name, 0)

    def increment_counter(self, entity_name: str) -> int:
        """Increments a counter by one and returns the new value."""
        with self._lock:
            current_value = self._counters.get(entity_name, 0)
            new_value = current_value + 1
            self._counters[entity_name] = new_value
            return new_value

    def close(self):
        """No action needed for the in-memory handler."""
        pass

ORCID integration

Basic setup

environment:
  # ORCID OAuth Configuration
  - ORCID_CLIENT_ID=your-client-id
  - ORCID_CLIENT_SECRET=your-client-secret

  # ORCID Endpoints
  - ORCID_AUTHORIZE_URL=https://orcid.org/oauth/authorize
  - ORCID_TOKEN_URL=https://orcid.org/oauth/token
  - ORCID_API_URL=https://pub.orcid.org/v2.1
  - ORCID_SCOPE=/authenticate

  # Access Control
  - ORCID_SAFELIST=your-allowed-orcid-1,your-allowed-orcid-2

Setting up ORCID

1. Create an ORCID Account: If you don’t already have one, create an ORCID account.

2. Get Credentials:

   • Go to ORCID Developer Tools.
   • Create a new application and note your Client ID and Client Secret.

3. Configure Redirect URI:

   • In your ORCID application settings, add the redirect URI. The path is /auth/callback.
   • For local development: https://127.0.0.1:5000/auth/callback
   • For production: https://your-domain.com/auth/callback

4. Add Credentials to docker-compose.yml: Update your environment variables with the credentials from ORCID.

   environment:
     - ORCID_CLIENT_ID=APP-XXXXXXXXXX  # From ORCID registration
     - ORCID_CLIENT_SECRET=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx

5. Safelist Your ORCID ID: To enable access, you must add authorized ORCID IDs to the ORCID_SAFELIST. The application automatically extracts the ID from full ORCID URLs, so you can use either format.

   environment:
     # Allow specific researchers by ID or full URL (comma-separated)
     - ORCID_SAFELIST=0000-0002-1825-0097,https://orcid.org/0000-0001-5109-3700

Entity handling strategies

HERITRACE provides configurable strategies to manage how the application handles entities that are left without connections after a deletion. These strategies apply to two distinct types of entities: orphans and proxies.

environment:
  # Entity Handling Strategies
  - ORPHAN_HANDLING_STRATEGY=ASK  # Options: ASK, DELETE, KEEP
  - PROXY_HANDLING_STRATEGY=ASK   # Options: ASK, DELETE, KEEP

Strategy options

Orphan Handling

An orphan is an entity that is no longer referenced by any other entity in the database. For example, if you remove the last author from a book, the author’s record becomes an orphan if no other books or entities refer to it.

This strategy controls what happens to these orphaned entities:

• ASK: (Default) Prompts the user for confirmation before deleting any orphaned entities.
• DELETE: Automatically deletes any entities that become orphans as a result of a deletion.
• KEEP: Keeps the orphaned entities in the database, even if they are no longer connected to anything.

Proxy Handling

A proxy (or intermediate) entity is a resource that links two other entities, often to add specific attributes to their relationship. For example, a pro:RoleInTime entity can connect a foaf:Person to a fabio:Book, defining their role as an author.

When the primary relationship (e.g., the book’s author entry) is deleted, this strategy determines what happens to the intermediate pro:RoleInTime entity:

• ASK: (Default) Prompts the user for confirmation before deleting.
• DELETE: Automatically deletes the intermediate entity.
• KEEP: Retains the intermediate entity in the database.

Defining a Proxy Relationship

A property is treated as a proxy relationship in resources/display_rules.yaml by using the intermediateRelation key. This key tells HERITRACE to create an intermediate entity to connect the subject to the target object.

You must specify two things under intermediateRelation:

• class: The RDF class of the intermediate entity (e.g., pro:RoleInTime).
• targetEntityType: The RDF class of the final entity you want to create and link to (e.g., foaf:Agent).

Configuration Example

In this example, pro:isDocumentContextFor is configured as a proxy. When a user adds an author, HERITRACE creates a pro:RoleInTime entity that links the bibliographic resource to a foaf:Agent. See the Display Rules documentation for more details.

# In resources/display_rules.yaml
- property: "http://purl.org/spar/pro/isDocumentContextFor"
  displayName: "Author"
  intermediateRelation:
    class: "http://purl.org/spar/pro/RoleInTime"
    targetEntityType: "http://xmlns.com/foaf/0.1/Agent"
  displayRules:
    - shape: "http://schema.org/AuthorShape"
      displayName: "Author"
      # ...

Catalogue settings

These settings control the behaviour and display of the catalogue and time vault interfaces.

environment:
  # Query Configuration
  - COUNT_LIMIT=10000

  # Catalogue pagination settings
  - CATALOGUE_DEFAULT_PER_PAGE=50
  - CATALOGUE_ALLOWED_PER_PAGE=50,100,200,500

Environment Variable	Type	Description	Default
COUNT_LIMIT	Integer	Maximum count for entity queries. When exceeded, displays “LIMIT+” in catalog	10000
CATALOGUE_DEFAULT_PER_PAGE	Integer	Default number of items displayed per page in the catalogue	50
CATALOGUE_ALLOWED_PER_PAGE	String	Comma-separated list of pagination options available to users	50,100,200,500

Query Performance Optimization

COUNT_LIMIT optimizes catalog queries for large datasets. When counting entities of a specific class, the query stops at COUNT_LIMIT + 1. If the limit is exceeded, the catalog displays “LIMIT+” (e.g., “10000+”) instead of the exact count, preventing timeouts on extremely large result sets.

Pagination Performance Considerations

For large datasets, consider limiting the maximum items per page to maintain good performance.