RAMOSE

from pathlib import Path

from helper import call

TOKEN = Path("ramose/state/token").read_text().strip()

A simple request

Looking up a DOI returns the article’s title from OpenCitations Meta.

call("http://localhost:8081/api/v1/articles/10.1007/s11192-022-04367-w?format=json")

curl 'http://localhost:8081/api/v1/articles/10.1007/s11192-022-04367-w?format=json' 

# 200 OK

[
  {
    "doi": "10.1007/s11192-022-04367-w",
    "title": "Identifying And Correcting Invalid Citations Due To DOI Errors In Crossref Data"
  }
]

The join

RAMOSE reaches both OpenCitations endpoints and combines their results inside one operation. Where the other tools can only join on terms that already coincide in the data, RAMOSE splits an operation into steps and joins the resulting tables on an arbitrary key: here it resolves the DOI to an OMID, then combines the title from Meta with the reference count from Index.

call("http://localhost:8081/api/v1/citations/10.1007/s11192-022-04367-w?format=json")

curl 'http://localhost:8081/api/v1/citations/10.1007/s11192-022-04367-w?format=json' 

# 200 OK

[
  {
    "doi": "10.1007/s11192-022-04367-w",
    "title": "Identifying And Correcting Invalid Citations Due To DOI Errors In Crossref Data",
    "br": "https://w3id.org/oc/meta/br/061202127149",
    "oc_reference_count": "30"
  }
]

Output

The same operation in JSON and CSV.

call("http://localhost:8081/api/v1/citations/10.1007/s11192-022-04367-w?format=csv")

curl 'http://localhost:8081/api/v1/citations/10.1007/s11192-022-04367-w?format=csv' 

# 200 OK

doi,title,br,oc_reference_count
10.1007/s11192-022-04367-w,Identifying And Correcting Invalid Citations Due To DOI Errors In Crossref Data,https://w3id.org/oc/meta/br/061202127149,30

Pagination

A bounded window with RFC 8288 Link headers and a known total.

call("http://localhost:8081/api/v1/references/10.1007/s11192-022-04367-w?format=json&page=1&page_size=2", show_headers=True)

curl -i 'http://localhost:8081/api/v1/references/10.1007/s11192-022-04367-w?format=json&page=1&page_size=2' 

# 200 OK
# Content-Type: application/json
# Link: </api/v1/references/10.1007/s11192-022-04367-w?format=json&page=2&page_size=2&total_items=30>; rel="next", </api/v1/references/10.1007/s11192-022-04367-w?format=json&page=1&page_size=2&total_items=30>; rel="first", </api/v1/references/10.1007/s11192-022-04367-w?format=json&page=15&page_size=2&total_items=30>; rel="last"

[
  {
    "doi": "10.1007/s11192-022-04367-w",
    "br": "https://w3id.org/oc/meta/br/061202127149",
    "cited": "https://w3id.org/oc/meta/br/062501777134"
  },
  {
    "doi": "10.1007/s11192-022-04367-w",
    "br": "https://w3id.org/oc/meta/br/061202127149",
    "cited": "https://w3id.org/oc/meta/br/061302130520"
  }
]

Versioning

The API version is carried in the base path (/api/v1).

API description

OpenAPI 3.2. RAMOSE serves a Swagger UI at http://localhost:8081/docs and the spec at http://localhost:8081/api/v1/openapi.yaml.

import requests
import yaml

from helper import embed_swagger

spec = yaml.safe_load(requests.get("http://localhost:8081/api/v1/openapi.yaml", timeout=120).text)
embed_swagger(spec, base_url="http://localhost:8081/")

Consumer authentication

Bearer tokens. An operation marked #auth required in the spec file needs a token; the others stay open. RAMOSE keeps only the SHA-256 hash of each token in a local SQLite store and issues them from the CLI (--token-create). The /protected operation below requires a token, so without one the request is rejected with 401.

call("http://localhost:8081/api/v1/protected/10.1007/s11192-022-04367-w?format=json")

curl 'http://localhost:8081/api/v1/protected/10.1007/s11192-022-04367-w?format=json' 

# 401 UNAUTHORIZED

{
  "type": "about:blank",
  "title": "Unauthorized",
  "status": 401,
  "detail": "missing or invalid bearer token",
  "instance": "/api/v1/protected/10.1007/s11192-022-04367-w?format=json"
}

With a valid bearer token the same request succeeds.

call(
    "http://localhost:8081/api/v1/protected/10.1007/s11192-022-04367-w?format=json",
    headers={"Authorization": f"Bearer {TOKEN}"},
)

curl -H 'Authorization: Bearer 4lJYe9RVBt_frk0NIZUNT23GlfZXAeJxZJ3kC96uhxc' 'http://localhost:8081/api/v1/protected/10.1007/s11192-022-04367-w?format=json' 

# 200 OK

[
  {
    "doi": "10.1007/s11192-022-04367-w",
    "title": "Identifying And Correcting Invalid Citations Due To DOI Errors In Crossref Data"
  }
]

Endpoint authentication

Each --backend-auth option gives RAMOSE a credential that it sends to one SPARQL endpoint only. A fixed Authorization header covers Basic and Bearer, while a Digest user:password entry makes RAMOSE answer each challenge from the store.

print(Path("ramose/entrypoint.sh").read_text())
call(f"http://localhost:8081/api/v1/basic-auth/10.1007/s11192-022-04367-w?format=json")
call(f"http://localhost:8081/api/v1/digest-auth/10.1007/s11192-022-04367-w?format=json")

#!/bin/sh
set -e

mkdir -p /app/state
python -m ramose --auth-db /app/.auth --token-create demo | tail -n1 > /app/state/token

exec python -m ramose -s /app/oc.hf -w 0.0.0.0:8081 --auth-db /app/.auth \
  --backend-auth "http://meta-basic:3030/sparql=Basic $(printf demo:demo | base64)" \
  --backend-auth "http://meta-digest:3030/sparql=Digest demo:demo"

curl 'http://localhost:8081/api/v1/basic-auth/10.1007/s11192-022-04367-w?format=json' 

# 200 OK

[
  {
    "doi": "10.1007/s11192-022-04367-w",
    "title": "Identifying And Correcting Invalid Citations Due To DOI Errors In Crossref Data"
  }
]
curl 'http://localhost:8081/api/v1/digest-auth/10.1007/s11192-022-04367-w?format=json' 

# 200 OK

[
  {
    "doi": "10.1007/s11192-022-04367-w",
    "title": "Identifying And Correcting Invalid Citations Due To DOI Errors In Crossref Data"
  }
]

Non-RDF sources

This RAMOSE demo exposes a REST API over a non-RDF CSV source. The operation below reads the same article from a CSV file through SPARQL Anything.

call("http://localhost:8081/api/v1/non-rdf/10.1007/s11192-022-04367-w?format=json")

curl 'http://localhost:8081/api/v1/non-rdf/10.1007/s11192-022-04367-w?format=json' 

# 200 OK

[
  {
    "doi": "10.1007/s11192-022-04367-w",
    "title": "Identifying And Correcting Invalid Citations Due To DOI Errors In Crossref Data",
    "venue": "Scientometrics"
  }
]

Operations

An operation whose #method is post, put, or delete carries a SPARQL Update in place of a query, sent to the #update_endpoint. The /titles/{omid} operations below write to a copy of OpenCitations Meta that accepts updates, and each one is marked #auth required. A POST creates a resource.

AUTH = {"Authorization": f"Bearer {TOKEN}"}

call("http://localhost:8081/api/v1/titles/0699", method="POST", headers=AUTH, data={"title": "A title"})
call("http://localhost:8081/api/v1/titles/0699?format=json")

curl -X POST -H 'Content-type: application/json' --data '{"title": "A title"}' -H 'Authorization: Bearer 4lJYe9RVBt_frk0NIZUNT23GlfZXAeJxZJ3kC96uhxc' http://localhost:8081/api/v1/titles/0699 

# 200 OK

{
  "status": 200,
  "message": "operation completed"
}
curl 'http://localhost:8081/api/v1/titles/0699?format=json' 

# 200 OK

[
  {
    "br": "https://w3id.org/oc/meta/br/0699",
    "title": "A title"
  }
]

A PUT replaces its title.

call("http://localhost:8081/api/v1/titles/0699", method="PUT", headers=AUTH, data={"title": "Another title"})
call("http://localhost:8081/api/v1/titles/0699?format=json")

curl -X PUT -H 'Content-type: application/json' --data '{"title": "Another title"}' -H 'Authorization: Bearer 4lJYe9RVBt_frk0NIZUNT23GlfZXAeJxZJ3kC96uhxc' http://localhost:8081/api/v1/titles/0699 

# 200 OK

{
  "status": 200,
  "message": "operation completed"
}
curl 'http://localhost:8081/api/v1/titles/0699?format=json' 

# 200 OK

[
  {
    "br": "https://w3id.org/oc/meta/br/0699",
    "title": "Another title"
  }
]

A DELETE removes the resource, so the last read finds nothing.

call("http://localhost:8081/api/v1/titles/0699", method="DELETE", headers=AUTH)
call("http://localhost:8081/api/v1/titles/0699?format=json")

curl -X DELETE -H 'Authorization: Bearer 4lJYe9RVBt_frk0NIZUNT23GlfZXAeJxZJ3kC96uhxc' http://localhost:8081/api/v1/titles/0699 

# 200 OK

{
  "status": 200,
  "message": "operation completed"
}
curl 'http://localhost:8081/api/v1/titles/0699?format=json' 

# 200 OK

[]

Caching

RAMOSE stores the processed result of each call in a local SQLite cache. The /cached operation reaches OpenCitations Meta through a proxy that records every request it forwards. The write operations above emptied the cache, so the first call reaches the endpoint, while the second one is answered from the cache.

from helper import count_endpoint_requests

count_endpoint_requests("http://localhost:8081/api/v1/cached/10.1007/s11192-022-04367-w?format=json")

First call: 200 OK, SPARQL requests that reached the endpoint: 1
Second call: 200 OK, SPARQL requests that reached the endpoint: 0

Control over JSON

A #format entry binds a format name to a converter in the addon module, which receives the result table and returns any structure. The nested format of the /references operation renames the fields and groups the cited resources under their article, where the default JSON returns one flat record per citation.

print(Path("ramose/addon.py").read_text())
call("http://localhost:8081/api/v1/references/10.1007/s11192-022-04367-w?format=json", max_lines=12)
call("http://localhost:8081/api/v1/references/10.1007/s11192-022-04367-w?format=nested", max_lines=12)

import csv
import io
import json


def to_nested(csv_string, request_url="", base_url=""):
    rows = list(csv.DictReader(io.StringIO(csv_string)))
    articles = {}
    for row in rows:
        article = articles.setdefault(row["br"], {"identifier": row["doi"], "omid": row["br"], "references": []})
        article["references"].append(row["cited"])
    return json.dumps(list(articles.values()), indent=2)

curl 'http://localhost:8081/api/v1/references/10.1007/s11192-022-04367-w?format=json' 

# 200 OK

[
  {
    "doi": "10.1007/s11192-022-04367-w",
    "br": "https://w3id.org/oc/meta/br/061202127149",
    "cited": "https://w3id.org/oc/meta/br/062501777134"
  },
  {
    "doi": "10.1007/s11192-022-04367-w",
    "br": "https://w3id.org/oc/meta/br/061202127149",
    "cited": "https://w3id.org/oc/meta/br/061302130520"
  },
  {
... (140 more lines)
curl 'http://localhost:8081/api/v1/references/10.1007/s11192-022-04367-w?format=nested' 

# 200 OK

[
  {
    "identifier": "10.1007/s11192-022-04367-w",
    "omid": "https://w3id.org/oc/meta/br/061202127149",
    "references": [
      "https://w3id.org/oc/meta/br/062501777134",
      "https://w3id.org/oc/meta/br/061302130520",
      "https://w3id.org/oc/meta/br/062601255589",
      "https://w3id.org/oc/meta/br/061302130471",
      "https://w3id.org/oc/meta/br/06903303973",
      "https://w3id.org/oc/meta/br/06902330758",
      "https://w3id.org/oc/meta/br/06250648394",
... (26 more lines)