Ga naar hoofdinhoud
AcademytutorialPull external data into your register with OpenConnector

Pull external data into your register with OpenConnector

Feed your OpenRegister Pet Store from a live external API without writing integration code. You register the public Swagger Pet Store as a source, test the connection, write a Twig mapping that reshapes an external pet into your pet-academy schema and test it against a sample record, tie source, endpoint, results/id position, mapping and target register together in a synchronisation, run it for real to create pets in your register, and finally schedule it so new records arrive on their own — with id-based deduplication on every re-run. Pet Store domain, with UI, API and PHP tabs throughout.

TutorialOpenConnectorOpenRegisterSynchronisationIntegrationMappingTwigTutorial
23 min read

OpenConnector is the integration engine for Conduction apps. Where OpenRegister stores the data you own, OpenConnector reaches out to data you do not — a case-management API, an open-data portal, a partner's REST service — and pulls it into a register on a schedule, reshaped to fit your own schema. It does this without a line of integration code: you describe where the data lives, how a record should be reshaped, and what should run, and OpenConnector does the fetching, the transforming, the writing and the deduplicating for you.

This tutorial fills the Pet Store register you built in Part 1 from a real, public, external API — the Swagger Pet Store. You register that API as a source, test it, write a Twig mapping that turns an external pet into a pet-academy object and test the mapping against a sample record, build a synchronisation that ties it all together, run it for real to create pets in your register, and schedule it so it keeps the register current. No code, no cron file, no glue.

You are in the middle of a full data pipeline, all in the Pet Store domain:

  1. Model your data with OpenRegister — design the schemas and store records.
  2. OpenConnector (you are here) — feed the register from an external API on a schedule.
  3. Publish a public catalog with OpenCatalogi — expose the register as an open, harvestable catalog.

Each tutorial stands on its own, but together they take a record from "modelled" to "public". This part assumes the petstore-academy register and its pet-academy schema from Part 1; if you do not have them, run Part 1 first — it takes about half an hour.

Most steps below have three tabs — UI, API, and PHP — so you can follow the click-path, automate it with curl, or call OpenConnector's OCA\ services in code. The UI is the star; the API and PHP tabs show the same action as automation. One step (scheduling) is UI-only, and the tutorial says so when you get there.

The three pieces of an integration

Before clicking, hold these three words in your head. OpenConnector splits every integration into three objects, and you build them in this order:

  • A source is where the data comes from: a base URL and how to authenticate. The Swagger Pet Store is a source.
  • A mapping is how a record is reshaped: each field of your schema gets a Twig expression that reads from the external record. "An external pet becomes a pet-academy object" is a mapping.
  • A synchronisation is what runs: it pulls from a source endpoint, applies a mapping, and writes into a register and schema. It is the thing you run, and later schedule.

Open OpenConnector from the Nextcloud app menu. The dashboard counts each piece — Sources, Mappings, Synchronizations, Contracts, Jobs, Endpoints — and charts the outgoing calls, job executions and synchronisation runs over time. On a fresh instance every tile reads zero; you are about to make them count.

The OpenConnector dashboard with count tiles for Sources, Mappings, Synchronizations, Contracts, Jobs and Endpoints, and time-series charts for outgoing calls, job executions and synchronisation runs
The OpenConnector dashboard. The six tiles are the building blocks of every integration; the charts track what runs. The left navigation groups them under Connections (sources, endpoints) and Automation (mappings, synchronisations, jobs).

Step 1: Register the Swagger Pet Store as a source

In the left navigation open Connections → Sources. This is the list of every external system OpenConnector knows about. The toolbar carries an Add Source button.

Click Add Source. The Create Source dialog asks for a name, a type, an optional description, and an isEnabled toggle (on by default). Name it Swagger Pet Store (Academy), set the type to api (the type tells OpenConnector's CallService which adapter to dispatch at runtime — api is plain HTTP), add a short description, and leave it enabled. Click Create.

The Create Source dialog with a description, an isEnabled toggle set on, name set to Swagger Pet Store (Academy), and type set to api
The Create Source dialog: a name, a type (api), a description and the enabled toggle. The type is the runtime adapter; api is a plain HTTP source.

Now give the source its location — the base URL it talks to. Open the source from the list (its Actions → View) and set the location to https://petstore.swagger.io/v2. This public API needs no authentication, so there are no credentials to add. The detail page shows the source's full record: location, type, isEnabled, and the auth, rate-limit and retention fields you would fill in for a real, secured source.

The source detail page for Swagger Pet Store (Academy) showing its data grid with location set to https://petstore.swagger.io/v2, type api, isEnabled ticked, and empty auth, rate-limit and credential fields
The source's detail page. location is https://petstore.swagger.io/v2; the auth, credential and rate-limit fields stay empty because this public API needs none.

Use the source's Actions → Test connection to confirm OpenConnector can reach the host. OpenConnector pings the source's base location and records the response. For the Swagger Pet Store the bare base path returns a server response (the host is reachable; the root simply has no resource of its own) — the real end-to-end check is running the synchronisation in Step 4, which calls an actual endpoint.

The Sources table with the row Actions menu open for Swagger Pet Store (Academy), showing View, Test connection, View logs, Edit, Copy and Delete
Each source row's Actions menu carries Test connection and View logs. Test confirms OpenConnector can reach the host; View logs records every call it made, with the exact request and response.

Verify: Swagger Pet Store (Academy) appears in Connections → Sources with type api, and its detail page shows the location https://petstore.swagger.io/v2.

Step 2: Write the mapping

A mapping turns one external record into one object in your schema. The Swagger Pet Store returns pets as a bare JSON array; one record looks like this:

{
  "id": 9,
  "name": "doggie",
  "status": "available",
  "category": { "id": 1, "name": "Dogs" },
  "photoUrls": ["..."],
  "tags": [{ "id": 1, "name": "friendly" }]
}

Your pet-academy schema (from Part 1) wants name, species and status. The species is not a top-level field on the source — it sits inside the nested category object — so the mapping has to reach into it. Each target field is a Twig expression that reads from the source record:

pet-academy fieldExpression
name{{ name }}
species{{ category.name }}
status{{ status }}

Open Automation → Mappings → Add Mapping. OpenConnector creates the mapping and drops you straight into its editor. The editor has a General section (name, description, and a Pass through toggle), a Transformation rules section with Mapping rules, Cast rules and Unset rules tabs, and a Live preview that re-renders as you work.

On the Mapping rules tab click Add mapping rule. A dialog asks for a Target property and a Twig template. Add the three rows from the table above — name{{ name }}, then species{{ category.name }}, then status{{ status }}.

The Add mapping rule dialog with Target property set to name and Twig template set to a name expression, with an Add rule button
The Add mapping rule dialog. Each rule maps one target property to a Twig template; {{ category.name }} reaches into the nested category object for the species.

Now the important toggle: turn Pass through off. With pass-through on, every field of the source record (its id, photoUrls, raw tags) is copied through alongside your three derived fields — handy sometimes, but here it would pollute your clean pet-academy object with source plumbing. Off, the output is exactly the three fields you mapped.

Paste a sample pet into the Sample input (JSON) box and watch the Output re-render live. With pass-through off and the three rules in place, the output is precisely what pet-academy expects:

{ "name": "doggie", "species": "Dogs", "status": "available" }
The mapping editor with three mapping rules (name, species, status), Pass through disabled, a sample Swagger pet in the input box and the output showing name doggie, species Dogs, status available
The mapping editor's Live preview. With Pass through off and three rules, a Swagger pet reshapes to exactly name, species and status — the live output updates as you type.

Give the mapping a name — Pet to pet — and a description from the editor's General section (or rename it later from the mappings list). It now reads cleanly as the thing it is: a Swagger pet, reshaped into one of your pets.

The Pet to pet mapping detail showing its name, description, Pass through disabled, the Mapping rules tab with three rules, and the live preview below
The finished Pet to pet mapping: three rules, pass-through off, and a live preview that proves it produces a clean pet-academy object.

Before you wire it into a synchronisation, run the mapping's Test mapping action. It opens a dialog where you paste an input object, optionally pick a schema to validate the output against, and click Run test — the Result panel shows the reshaped object, so you confirm the mapping does what you expect against a real record, not a guess.

The Test mapping dialog with a Swagger pet in the Input object box and the Result panel showing name doggie, species Dogs, status available
The Test mapping dialog: an input on the left, the transformed result on the right. Run it before wiring the mapping up, so you know the transformation is right.

Verify: the Test mapping dialog (or the API test call) returns { "name": "doggie", "species": "Dogs", "status": "available" } for a sample Swagger pet.

Step 3: Create the synchronisation

The synchronisation ties the three pieces together. It needs:

  • Source: the Swagger Pet Store (Academy) source from Step 1.
  • Source endpoint: /pet/findByStatus?status=available, the path appended to the source's location.
  • Results position: where the list of records sits in the response. The Pet Store returns a bare array, addressed as _root.
  • Id position: id, the field that uniquely identifies a record. OpenConnector keys deduplication on it.
  • Source → Target mapping: the Pet to pet mapping from Step 2.
  • Target: your petstore-academy register and its pet-academy schema.

Open Automation → Synchronizations → Add Synchronization. The Create Synchronization dialog asks for a name, a sourceType and a targetType. Name it Pets to Pet Store register, set sourceType to api and targetType to register/schema, and click Create.

The Create Synchronization dialog with name Pets to Pet Store register, sourceType api and targetType register/schema
The Create Synchronization dialog: a name and two discriminators — sourceType: api (pull from an HTTP API) and targetType: register/schema (write into an OpenRegister schema).

Open the new synchronisation's Actions → Open editor. The editor is laid out in panels: Source (source type, the source record, endpoint, id position, results position), Target (target type, register, schema, an optional object filter), and Mapping (the source → target mapping, plus optional reverse and hash mappings). Fill them from the dropdowns and fields:

  • In Source, pick Swagger Pet Store (Academy), set Endpoint to /pet/findByStatus?status=available, ID position to id, and Results position to _root.
  • In Target, pick the Pet Store register and the Pet schema.
  • In Mapping, pick Pet to pet as the Source → Target mapping.

Then Save changes.

The synchronisation editor showing the Source panel (source type API, endpoint, ID position, Results position _root), the Target panel (Register/Schema, register, schema, object filter) and the Mapping panel
The synchronisation editor. The Source panel sets the endpoint and where to find the records and their ids; the Target panel picks the register and schema; the Mapping panel supplies the transformation. That is the whole integration on one screen.

Verify: the synchronisation's editor shows the source, endpoint /pet/findByStatus?status=available, results position _root, id position id, the Pet to pet mapping, and the Pet Store register and Pet schema as the target.

Step 4: Run it

With the synchronisation configured, run it. From the Synchronizations list, open the row's Actions menu and choose Run now (the same menu offers Test (dry run), View logs and View contracts). OpenConnector fetches the endpoint, applies the mapping to each record, and writes the results into your register.

The Synchronizations table with the row Actions menu open showing Open editor, Run now, Test (dry run) and View logs
The synchronisation row's Actions menu. Run now executes it immediately; the run reports how many records it found, created, updated and skipped.

Run it with Run now. The run reports a result — how many records it read (found), and how many it created, updated and skipped. On a first run against your empty target, every found record is created. On our run the register went from the six pets you typed in Part 1 to ten: four new pets created from the Swagger source, no duplicates of the six you already had.

Now open OpenRegister → Data → Search / views, select the Pet Store register and the Pet schema in the sidebar, and the imported pets appear in the table — right next to Rex, Bella and the rest you created by hand in Part 1. The external records are now first-class objects in your register: searchable, versioned, permissioned, exactly like the ones you typed.

OpenRegister's Search / views scoped to the Pet Store register and Pet schema, showing the hand-typed pets (Rex, Bella, Whiskers, Coco, Spike, Sunny) and imported 'doggie' pets together in one table
The pets in OpenRegister, scoped to the Pet Store register and Pet schema. The hand-typed pets from Part 1 (Rex, Bella, …) and the imported doggie records from the Swagger source sit in one table — the synchronisation wrote straight into your pet-academy schema.

Verify: the run reports records created, and in OpenRegister's Search / views — scoped to the Pet Store register and Pet schema — the imported pets appear alongside the ones from Part 1.

Step 5: Schedule it

A synchronisation that runs once is an import. To keep the register current you schedule it, and OpenConnector registers a background job — its cron task runs the same synchronisation on the interval you pick. Open the synchronisation and set a schedule from its settings; from then on it runs on its own.

Scheduling is its own object (a job), so unlike the steps above it has no clean one-line API or PHP equivalent — create it from the UI. Once it exists, the Jobs tile on the dashboard counts it, and each execution shows up in the job-execution charts.

On every scheduled run, OpenConnector reads each record's id (the id position you set in Step 3) and compares it against what it already wrote — it keeps a contract linking each source record to the target object it produced. New ids are created, records whose content changed are updated, and unchanged records are skipped. The register tracks the source faithfully, and re-running never duplicates a pet you already have.

A daily open-data export does not need an hourly sync. Frequent runs against a slow or rate-limited API waste both sides and can get you throttled. Pick the longest interval that keeps the data fresh enough for its use.

Troubleshooting

The synchronisation runs but creates zero objects. Check the results position. If it does not point at the list of records in the response, OpenConnector finds nothing to process. Open the source endpoint in a browser and look at where the array sits: a bare top-level array is _root; a list wrapped in { "results": [...] } is results.

You run the synchronisation twice and get duplicate pets. Check the id position. Without a correct id field, OpenConnector cannot recognise a record it has already imported, so every run creates fresh copies. Point it at the field that uniquely identifies a record (here, id) and the second run updates or skips instead of duplicating.

A target field arrives empty. Check the mapping's Twig expression against the real shape of the source record. species comes from {{ category.name }}, not {{ species }}, because the source nests the category — use the mapping's live preview or Test dialog against a real record to catch this before you run.

Test yourself

You have pulled the data in if all of these are true:

  • A source named Swagger Pet Store (Academy) exists with location https://petstore.swagger.io/v2, type api, and its connection has been tested.
  • The Pet to pet mapping reshapes a Swagger pet to { name, species, status }, with species coming from {{ category.name }} and pass-through off, confirmed in the Test dialog.
  • The synchronisation ties the source, the endpoint /pet/findByStatus?status=available, results position _root, id position id, the Pet to pet mapping, and the petstore-academy / pet-academy target together.
  • Running it created pets in the register, visible in OpenRegister's Search / views alongside the Part 1 pets.
  • You understand that a schedule turns the run into a background job, and that the id position makes re-runs deduplicate — create, update, skip — instead of duplicating.

Where to go from here

Your register now fills itself from an external API, on a schedule, without code. The last part of the chain takes that register public.

Volgende stappen