External Market & Third-Party Data Connectors

Business Context

Business-Level Definition

Energy costs cannot be understood from consumption alone — the price paid per unit of energy changes constantly and a growing share of contracts are priced directly against public market prices rather than a fixed tariff.

This feature connects the system to external market data feeds and to suppliers' own billing systems, so that market prices and supplier invoices are available as structured data rather than something a user has to look up or key in by hand.

It is built as a single, reusable connector mechanism rather than one-off integrations, so that adding the next external data source or the next supplier's billing API in the future is a configuration change (not a new integration built from scratch).

Requirements Definition

• Show real, current market prices for electricity and gas wherever cost-awareness matters (dashboards, reports), instead of no data or manually maintained figures.
• Reduce manual invoice entry for suppliers that expose their own billing API, without removing or disrupting the existing way invoices are entered or imported today.
• Make it possible to add a new external data source or a new supplier's billing connection through configuration, without bespoke integration code each time.

Acceptance Criteria

• Day-ahead market prices for electricity and gas are available at 15-minute resolution, for the current day and historically, without requiring a live call to the external source at the moment someone views them.
• A connector to an external provider can be configured, edited and disabled per tenant.
• Invoices from a supplier that exposes a billing API are ingested automatically into the same invoice records used by manual and document-based entry, correctly matched to the meter they belong to.
• A repeated ingestion (e.g. after a scheduled job re-runs) never creates duplicate price records or duplicate invoices.
• If a single provider or a single tenant's connection fails, it does not stop data from being ingested for any other configured connection.
• Every call made to an external provider is recorded, so a failure can be diagnosed without needing to reproduce it.

Loom Link

N/A — not available in the source material.

Technical Context

User Stories / Use Cases

View current and historical market prices: A user viewing a cost dashboard or report sees the current day-ahead market price for electricity or gas, and can look back at prices for previous periods. The price shown reflects real published market data, refreshed automatically each day.

Configure a connection to a supplier's billing system: An administrator with access to tenant settings adds a connection to a supplier that offers a billing API — selecting the supplier, entering the required credentials, and enabling the connection. From that point on, invoices from that supplier are ingested automatically on the existing schedule; the administrator does not need to take any further action per invoice.

Invoice arrives through an automated connection: A new invoice becomes available from a connected supplier. The system downloads it, matches it to the correct meter using the supplier's own identifier for the delivery point, and creates the invoice record together with its line items and any attached documents (e.g. a PDF copy) — exactly as if it had been entered through the existing invoice screens.

A connection fails: A supplier's API is temporarily unavailable, or a tenant's credentials expire. The affected connection's next scheduled attempt is logged as a failure; other tenants and other suppliers continue to be processed normally. An administrator can see the failure by checking the connection's recent call history.

UI/UX Design

Pending UI/UX review. UI/UX design has not been completed. Acceptance criteria describe functional behaviour only. At minimum, the design will need to cover: a settings screen for administrators to add, edit and disable a connection per tenant; and a way to view recent call history for a connection when diagnosing a failure.

Functional Requirements

FR1. Market price data

Day-ahead market prices for electricity and gas are ingested on a recurring daily schedule, at 15-minute resolution, from the relevant market operator (currently the Czech day-ahead market). A price applies identically to every tenant using the system — it is not tenant-specific data, is not entered manually, and is not something any tenant configures. If the source is temporarily unavailable when the scheduled ingestion runs, the gap is logged and is filled automatically on the next successful run; it does not block anything else in the system.

If the system is extended to a market outside its current scope, that market's prices are added as their own independent price series, rather than as a variant of the existing one — a different market is not simply "more data" for the same series, since prices, currency and market rules differ by country.

FR2. Connections to external providers

A tenant's connection to an external provider (for example, a supplier's billing API) is represented as a single configuration: which provider, which category of connector (billing, market data, open data), and the credentials needed to call it. A connection can be enabled or disabled at any time without deleting its configuration.

The set of providers available to connect to is maintained as configurable data, not fixed in the software — adding support for a new supplier does not require a release.

FR3. Supplier invoice ingestion

Where a tenant has an active connection to a supplier's billing API, new invoices are downloaded on a recurring schedule and created as ordinary invoice records — the same records used by manual entry and by the existing document-based import (e.g. emailed invoice files). All three paths coexist; none replaces another.

An incoming invoice is matched to the correct meter using the delivery-point identifier the supplier assigns to it (the same kind of identifier used for this purpose today). An invoice that cannot be matched to any meter is not discarded silently — it is recorded as a warning for follow-up, and does not stop the rest of that batch from being processed.

An invoice already ingested for a given invoice number and meter is not created again if the same period is ingested a second time.

A credit note is recognised as such — either because the supplier's own data marks it explicitly, or, where a supplier does not provide that, because the invoiced amount is negative.

Where an invoice includes any file attachments (e.g. a PDF copy), they are downloaded and linked to the invoice and to the relevant meter.

FR4. Failure isolation and diagnostics

Every call made to an external provider — whether it succeeds, fails, or is skipped — is recorded, including which connection it was made through and, where relevant, which invoice or price record it produced. A failure affecting one tenant's connection, or one provider, has no effect on any other configured connection; each is processed independently.

A call that fails due to a temporary condition (timeout, temporary server error) is retried automatically before being recorded as a failure.

Non-Functional Requirements

Performance

• Market price ingestion for a given day completes in time for that data to be available before it is first needed for display (dashboards, reports).

Security

• Provider credentials are stored encrypted at rest and are never returned in plaintext by any API.

Transactional Operations

• Creating a supplier invoice — the invoice record together with its line items and any file attachments — is committed as a single atomic operation. A partially created invoice (e.g. line items without a parent invoice record, or vice versa) must never be visible; on any failure partway through, the whole operation rolls back and the ingestion run records it as a failure for that invoice rather than leaving a partial record behind.

API Analysis

GET    /v1/energy-prices                        List day-ahead market prices
GET    /v1/external-connectors                   List connector configurations for the current tenant
POST   /v1/external-connectors                   Create a connector configuration
PATCH  /v1/external-connectors/:id               Update a connector configuration
DELETE /v1/external-connectors/:id               Disable a connector configuration
GET    /v1/external-connectors/:id/logs          List recent call history for a connector

Domain Model (ER diagram) & Data Attribute Table

No ER diagram in source material. Related entity pages (Confluence, not yet migrated into this repo's Entity DAT catalog): an energy-price series entity, an external-connector configuration entity, and a connector call-log entity.

Data

No seed data is required. Market price data is populated entirely by the scheduled ingestion job from first run onward. The list of providers available for connection is maintained as configurable reference data.

Test Data

N/A — not covered in source material.

Logging

.info

• Market price ingestion run completed (record count, duration, date range covered)
• Supplier invoice ingestion run completed (invoice count, duration)

.warn

• An incoming invoice or reading could not be matched to any meter

.error

• A call to an external provider failed after retries were exhausted

Monitoring

N/A — not covered in source material beyond the logging entries above.

Caching

N/A — not covered in source material.

Backward Compatibility and Migration

Invoices could previously only be entered manually or brought in through the existing document-based import (e.g. emailed invoice files). This feature adds an additional, automated ingestion path for suppliers that offer their own billing API — it does not replace or change the manual and document-based paths, which continue to work exactly as before.

Market price data did not previously exist anywhere in the system; there is no prior data or behaviour to preserve.

Legal Context

N/A — not addressed in the source Confluence page; migrated as a reference copy without new legal analysis.

Cybersecurity Considerations

Provider credentials are stored encrypted at rest and are never returned in plaintext by any API (see Non-Functional Requirements → Security above). No further data-privacy assessment is present in the source. Beyond this, N/A — migrated as a reference copy without new security analysis.

Risk Assessment

N/A — not addressed in the source Confluence page; migrated as a reference copy without new risk analysis.

Auditing, Reporting & Measurement

N/A — not addressed in the source Confluence page beyond the Logging entries above.

---

Migrated as a reference copy from Confluence page "External Market & Third-Party Data Connectors" (id 723320834). Content reorganized to fit this repo's feature-doc template; not re-analyzed against the current codebase.