First impressions, feedback, and technical questions on CIVITAS/CORE V2 (RC)

CIVITAS / CORE V2 User Questions & Answers
1

Hi all,

I deployed CIVITAS/CORE V2 (RC) to a Kubernetes cluster (AKS) and it went through without any major problems. As a first use case, I followed the Smart Meter guide from the docs: master data from a PostgreSQL data source, measurement data via MQTT, pipeline into the Sensor Data Storage, and published as a SensorThings API.

1. MQTT over TLS
The MQTT connector form has a „TLS Enabled“ switch, but when I released the dataset, it didn’t work. I checked the code, and it seems the TLS option isn’t fully implemented yet, and something is missing on the NiFi side. Currently, the portal offers an option that the pipeline engine rejects. Has anyone managed to use a TLS MQTT broker, or is plain tcp:// currently the only working option? Is TLS support planned?

2. NiFi vs. Redpanda Connect
Regarding NiFi and Redpanda Connect: does the platform currently use NiFi or Redpanda Connect? In the past, I noticed the platform seemed to mainly rely on Redpanda Connect.

3. Where do I see whether a data source connection works?
For both MQTT and SQL sources, the data source shows „Available“ and a „last active“ time, but I could not find a test-connection button or any error display. Is there a place in the portal where I can verify the connection is actually working, or is checking the logs the intended way for now?

4. How do I see whether a pipeline is deployed and processing data?
This one cost me the most time. When releasing the dataset failed (due to a mapping problem on my side), the dataset just went back to „Ready“ with no error shown anywhere in the portal. The API tile stayed visible too, so everything looked fine while nothing was actually deployed. I only found the real cause in the config-adapter logs on the cluster. Similarly, at runtime, records that fail in the pipeline are dropped silently and the portal shows nothing. Is there a recommended way to monitor pipeline health and throughput?

(Side note: I also hit the pipeline editor bug where deleting an edge doesn’t really delete it, so I had to rebuild the pipeline. I found that one already reported on GitLab, and the NiFi→FROST authentication issue also seems to be tracked already, so I won’t repeat those here.)

5. Superset / Grafana on top of FROST and GeoServer
What is the intended way to show data from the FROST server (or GeoServer) in Superset or Grafana? Superset is part of the deployment, but I found no documented integration path. Is there an official or planned integration?

6. Access tokens for the SensorThings API
For external applications that need to consume the dataset API (/v1/datasets//frost-server), what is the recommended way to obtain access tokens? I was not able to access the data from the API at all. Is it possible to change the settings to make it public and unprotected?

7. Modeling guidelines for SensorThings data structures
Are there guidelines for how to model the data structure used as the mapping target for the Sensor Data Storage? I learned by trial and error that the Thing needs a nested „properties“ class with an attribute flagged as {id} (the match key), that the Datastream tier needs the same, and that create fields are all-or-nothing per entity. The how-to guides describe the overall flow nicely, but I only figured out the concrete modeling rules for FROST targets from error messages. Did I miss a docs page, and if not, is one planned?

8. Questions about Add-ons
What is the intended purpose of add-ons? I checked the code and it looks like a really promising way to share applications between municipalities. What do you think about using add-ons for platforms like MQTT, Decidim, or ThingsBoard? Also, where can I see how to manage user roles? Do I need to write a custom integration to talk to Keycloak? Finally, is it possible for add-ons to change the Keycloak realm for intended applications? Some applications (e.g., citizen-facing ones) might need a different realm with less strict security policies.

Sorry for highlighting so many issues! I am really happy with the concept of add-ons because they offer an incredibly easy way to share open-source solutions across municipalities with high security practices and low maintenance costs.

My overall impression is very positive, and the platform does a lot out of the box. Most of my friction was about visibility: knowing whether connections, releases, and pipelines actually succeeded. I’m happy to provide details or open GitLab issues for anything that isn’t tracked yet.

Thanks!

Marek Mráz

2

Dear Marek, thank you for your detailed post. I will address the individual points one by one:
1.
This is a bug and we will take it up. It was already implemented before and seems to have been lost during the switch from Redpanda to Nifi. TLS is definitely supported.
2.
Due to security requirements from the Federal Office for Information Security (BSI), which defines mandatory standards, we had to switch from Redpanda to Nifi, since thread safety cannot be guaranteed with Redpanda. Apache Nifi is therefore the pipeline engine in version 2.0-rc. As a result, pipelines created on version 2.0-beta unfortunately can no longer be used.
3.
Unfortunately, we do not yet have a mechanism to validate a datasource. We are bringing this into development, but you are also very welcome to contribute.
4.
This is the same as in point 3. We still need to solve the issue that pipeline errors are currently not routed to the user. This has also been recorded. A workaround, admittedly not very user-friendly, is to deploy Loki + Grafana with the platform and evaluate the ConfigAdapter and Nifi logs. Of course, this only makes sense in a test setup and is not practical for the standard user of the platform.
5.
Proper integration with Superset will happen in the next version together with Trino, so that data can be accessed with OPA-based authorization. At the moment, Superset is only an intermediate step as an add-on. For now, the FROST DB could be integrated as a datasource. In that case, however, there will be no dataset-specific authorization logic. Grafana as a business dashboard tool is not directly supported by us at the moment. We offer Grafana as an add-on. In that context, FROST data can be integrated and displayed via the APIs of the respective datasets.
6.
Yes, you are right here: the API documentation is not finished yet. There is a Bruno collection link, in which the request for the token is stored under payload-api -> get-api-access-token. For this, a corresponding environment must be created. The token can then be added as a Bearer token to the request for the dataset. What is still missing are the user-friendly requests for the dataset payload APIs in the collection. I have already recorded that as a bug ticket.
7.
Yes, this is currently very complicated and not user-friendly. We still have a few open bugs around FROST and are actively working on them. In the future, we also want to include the STA model by default, but we will probably not make that for release 2.0 yet. We also want to create a dedicated how-to for STA, but that is not finished yet.
8.
We are working in parallel on an add-on / plugin concept. In short: add-ons extend the platform externally, while plugins extend it internally. Add-ons only access external interfaces, such as the Payload API for datasets, as well as Keycloak for SSO. Plugins, on the other hand, access internal interfaces. With plugins, it becomes possible to connect new datasources or datasinks without always having to adapt the CORE. For us, this is the tool for horizontal scaling in the use of the platform.

Thank you very much for your work and your input. It helps us a lot, and we look forward to continuing the exchange!

Best regards

Mario

3

Hi Mario,

thank you for taking the time to go through my list and answer everything. While working with the platform I collected a few more questions and suggestions. These are about where the platform could go, especially for non-IoT data and the future NGSI-LD integration.

9. Datasets without a pipeline (master data only)

A dataset today needs a pipeline to become useful. But a lot of municipal data is not measured in real time. It is master data: parking zones, waste collection points, opening hours of city facilities. For this kind of data, an employee should be able to define the structure once, enter or upload the content, set permissions, and publish, without any data source or pipeline. Pipelines then handle what they do best: measurement data arriving in real time. Is a static / master-data dataset type on the roadmap? With NGSI-LD coming, this looks like a space where municipality employees can create datasets from non-IoT data themselves, with the right permissions.

10. Browsing and exporting dataset content for non-technical users

Related, from the consumer side: a colleague from a city department cannot do much with an API endpoint URL. Non-technical employees need a way to find a dataset, open it, and look at what is inside. A table view of the data plus a CSV/Excel download would cover most of their needs. Is a dataset content viewer or export planned? Today you need an API client to look inside a dataset, which limits the audience to developers.

11. NGSI-LD: tenants, federation, and combined datasets

Looking ahead to the NGSI-LD integration: NGSI-LD brokers bring tenants and federation as first-class concepts, and it would be a pity to hide them. I would love to see this flow: a user creates an NGSI-LD tenant (which could map to a dataset), then creates a second tenant that federates over one or more existing tenants, and publishes that federated tenant as a new dataset. The result is a published dataset that combines two tenants without copying any data. Or, simpler: let advanced users create and experiment with tenants. Is the dataset↔tenant mapping for NGSI-LD already decided, and is federation between datasets in scope?

12. Notifications / subscriptions on dataset changes (MIMs)

The new release of the OASC Minimal Interoperability Mechanisms requires under „Accessing data“ that a subscriber can be notified when data changes (capability C3 / requirement R3.1: Capabilities and Requirements | Accessing Data | MIMs Landing Page). I found no user-facing way to get notified when a dataset or an entity changes. Is at least one supported notification path planned, for example SensorThings MQTT subscriptions per dataset, or NGSI-LD subscriptions once that lands? A plain „observe this dataset“ option with a webhook?

13. Missing translations / internationalization strategy

The portal ships with next-intl and de/en message files, but some strings live outside them, hardcoded in German. Two examples I hit: the UML modeler’s default element names (umlTypes.ts):

export const DEFAULT_NAMES = {
  class: 'NeueKlasse',
  interface: 'NeuesInterface',
  abstractClass: 'NeueAbstrakteKlasse',
  enumeration: 'NeueAufzählung',
  ...
}

and the default role descriptions seeded by the backend (Data Gatekeeper, Data Owner, Data Steward, …), which show up in the UI in German regardless of the selected language („Verantwortung für Datenschutz und Daten-Governance.“ etc.).

We plan a deployment outside Germany this fall, so the language barrier matters for us: our municipality staff will need the portal in their own language, and German-only strings in the middle of an otherwise translated UI hurt adoption. Two questions: is completing the translation coverage on the roadmap, and do you consider a community translation platform such as Crowdin (or Weblate) for adding new languages? We would contribute a full translation for our language if there were a workflow for it.

Thanks!

Marek Mráz

4

Dear Marek,

thank you for your questions, which are also being incorporated into our concepts and considerations. Due to the holiday period and release preparations, my reply has unfortunately been delayed somewhat, and I would like to apologize for that.

The requirement in point 9 is very close to what we have already received from the community, specifically in the context of metadata. CORE itself does not store the actual data, but only the metadata. In addition, there should also be file connectors that can import “static data” beyond metadata, meaning data that is uploaded once and then changes only little or not at all.

Point 10 ties directly into that, and I would refer here to the metadata catalog. With it, users should not only be able to browse the metadata, but also download the actual data via distributions. A prerequisite for that would be something like a generic REST interface, because then it would be easy to offer not only XML/JSON output, but also CSV/Excel.

Your comments on NGSI-LD are understandable and will be taken into account in the concept.

Notifications for dataset changes are also not yet on the roadmap. With context brokers such as Orion, subscriptions are already possible, but whether and how this can be extended to other data, and what priority it should have, still needs to be clarified.

I also understand your point regarding translations. The remaining hardcoded parts in the code will be refactored. You are also very welcome to contribute and help clean up these areas.

For the broader decision on internationalization, we need a decision from the Strategic Committee, which is being prepared. The background here is the additional maintenance effort, for example through Crowdin, for us as maintainers, without there currently being a way for foreign municipalities to join the association. That means we would have more effort without compensation, and this decision must be made by the Strategic Committee.

We would also like to explicitly thank you for your contributions, which help us greatly in reconsidering our own concepts and processes.

Best regards

Mario