Dev deployment V2 - 403 for every request

CIVITAS / CORE V2 Deployment & Operations
1

I deployed Civitas/Core V2 on a VM (Ubuntu 26, K3d) following these instructions: Local Deployment | . Everything (more or less) is deployed and running, but I get 403 for every configured URL, example:

$ curl -ik https://portal.civitas.test
HTTP/2 403 
date: Tue, 14 Jul 2026 15:32:45 GMT
content-length: 0
strict-transport-security: max-age=31536000; includeSubDomains

Any idea, what might be wrong here? I assume, that APISix don’t get proper routes and thus nothing works. Would be awesome, if someone can shorten debugging :slight_smile:

These are the currently running pods:

$ kubectl -n dev get pods
NAME                                               READY   STATUS             RESTARTS      AGE
apisix-544f6d6cdb-45fjx                            2/2     Running            0             10m
authz-authz-authz-858cccdff8-8tzwk                 2/2     Running            0             11m
authz-authz-opa-74d6fc9797-984xf                   2/2     Running            0             11m
config-adapter-b486d5dc6-7b2zv                     2/2     Running            0             4m50s
etcd-etcd-0                                        2/2     Running            0             15m
frost-frost-frost-server-http-66fcc87d48-rbgqc     2/2     Running            0             9m33s
geoserver-geoserver-gsc-gateway-b58f78c6c-gbn85    2/2     Running            0             7m12s
geoserver-geoserver-gsc-rest-76c5965798-sns7m      2/2     Running            0             7m12s
geoserver-geoserver-gsc-wcs-d88fff45b-8b8kl        2/2     Running            0             7m12s
geoserver-geoserver-gsc-webui-56fdb659cb-ckbhp     2/2     Running            0             7m12s
geoserver-geoserver-gsc-wfs-5d69dffd5b-t9kws       2/2     Running            0             7m12s
geoserver-geoserver-gsc-wms-65b7f95dfb-tthkz       2/2     Running            0             7m12s
geoserver-rabbitmq-0                               2/2     Running            0             7m53s
kafka-cluster-brokers-0                            2/2     Running            0             13m
kafka-cluster-controllers-1                        2/2     Running            0             13m
kafka-cluster-entity-operator-7cfccddb6c-nggps     3/3     Running            0             13m
kafka-ui-7d44b6f9cf-gxvkb                          2/2     Running            0             13m
keycloak-app-keycloakx-0                           2/2     Running            0             12m
nifi-nifi-0                                        1/2     CrashLoopBackOff   4 (74s ago)   4m28s
portal-portal-backend-847dc888cf-58t9m             2/2     Running            0             8m56s
portal-portal-frontend-b7877dbbb-z9rww             2/2     Running            0             8m56s
postgres-cluster-1                                 2/2     Running            0             15m
postgres-operator-cloudnative-pg-9f7bccd45-sdbmw   2/2     Running            0             16m
strimzi-cluster-operator-7f77b5b679-5jfgw          2/2     Running            0             14m
superset-superset-68cc79848d-4ttg9                 2/2     Running            0             6m2s
superset-superset-init-db-jclxp                    0/2     Completed          0             5m40s
superset-superset-worker-5864bd7db5-jdgt2          2/2     Running            0             6m2s
valkey-valkey-8479c5bcc4-4c2fw                     2/2     Running            0             6m15s

Nifi crashes because it won’t eat self signed TLS certificate.

I added this config to global.yaml.gotmpl

global:
  domain: civitas.test
  instanceSlug: dev
  profile: development
  initialUserEmail: replace-with-your-email@example.com
  ingress:
    clusterIssuer: 'letsencrypt-prod'
    ingressClass: 'nginx'

components:
  - prepare
  - secrets
  - networkpolicies
  - runtime-policies
  - postgres
  - etcd
  - kafka
  - keycloak
  - authz
  - apisix
  - frost
  - portal
  - geoserver
  - valkey
  - superset
  - config-adapters
  - nifi

Edit:

  • corrected instruction link
1 „Gefällt mir“
2

Hello Markus,
i’ll pass your question on to DevOps and get back to you soon.
Cheers,
Mario

1 „Gefällt mir“
3

Thanks! I can support with debugging, lot’s of experience with K8s here… You can find me on Matrix here: @ztarbug:matrix.org

4

Hello Markus,

u use civitas.test as domain but letsencrypt. You have to use a valid domain for the certificates. If you want to test the deployment, you have to follow another walktrough local deployment with selfsigned certificates
Cheers,
Mario

5

Sorry wrong link - I actually followed tutorial you mentioned. Only difference ist, that I used a newer version of LinkerD. Will retry with the older version referenced in the tutorial.

6

Minor feedback for the installation instruction. This command ist not working as intended:

linkerd install --crds | kubectl apply -f -

Result:

The Gateway API CRDs must be installed prior to installing Linkerd. Run:

kubectl apply --server-side -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.4.0/standard-install.yaml

or see https://gateway-api.sigs.k8s.io/guides/#installing-gateway-api for more options.
error: no objects passed to apply

Do you want Gitlab issues for stuff like that?

7

Hello Markus,
thank you for your reporting.
You can create Incidents for deployment issues in https://gitlab.com/civitas-connect/civitas-core/civitas-core-v2/civitas-core-deployment/-/work_items/views/1024984
Thank you!
Mario

8

I repeated installation with a new cluster and result stays the same. No route is working (return 403). Nifi is still not coming up because of self signed certificate.

Any ideas, how I can further debug this problem?

Edit: Opened bug ticket on Gitlab.