Initial login fails when running development profile on k3s

CIVITAS / CORE V2 Deployment & Operations
1

I deployed v2.0-beta on a fresh k3s cluster. I tried following the deployment guide. My only deviation was to disable IPv6 in APISIX because my network does not support it.

All components are in Running state but I can not sign in with my initial user.

The sequence of events is as follows:

  • Visit portal.<domain>
  • „Sign in with Keycloak“
  • Enter credentials and submit
  • 302 idm authenticate
  • 302 portal keycloak
  • 307 portal /
  • 307 portal signout
  • Back to the portal; not logged in

In the logs of portal-front I found:

frontend Server fetch failed for /users/me: Error [AuthError]: Backend responded with status 401                                                                 │
frontend     at e (.next/server/chunks/ssr/[root-of-the-server]__8e133fb0._.js:2:2165)                                                                           │
frontend     at async ao (.next/server/chunks/ssr/_c08c328d._.js:1:68298)                                                                                        │
frontend Keycloak logout completed

The culprit seems to be in portal-backend (shortened):

ERROR 1 --- [CIVITAS/CORE Data Management API] [nio-8080-exec-5] o.a.c.c.C.[.[.[/v1].[dispatcherServlet]  : Servlet.service() for servlet [dispatcherServlet] in context with path [/v1] threw exception

org.springframework.security.authentication.AuthenticationServiceException: An error occurred while attempting to decode the Jwt: I/O error on GET request for "https://idm.<domain>/realms/dev/protocol/openid-connect/certs": (certificate_unknown) PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
[...]
Caused by: org.springframework.security.oauth2.jwt.JwtException: An error occurred while attempting to decode the Jwt: I/O error on GET request for "https://idm.<domain>/realms/dev/protocol/openid-connect/certs": (certificate_unknown) PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
[...]
Caused by: com.nimbusds.jose.RemoteKeySourceException: I/O error on GET request for "https://idm.<domain>/realms/dev/protocol/openid-connect/certs": (certificate_unknown) PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
[...]
Caused by: org.springframework.web.client.ResourceAccessException: I/O error on GET request for "https://idm.<domain>/realms/dev/protocol/openid-connect/certs": (certificate_unknown) PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
[...]
Caused by: javax.net.ssl.SSLHandshakeException: (certificate_unknown) PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
[...]
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
[...]
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
[...]
WARN 1 --- [CIVITAS/CORE Data Management API] [nio-8080-exec-5] d.c.p.c.e.SecurityExceptionHandler       : Authentication failed: Full authentication is required to access this resource

This looks like the backend not trusting the ingress certificate.

How would I make portal-backend trust my k3s’ self-signed CA certificate (/var/lib/rancher/k3s/server/tls/server-ca.crt)?

If I have overlooked anything, please do let me know.

1 „Gefällt mir“
2

Thanks for the report. I’ll forward the issue to the developers and we come back to you asap.

1 „Gefällt mir“
3

Hi,
This looks indeed like a certificate issue. You could have a look or use the startup.sh script here. This creates a local k3d cluster with a self signed certificate. Since you have already a k3s cluster running, you can only have a look at line 300 with with text Applying CA configuration. This way you can try to configure your cluster the same way, as we did locally.

I think it may be a naming issue, that your cert has another name, as we use for the local deployment.
This is currently the way, we get the self signed secret: CA_CERT=$(kubectl get secret ca-secret -n cert-manager -o jsonpath='{.data.tls\.crt}')

I will also create an issue improve the documentation, when working with self signed certificates and maybe we also need to change the code, to make it more generic.

2 „Gefällt mir“
4

Ok, I’ve got the first sign of life – I managed to log in and have already taken a look around the portal.

Looking at startup.sh (which I had missed before because I did not want to develop, so I followed the ‚Deployment‘ guide instead), I could see that the cluster setup is more elaborate than expected.

I switched to startup.sh and k3d did not want to talk to its agents, so I ended up using minikube (startup.sh -m). I had to change two things:

  1. storageClass is now back to the default.
  2. I may have had clusterIssuer set incorrectly before, which may have been the source of the problem, but I did not cross-check with k3s.

Thanks for the helpful reply. For the moment, I am happy with this setup.

1 „Gefällt mir“