Symptoms
Intermittent unavailability of the authentication with error "You must be logged in to the server (Unauthorized)" is displayed on the commandline". Users authenticate via pinniped/dex against our LDAP server and login to the workload cluster
Cause
This error message can appear for a number of reasons but one possible cause is a significant time skew between workload cluster nodes and all management cluster nodes.This "unexpected validation error" is encountered in one specific case: the API server believes that the token is not valid yet. This will only occur when nodes have a significant clock skew that causes tokens issued from one node to be considered not valid yet by another node.Many instances showing below error:
1398:E1026 13:40:56.079238 1 authentication.go:63] "Unable to authenticate the request" err="[invalid bearer token, Token could not be validated.]" and 52:E1026 12:48:11.627682 1 claims.go:126] unexpected validation error: *errors.errorString
Resolution
Fixing time skew between workload cluster nodes and all management cluster nodes.See NTP in Preparing to Deploy Management Clusters or Configuring NTP without DHCP Option 42
Related Information
These error messages originate from https://github.com/kubernetes/kubernetes/blob/10988997f225447f89841bac08e8848852d7cb55/pkg/serviceaccount/claims.go#L126-L127
