### Terraform Version ```shell terraform version Terraform v1.3.5 on darwin_arm64 ``` ### Terraform Configuration Files ```terraform provider "kubernetes" { host = "127.0.0.1:8001" insecure = true } resource "test_assertions" "deployment" { component = "default-deployment" equal "deployment_name" { description = "Deployment name should be as expected." got = module.this.deployment.metadata.0.name want = "cluster-autoscaler" } equal "deployment_namespace" { description = "Deployment namespace should be as expected." got = module.this.deployment.metadata.0.name want = "kube-system" } } module "this" { source = "../../" command = [] arguments = ["sleep", "3600"] image_version = "v1.14.7" envs = { foo = "bar" } } ``` ### Debug Output I think that error level is sufficient as proof of the failure ``` 2022-11-23T15:02:22.067-0500 [ERROR] AttachSchemaTransformer: No provider config schema available for provider["terraform.io/builtin/test"] 2022-11-23T15:02:22.224-0500 [ERROR] AttachSchemaTransformer: No provider config schema available for provider["terraform.io/builtin/test"] 2022-11-23T15:02:22.370-0500 [ERROR] AttachSchemaTransformer: No provider config schema available for provider["terraform.io/builtin/test"] 2022-11-23T15:02:22.373-0500 [ERROR] AttachSchemaTransformer: No provider config schema available for provider["terraform.io/builtin/test"] 2022-11-23T15:02:22.649-0500 [ERROR] provider.terraform-provider-kubernetes_v2.16.0_x5: Response contains error diagnostic: diagnostic_detail= diagnostic_severity=ERROR tf_resource_type=kubernetes_service_account @module=sdk.proto diagnostic_summary="Post "https://127.0.0.1:8001/api/v1/namespaces/kube-system/serviceaccounts": dial tcp 127.0.0.1:8001: connect: connection refused" tf_proto_version=5.3 tf_provider_addr=registry.terraform.io/hashicorp/kubernetes tf_req_id=554ae2ce-56eb-e0fb-cfb8-46a56361bfa9 tf_rpc=ApplyResourceChange @caller=github.com/hashicorp/terraform-plugin-go@v0.14.0/tfprotov5/internal/diag/diagnostics.go:55 timestamp=2022-11-23T15:02:22.649-0500 2022-11-23T15:02:22.649-0500 [ERROR] vertex "module.this.kubernetes_service_account.this" error: Post "https://127.0.0.1:8001/api/v1/namespaces/kube-system/serviceaccounts": dial tcp 127.0.0.1:8001: connect: connection refused 2022-11-23T15:02:22.651-0500 [ERROR] provider.terraform-provider-kubernetes_v2.16.0_x5: Response contains error diagnostic: diagnostic_summary="Post "https://127.0.0.1:8001/apis/rbac.authorization.k8s.io/v1/namespaces/kube-system/roles": dial tcp 127.0.0.1:8001: connect: connection refused" tf_resource_type=kubernetes_role tf_req_id=05b7feb4-c67c-ea19-1c6f-6430980a2dc7 tf_rpc=ApplyResourceChange @caller=github.com/hashicorp/terraform-plugin-go@v0.14.0/tfprotov5/internal/diag/diagnostics.go:55 @module=sdk.proto diagnostic_detail= diagnostic_severity=ERROR tf_proto_version=5.3 tf_provider_addr=registry.terraform.io/hashicorp/kubernetes timestamp=2022-11-23T15:02:22.651-0500 2022-11-23T15:02:22.651-0500 [ERROR] vertex "module.this.kubernetes_role.this" error: Post "https://127.0.0.1:8001/apis/rbac.authorization.k8s.io/v1/namespaces/kube-system/roles": dial tcp 127.0.0.1:8001: connect: connection refused 2022-11-23T15:02:22.654-0500 [ERROR] provider.terraform-provider-kubernetes_v2.16.0_x5: Response contains error diagnostic: @caller=github.com/hashicorp/terraform-plugin-go@v0.14.0/tfprotov5/internal/diag/diagnostics.go:55 @module=sdk.proto diagnostic_detail= tf_req_id=fa0ced9f-7299-9c4c-3cf9-0d62b9f2b6f2 tf_resource_type=kubernetes_cluster_role tf_rpc=ApplyResourceChange diagnostic_severity=ERROR diagnostic_summary="Post "https://127.0.0.1:8001/apis/rbac.authorization.k8s.io/v1/clusterroles": dial tcp 127.0.0.1:8001: connect: connection refused" tf_proto_version=5.3 tf_provider_addr=registry.terraform.io/hashicorp/kubernetes timestamp=2022-11-23T15:02:22.654-0500 2022-11-23T15:02:22.654-0500 [ERROR] vertex "module.this.kubernetes_cluster_role.this" error: Post "https://127.0.0.1:8001/apis/rbac.authorization.k8s.io/v1/clusterroles": dial tcp 127.0.0.1:8001: connect: connection refused 2022-11-23T15:02:22.655-0500 [ERROR] AttachSchemaTransformer: No provider config schema available for provider["terraform.io/builtin/test"] 2022-11-23T15:02:23.017-0500 [ERROR] AttachSchemaTransformer: No provider config schema available for provider["terraform.io/builtin/test"] 2022-11-23T15:02:23.134-0500 [ERROR] AttachSchemaTransformer: No provider config schema available for provider["terraform.io/builtin/test"] ╷ │ Warning: The "terraform test" command is experimental │ │ We'd like to invite adventurous module authors to write integration tests │ for their modules using this command, but all of the behaviors of this │ command are currently experimental and may change based on feedback. │ │ For more information on the testing experiment, including ongoing research │ goals and avenues for feedback, see: │ https://www.terraform.io/docs/language/modules/testing-experiment.html ╵ Success! All of the test assertions passed. ``` ### Expected Behavior When running `terraform test`, I expect that if terraform isn't able to execute the test sequence: (init -> valudate -> apply -> destroy) to either exit with a status code other then 0 and not write the following: ``` ╷ │ Warning: The "terraform test" command is experimental │ │ We'd like to invite adventurous module authors to write integration tests │ for their modules using this command, but all of the behaviors of this │ command are currently experimental and may change based on feedback. │ │ For more information on the testing experiment, including ongoing research │ goals and avenues for feedback, see: │ https://www.terraform.io/docs/language/modules/testing-experiment.html ╵ Success! All of the test assertions passed. ``` Furthermore, I would expect that, when using the junit/xunit export function. An additional test which could be called terraform-sequence-{folder_name} with be set to failed, and all other would be skipped as a result. ### Actual Behavior When the terraform test sequence failed, only in the junit/xunit report, you can guess something is wrong because all of the tests are skipped ### Steps to Reproduce 1. Clone the code defined in this MR: https://gitlab.com/wild-beavers/terraform/module-kubernetes-cluster-autoscaler/-/merge_requests/2 2 Execute `terraform test` > Note you can reproduce it with any module where a terraform apply would fail. ### Additional Context I tested this as well with gitlab-ci where I provide an ephemeral k8s cluster to execute the test on. I knew something was wrong when the pipeline passed on the first attempt so I investigated and found this out. I might take a couple of days to reply to additional questions ### References _No response_