### Terraform Version ```shell Terraform v1.16.6 on linux_amd64 + provider registry.terraform.io/hashicorp/random v3.9.1 ``` Also reproduced with v1.15.8, v1.15.9, and a build of `main` at 9ec49e06a868 (v1.18.0-dev). ### Terraform Configuration Files `main.tf`: ```hcl terraform { required_providers { random = { source = "hashicorp/random" } } } resource "random_password" "this" { count = 1 length = 16 } ``` `tests/main.tftest.hcl`: ```hcl mock_provider "random" {} run "assertion_fails" { command = apply assert { condition = length(random_password.this) == 0 error_message = "expected no password" } } ``` The failing assertion triggers the panic tracked in #39365. That panic is used here only because it is a deterministic way to make Terraform crash while a provider plugin is running. ### Debug Output A full TF_LOG=trace log is available on request. Panic excerpt: ``` panic: unexpected error marshalling value: value has marks, so it cannot be serialized as JSON goroutine 86 [running]: runtime/debug.Stack() runtime/debug/stack.go:26 +0x5e github.com/hashicorp/terraform/internal/logging.PanicHandler() github.com/hashicorp/terraform/internal/logging/panic.go:84 +0x15c ... github.com/hashicorp/terraform/internal/command/views/json.NewDiagnostic({0x4ab02a0, 0x35a38c928120}, 0x35a38ca54c30) github.com/hashicorp/terraform/internal/command/views/json/diagnostic.go:278 +0x18de ... github.com/hashicorp/terraform/internal/moduletest/graph.(*NodeTestRun).Execute(0x35a38c4c2cf0, 0x35a38bfc1260) github.com/hashicorp/terraform/internal/moduletest/graph/node_test_run.go:147 +0x4b8 github.com/hashicorp/terraform/internal/moduletest/graph.Walk.func1({0x4445fa0, 0x35a38c4c2cf0}) github.com/hashicorp/terraform/internal/moduletest/graph/test_graph_builder.go:130 +0x257 ``` In the trace log, the `random` plugin is started (`provider: plugin started: ... pid=...`) and never logs `plugin process exited`. ### Expected Behavior When Terraform crashes, it still terminates the provider plugin processes it started, as it does on a normal exit. No `terraform-provider-*` process outlives the `terraform` process. ### Actual Behavior `terraform test` exits with status 11 after the crash banner. The `terraform-provider-random` process it started keeps running. It is reparented to the init or subreaper process and is still running two minutes later: ``` $ terraform test; echo "exit=$?" ... exit=11 $ ps -o pid,ppid,stat,etime,rss,comm -p 3250962 # 10 s after exit PID PPID STAT ELAPSED RSS COMMAND 3250962 594 Sl 00:10 21504 terraform-provi $ ps -o pid,ppid,stat,etime,rss,comm -p 3250962 # 120 s after exit PID PPID STAT ELAPSED RSS COMMAND 3250962 594 Sl 02:00 10496 terraform-provi ``` PID 594 is the session's subreaper. The process has to be killed by hand. With a large provider such as `hashicorp/aws`, each orphan holds roughly 375-380 MB RSS. Repeated crashes in CI or in scripted test loops accumulate these processes. ### Steps to Reproduce 1. Save the two files above. 2. `terraform init` 3. `terraform test; echo "exit=$?"` prints `exit=11` after the crash banner. 4. `pgrep -a terraform-provider` lists the `random` plugin that is still running. No credentials are needed. ### Additional Context The plugin is orphaned only when a provider instance is still alive at the moment of the panic. In `terraform test`, that is the case when the test file declares a `provider` or `mock_provider` block: | Test file declares | Panic | Orphaned plugins | |---|---|---| | no provider block (per-run provider) | yes | 0 | | `provider "random" {}` | yes | 1 | | `mock_provider "random" {}` | yes | 1 | `TestProvidersTransformer` (`internal/moduletest/graph/transform_providers.go`) starts file-level providers while it builds the graph. They are closed only by `NodeProviderClose`, after all runs. Per-run providers are closed before the run's result is rendered, so they are already gone when the panic happens. The orphaning itself is not specific to `terraform test`: - Worker goroutines defer `logging.PanicHandler()`. That function ends with `os.Exit(11)` (`internal/logging/panic.go:89`). - `os.Exit` skips deferred calls, including `defer plugin.CleanupClients()` in `realMain` (`main.go:248`). - Managed plugins (`internal/command/meta_providers.go:467`) are therefore never killed. The same `defer logging.PanicHandler()` pattern guards goroutines in `internal/terraform/graph.go`, `internal/terraform/context.go`, `internal/backend/local/backend_plan.go`, `backend_apply.go` and `backend_refresh.go`. A panic in any of them would orphan running plugins. Only the `terraform test` path is demonstrated here. A local experiment that calls `plugin.CleanupClients()` in `PanicHandler` immediately before `os.Exit(11)` removed the orphans in every case tried. Those cases were the configuration above, the same with `provider "random" {}`, plan mode, and a module using `hashicorp/aws` under `mock_provider`. The exit status stayed 11 and the crash report was unchanged. Whether that is the right place for the cleanup is open for discussion. ### References - #39365: the `terraform test` panic used as the trigger here. - #39239: the same panic. It notes that the crash also skips the test's apply cleanup. ### Generative AI / LLM assisted development? An LLM coding agent helped narrow down the reproduction and read the source. Every result above was run and checked on the listed versions.