### Terraform Version ```shell Terraform v1.15.8 on linux_arm64 ``` ### Terraform Configuration Files Not config-specific — see below. The operation was a plan/apply against an existing state that produced **"No changes."** The hang is unrelated to the resource graph; it occurs during provider-plugin teardown after the operation finishes. ### Debug Output I have none ### Expected Behavior After `terraform apply` completes (here: "No changes. Your infrastructure matches the configuration."), Terraform should close its provider plugins and exit. ### Actual Behavior Terraform prints the completed plan/apply output, then **hangs forever** (observed >5 hours / "318 minutes" in goroutine ages) during provider-plugin shutdown. The process consumes no CPU and produces no further output. It must be killed manually (`SIGINT`/`SIGTERM` did not recover it; `SIGQUIT` produced the stack dump below). ### Steps to Reproduce It occurs rarely (one out of a few hundred runs) and nondeterministically at the *end* of an operation, regardless of whether there were changes to apply. So I can not provide valid steps. But the problem exists for month and occurred several dozen times. I observed the problem with terraform v1.14 as well. Invoked as: ``` terraform plan --out terraform apply -auto-approve ``` ### Additional Context ## Root cause (from the SIGQUIT goroutine dump) The apply itself completed successfully — `goroutine 1` shows `(*ApplyCommand).Run` past `RunOperation`, and stdout shows "No changes." The deadlock is in the **deferred provider `Close()` after schema loading**. Three goroutines form the deadlock: **goroutine 72 — `[sync.WaitGroup.Wait, 318 minutes]`** — the deferred plugin close, blocked waiting for the gRPC client's reader to finish: ``` loadschemas.(*Plugins).ProviderSchema.deferwrap1() internal/schemarepo/loadschemas/plugins.go:120 → plugin.(*GRPCProvider).Close internal/plugin/grpc_provider.go:1649 → go-plugin.(*Client).Kill go-plugin@v1.7.0/client.go:535 → go-plugin.(*GRPCClient).Close go-plugin@v1.7.0/grpc_client.go:108 → grpc.(*ClientConn).Close grpc@v1.80.0/clientconn.go:1229 → sync.(*WaitGroup).Wait ← blocks forever ``` **goroutine 217 — `[semacquire, 318 minutes]`** — the transport teardown, blocked in `(*FD).Close` waiting on the reader to release the fd: ``` grpc.(*ClientConn).Close.func2 → addrConn.tearDown → transport.(*http2Client).Close http2_client.go:1047 → net.(*netFD).Close → internal/poll.(*FD).Close → runtime_Semacquire ← waits for the blocked reader (gr 251) ``` **goroutine 251 — `[syscall, 318 minutes]`** — the HTTP/2 reader, stuck in a **raw blocking `syscall.read`** on the plugin's TLS/gRPC socket, *not* registered with Go's netpoller, so `FD.Close` cannot interrupt it: ``` syscall.read syscall/zsyscall_linux_arm64.go:736 → net.(*conn).Read → crypto/tls.(*Conn).Read → http2.readFrameHeader → transport.(*http2Client).reader http2_client.go:1663 ``` ### Summary of the deadlock `ClientConn.Close` (gr 72) waits on a `WaitGroup` for the transport to finish; transport teardown (gr 217) waits in `FD.Close` for the reader goroutine to exit; the reader (gr 251) is parked in a blocking `read()` on the plugin socket that `Close` cannot unblock. Because the plugin process/connection never delivers EOF or an error on that socket, the reader never returns and all three goroutines wait indefinitely. This happens in the deferred `Close` on the schema-loading path (`loadschemas/plugins.go:120`), i.e. after the operation is otherwise done. This looks like a provider-plugin gRPC connection whose reader is on a netpoller-bypassed fd, colliding with `ClientConn.Close`'s expectation that the reader can be interrupted. ## Additional context - Also observed: `goroutine 119` and `goroutine 261` are gRPC client-stream goroutines in `select`/`waitOnHeader` for ~318 minutes on the same broker connection (`go-plugin/grpc_broker.go:199`, `newGRPCClient`), consistent with a plugin gRPC stream that never completed its handshake/close. - The full goroutine dump is available and can be attached. ## Full goroutine dump SIGQUIT stack dump (attach the full trace here — redact internal names first) ``` SIGQUIT: quit PC=0x93dec m=0 sigcode=0 goroutine 0 gp=0x67427c0 m=0 mp=0x6743b00 [idle]: runtime.futex(0x6743c50, 0x80, 0x0, 0x0, 0x0, 0x0) runtime/sys_linux_arm64.s:651 +0x1c fp=0xffffffc04530 sp=0xffffffc04530 pc=0x93dec runtime.futexsleep(0x67427c0?, 0x67427c0?, 0xffffffc045b8?) runtime/os_linux.go:76 +0x28 fp=0xffffffc04580 sp=0xffffffc04530 pc=0x50688 runtime.notesleep(0x6743c50) runtime/lock_futex.go:47 +0x8c fp=0xffffffc045c0 sp=0xffffffc04580 pc=0x2822c runtime.mPark(...) runtime/proc.go:1974 runtime.stoplockedm() runtime/proc.go:3268 +0x64 fp=0xffffffc04620 sp=0xffffffc045c0 pc=0x5c524 runtime.schedule() runtime/proc.go:4131 +0x34 fp=0xffffffc04660 sp=0xffffffc04620 pc=0x5efc4 runtime.park_m(0x4000362c40) runtime/proc.go:4280 +0x2ac fp=0xffffffc046c0 sp=0xffffffc04660 pc=0x5f4fc runtime.mcall() runtime/asm_arm64.s:241 +0x54 fp=0xffffffc046d0 sp=0xffffffc046c0 pc=0x905b4 goroutine 1 gp=0x40000021c0 m=nil [select, 318 minutes]: runtime.gopark(0x400086d7f8?, 0x2?, 0x30?, 0x7f?, 0x400086d7a4?) runtime/proc.go:460 +0xc0 fp=0x4000b15610 sp=0x4000b155f0 pc=0x8aa10 runtime.selectgo(0x4000b157f8, 0x400086d7a0, 0x676c040?, 0x0, 0x4000683a40?, 0x1) runtime/select.go:351 +0x6bc fp=0x4000b15750 sp=0x4000b15610 pc=0x6a34c github.com/hashicorp/terraform/internal/command.(*Meta).RunOperation(0x4000683a40, {0x421ebb8?, 0x40002a2dc0?}, 0x400017fa20) github.com/hashicorp/terraform/internal/command/meta.go:497 +0x12c fp=0x4000b15830 sp=0x4000b15750 pc=0x2ba2f2c github.com/hashicorp/terraform/internal/command.(*ApplyCommand).Run(0x4000683a40, {0x40003a3e50, 0x3, 0x3}) github.com/hashicorp/terraform/internal/command/apply.go:122 +0x6d4 fp=0x4000b15930 sp=0x4000b15830 pc=0x2b85dd4 github.com/hashicorp/cli.(*CLI).Run(0x40007ce500) github.com/hashicorp/cli@v1.1.7/cli.go:265 +0x420 fp=0x4000b15a10 sp=0x4000b15930 pc=0x3b45a0 main.realMain() github.com/hashicorp/terraform/main.go:337 +0x17b0 fp=0x4000b15f20 sp=0x4000b15a10 pc=0x2cdf8f0 main.main() github.com/hashicorp/terraform/main.go:62 +0x1c fp=0x4000b15f40 sp=0x4000b15f20 pc=0x2cde11c runtime.main() runtime/proc.go:285 +0x278 fp=0x4000b15fd0 sp=0x4000b15f40 pc=0x56a68 runtime.goexit({}) runtime/asm_arm64.s:1268 +0x4 fp=0x4000b15fd0 sp=0x4000b15fd0 pc=0x92984 goroutine 2 gp=0x4000002700 m=nil [force gc (idle), 3 minutes]: runtime.gopark(0x18730cf7f28ef?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:460 +0xc0 fp=0x40000b4f90 sp=0x40000b4f70 pc=0x8aa10 runtime.goparkunlock(...) runtime/proc.go:466 runtime.forcegchelper() runtime/proc.go:373 +0xb4 fp=0x40000b4fd0 sp=0x40000b4f90 pc=0x56db4 runtime.goexit({}) runtime/asm_arm64.s:1268 +0x4 fp=0x40000b4fd0 sp=0x40000b4fd0 pc=0x92984 created by runtime.init.7 in goroutine 1 runtime/proc.go:361 +0x24 goroutine 3 gp=0x4000002c40 m=nil [GC sweep wait]: runtime.gopark(0x6735a01?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:460 +0xc0 fp=0x40000b5760 sp=0x40000b5740 pc=0x8aa10 runtime.goparkunlock(...) runtime/proc.go:466 runtime.bgsweep(0x40000d6000) runtime/mgcsweep.go:323 +0x104 fp=0x40000b57b0 sp=0x40000b5760 pc=0x40154 runtime.gcenable.gowrap1() runtime/mgc.go:212 +0x28 fp=0x40000b57d0 sp=0x40000b57b0 pc=0x33de8 runtime.goexit({}) runtime/asm_arm64.s:1268 +0x4 fp=0x40000b57d0 sp=0x40000b57d0 pc=0x92984 created by runtime.gcenable in goroutine 1 runtime/mgc.go:212 +0x6c goroutine 4 gp=0x4000002e00 m=nil [GC scavenge wait]: runtime.gopark(0x6741ce0?, 0x1c3c2b?, 0x0?, 0x0?, 0x0?) runtime/proc.go:460 +0xc0 fp=0x40000b5f60 sp=0x40000b5f40 pc=0x8aa10 runtime.goparkunlock(...) runtime/proc.go:466 runtime.(*scavengerState).park(0x6741ce0) runtime/mgcscavenge.go:425 +0x5c fp=0x40000b5f90 sp=0x40000b5f60 pc=0x3dcac runtime.bgscavenge(0x40000d6000) runtime/mgcscavenge.go:658 +0xac fp=0x40000b5fb0 sp=0x40000b5f90 pc=0x3e22c runtime.gcenable.gowrap2() runtime/mgc.go:213 +0x28 fp=0x40000b5fd0 sp=0x40000b5fb0 pc=0x33d88 runtime.goexit({}) runtime/asm_arm64.s:1268 +0x4 fp=0x40000b5fd0 sp=0x40000b5fd0 pc=0x92984 created by runtime.gcenable in goroutine 1 runtime/mgc.go:213 +0xac goroutine 5 gp=0x4000003340 m=nil [GOMAXPROCS updater (idle), 318 minutes]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:460 +0xc0 fp=0x40000b6770 sp=0x40000b6750 pc=0x8aa10 runtime.goparkunlock(...) runtime/proc.go:466 runtime.updateMaxProcsGoroutine() runtime/proc.go:6720 +0xf4 fp=0x40000b67d0 sp=0x40000b6770 pc=0x65804 runtime.goexit({}) runtime/asm_arm64.s:1268 +0x4 fp=0x40000b67d0 sp=0x40000b67d0 pc=0x92984 created by runtime.defaultGOMAXPROCSUpdateEnable in goroutine 1 runtime/proc.go:6708 +0x48 goroutine 6 gp=0x4000003500 m=nil [finalizer wait, 318 minutes]: runtime.gopark(0x0?, 0x3c97420?, 0x0?, 0x60?, 0x2000000020?) runtime/proc.go:460 +0xc0 fp=0x40000b4580 sp=0x40000b4560 pc=0x8aa10 runtime.runFinalizers() runtime/mfinal.go:210 +0x104 fp=0x40000b47d0 sp=0x40000b4580 pc=0x32dd4 runtime.goexit({}) runtime/asm_arm64.s:1268 +0x4 fp=0x40000b47d0 sp=0x40000b47d0 pc=0x92984 created by runtime.createfing in goroutine 1 runtime/mfinal.go:172 +0x78 goroutine 7 gp=0x40002e8e00 m=nil [GC worker (idle), 318 minutes]: runtime.gopark(0x175ea378e04c4?, 0x3?, 0x7c?, 0x0?, 0x0?) runtime/proc.go:460 +0xc0 fp=0x40000b6f10 sp=0x40000b6ef0 pc=0x8aa10 runtime.gcBgMarkWorker(0x400007f030) runtime/mgc.go:1463 +0xe0 fp=0x40000b6fb0 sp=0x40000b6f10 pc=0x36460 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1373 +0x28 fp=0x40000b6fd0 sp=0x40000b6fb0 pc=0x36348 runtime.goexit({}) runtime/asm_arm64.s:1268 +0x4 fp=0x40000b6fd0 sp=0x40000b6fd0 pc=0x92984 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1373 +0x140 goroutine 17 gp=0x4000362380 m=nil [GC worker (idle), 318 minutes]: runtime.gopark(0x175ea378e3518?, 0x3?, 0x2b?, 0x5a?, 0x0?) runtime/proc.go:460 +0xc0 fp=0x40000e5f10 sp=0x40000e5ef0 pc=0x8aa10 runtime.gcBgMarkWorker(0x400007f030) runtime/mgc.go:1463 +0xe0 fp=0x40000e5fb0 sp=0x40000e5f10 pc=0x36460 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1373 +0x28 fp=0x40000e5fd0 sp=0x40000e5fb0 pc=0x36348 runtime.goexit({}) runtime/asm_arm64.s:1268 +0x4 fp=0x40000e5fd0 sp=0x40000e5fd0 pc=0x92984 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1373 +0x140 goroutine 33 gp=0x4000504000 m=nil [GC worker (idle), 318 minutes]: runtime.gopark(0x175ea37942c5c?, 0x3?, 0x64?, 0x47?, 0x0?) runtime/proc.go:460 +0xc0 fp=0x40000eaf10 sp=0x40000eaef0 pc=0x8aa10 runtime.gcBgMarkWorker(0x400007f030) runtime/mgc.go:1463 +0xe0 fp=0x40000eafb0 sp=0x40000eaf10 pc=0x36460 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1373 +0x28 fp=0x40000eafd0 sp=0x40000eafb0 pc=0x36348 runtime.goexit({}) runtime/asm_arm64.s:1268 +0x4 fp=0x40000eafd0 sp=0x40000eafd0 pc=0x92984 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1373 +0x140 goroutine 8 gp=0x40002e8fc0 m=nil [GC worker (idle)]: runtime.gopark(0x18730d0bd8dc2?, 0x1?, 0x21?, 0x1?, 0x0?) runtime/proc.go:460 +0xc0 fp=0x400069df10 sp=0x400069def0 pc=0x8aa10 runtime.gcBgMarkWorker(0x400007f030) runtime/mgc.go:1463 +0xe0 fp=0x400069dfb0 sp=0x400069df10 pc=0x36460 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1373 +0x28 fp=0x400069dfd0 sp=0x400069dfb0 pc=0x36348 runtime.goexit({}) runtime/asm_arm64.s:1268 +0x4 fp=0x400069dfd0 sp=0x400069dfd0 pc=0x92984 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1373 +0x140 goroutine 9 gp=0x40002e9180 m=nil [GC worker (idle), 318 minutes]: runtime.gopark(0x175ea37943133?, 0x3?, 0x45?, 0x96?, 0x0?) runtime/proc.go:460 +0xc0 fp=0x40000e6f10 sp=0x40000e6ef0 pc=0x8aa10 runtime.gcBgMarkWorker(0x400007f030) runtime/mgc.go:1463 +0xe0 fp=0x40000e6fb0 sp=0x40000e6f10 pc=0x36460 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1373 +0x28 fp=0x40000e6fd0 sp=0x40000e6fb0 pc=0x36348 runtime.goexit({}) runtime/asm_arm64.s:1268 +0x4 fp=0x40000e6fd0 sp=0x40000e6fd0 pc=0x92984 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1373 +0x140 goroutine 10 gp=0x40002e9340 m=nil [GC worker (idle)]: runtime.gopark(0x676fb40?, 0x1?, 0x9?, 0x1b?, 0x0?) runtime/proc.go:460 +0xc0 fp=0x40000ebf10 sp=0x40000ebef0 pc=0x8aa10 runtime.gcBgMarkWorker(0x400007f030) runtime/mgc.go:1463 +0xe0 fp=0x40000ebfb0 sp=0x40000ebf10 pc=0x36460 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1373 +0x28 fp=0x40000ebfd0 sp=0x40000ebfb0 pc=0x36348 runtime.goexit({}) runtime/asm_arm64.s:1268 +0x4 fp=0x40000ebfd0 sp=0x40000ebfd0 pc=0x92984 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1373 +0x140 goroutine 11 gp=0x40002e9500 m=nil [GC worker (idle), 318 minutes]: runtime.gopark(0x676fb40?, 0x1?, 0xd?, 0x76?, 0x0?) runtime/proc.go:460 +0xc0 fp=0x40000e4f10 sp=0x40000e4ef0 pc=0x8aa10 runtime.gcBgMarkWorker(0x400007f030) runtime/mgc.go:1463 +0xe0 fp=0x40000e4fb0 sp=0x40000e4f10 pc=0x36460 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1373 +0x28 fp=0x40000e4fd0 sp=0x40000e4fb0 pc=0x36348 runtime.goexit({}) runtime/asm_arm64.s:1268 +0x4 fp=0x40000e4fd0 sp=0x40000e4fd0 pc=0x92984 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1373 +0x140 goroutine 12 gp=0x40002e96c0 m=nil [GC worker (idle), 318 minutes]: runtime.gopark(0x175ea372888a9?, 0x3?, 0x97?, 0x71?, 0x0?) runtime/proc.go:460 +0xc0 fp=0x4000697f10 sp=0x4000697ef0 pc=0x8aa10 runtime.gcBgMarkWorker(0x400007f030) runtime/mgc.go:1463 +0xe0 fp=0x4000697fb0 sp=0x4000697f10 pc=0x36460 runtime.gcBgMarkStartWorkers.gowrap1() runtime/mgc.go:1373 +0x28 fp=0x4000697fd0 sp=0x4000697fb0 pc=0x36348 runtime.goexit({}) runtime/asm_arm64.s:1268 +0x4 fp=0x4000697fd0 sp=0x4000697fd0 pc=0x92984 created by runtime.gcBgMarkStartWorkers in goroutine 1 runtime/mgc.go:1373 +0x140 goroutine 13 gp=0x40002e9c00 m=nil [cleanup wait, 315 minutes]: runtime.gopark(0x400090d400?, 0xf6c4e9b2f600?, 0x78?, 0x7f?, 0x30004?) runtime/proc.go:460 +0xc0 fp=0x4000507f40 sp=0x4000507f20 pc=0x8aa10 runtime.goparkunlock(...) runtime/proc.go:466 runtime.(*cleanupQueue).dequeue(0x6742600) runtime/mcleanup.go:439 +0x110 fp=0x4000507f80 sp=0x4000507f40 pc=0x2f800 runtime.runCleanups() runtime/mcleanup.go:635 +0x40 fp=0x4000507fd0 sp=0x4000507f80 pc=0x30010 runtime.goexit({}) runtime/asm_arm64.s:1268 +0x4 fp=0x4000507fd0 sp=0x4000507fd0 pc=0x92984 created by runtime.(*cleanupQueue).createGs in goroutine 1 runtime/mcleanup.go:589 +0x108 goroutine 15 gp=0x4000362c40 m=nil [select, 318 minutes, locked to thread]: runtime.gopark(0x400050cfa0?, 0x2?, 0x98?, 0xce?, 0x400050cf8c?) runtime/proc.go:460 +0xc0 fp=0x400050ce00 sp=0x400050cde0 pc=0x8aa10 runtime.selectgo(0x400050cfa0, 0x400050cf88, 0x0?, 0x0, 0x0?, 0x1) runtime/select.go:351 +0x6bc fp=0x400050cf40 sp=0x400050ce00 pc=0x6a34c runtime.ensureSigM.func1() runtime/signal_unix.go:1085 +0x188 fp=0x400050cfd0 sp=0x400050cf40 pc=0x85158 runtime.goexit({}) runtime/asm_arm64.s:1268 +0x4 fp=0x400050cfd0 sp=0x400050cfd0 pc=0x92984 created by runtime.ensureSigM in goroutine 1 runtime/signal_unix.go:1068 +0xcc goroutine 65 gp=0x4000682e00 m=8 mp=0x40003de808 [syscall, 318 minutes]: runtime.notetsleepg(0x676e9e0, 0xffffffffffffffff) runtime/lock_futex.go:123 +0x34 fp=0x40000b2790 sp=0x40000b2760 pc=0x284b4 os/signal.signal_recv() runtime/sigqueue.go:152 +0x30 fp=0x40000b27b0 sp=0x40000b2790 pc=0x8cc00 os/signal.loop() os/signal/signal_unix.go:23 +0x1c fp=0x40000b27d0 sp=0x40000b27b0 pc=0x3b1e9c runtime.goexit({}) runtime/asm_arm64.s:1268 +0x4 fp=0x40000b27d0 sp=0x40000b27d0 pc=0x92984 created by os/signal.Notify.func1.1 in goroutine 1 os/signal/signal.go:152 +0x28 goroutine 66 gp=0x40006836c0 m=nil [chan receive, 318 minutes]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:460 +0xc0 fp=0x40000b2ef0 sp=0x40000b2ed0 pc=0x8aa10 runtime.chanrecv(0x4000370150, 0x0, 0x1) runtime/chan.go:667 +0x428 fp=0x40000b2f70 sp=0x40000b2ef0 pc=0x22ba8 runtime.chanrecv1(0x0?, 0x0?) runtime/chan.go:509 +0x14 fp=0x40000b2fa0 sp=0x40000b2f70 pc=0x22744 main.makeShutdownCh.func1() github.com/hashicorp/terraform/commands.go:493 +0x2c fp=0x40000b2fd0 sp=0x40000b2fa0 pc=0x2cdd88c runtime.goexit({}) runtime/asm_arm64.s:1268 +0x4 fp=0x40000b2fd0 sp=0x40000b2fd0 pc=0x92984 created by main.makeShutdownCh in goroutine 1 github.com/hashicorp/terraform/commands.go:491 +0xbc goroutine 119 gp=0x4000363500 m=nil [select, 318 minutes]: runtime.gopark(0x4000c267a0?, 0x2?, 0xa0?, 0x90?, 0x4000c26784?) runtime/proc.go:460 +0xc0 fp=0x4000c26600 sp=0x4000c265e0 pc=0x8aa10 runtime.selectgo(0x4000c267a0, 0x4000c26780, 0x40019e8000?, 0x0, 0x4001a66a40?, 0x1) runtime/select.go:351 +0x6bc fp=0x4000c26740 sp=0x4000c26600 pc=0x6a34c google.golang.org/grpc.newClientStreamWithParams.func4() google.golang.org/grpc@v1.80.0/stream.go:426 +0x80 fp=0x4000c267d0 sp=0x4000c26740 pc=0x781f80 runtime.goexit({}) runtime/asm_arm64.s:1268 +0x4 fp=0x4000c267d0 sp=0x4000c267d0 pc=0x92984 created by google.golang.org/grpc.newClientStreamWithParams in goroutine 261 google.golang.org/grpc@v1.80.0/stream.go:425 +0xaa8 goroutine 72 gp=0x4000f12540 m=nil [sync.WaitGroup.Wait, 318 minutes]: runtime.gopark(0x6750320?, 0x22744?, 0xa0?, 0x62?, 0x3009101?) runtime/proc.go:460 +0xc0 fp=0x40008ca7a0 sp=0x40008ca780 pc=0x8aa10 runtime.goparkunlock(...) runtime/proc.go:466 runtime.semacquire1(0x40041f0198, 0x0, 0x1, 0x0, 0x19) runtime/sema.go:192 +0x204 fp=0x40008ca7f0 sp=0x40008ca7a0 pc=0x6b0f4 sync.runtime_SemacquireWaitGroup(0x400b80e858?, 0xdc?) runtime/sema.go:114 +0x38 fp=0x40008ca830 sp=0x40008ca7f0 pc=0x8c138 sync.(*WaitGroup).Wait(0x40041f0190) sync/waitgroup.go:206 +0xa8 fp=0x40008ca860 sp=0x40008ca830 pc=0x9e308 google.golang.org/grpc.(*ClientConn).Close(0x4000e6e008) google.golang.org/grpc@v1.80.0/clientconn.go:1229 +0x234 fp=0x40008ca940 sp=0x40008ca860 pc=0x76b584 github.com/hashicorp/go-plugin.(*GRPCClient).Close(0x40019de0c0) github.com/hashicorp/go-plugin@v1.7.0/grpc_client.go:108 +0x70 fp=0x40008ca990 sp=0x40008ca940 pc=0x7c13a0 github.com/hashicorp/go-plugin.(*Client).Kill(0x400070b100) github.com/hashicorp/go-plugin@v1.7.0/client.go:535 +0x224 fp=0x40008caa90 sp=0x40008ca990 pc=0x7b8484 github.com/hashicorp/terraform/internal/plugin.(*GRPCProvider).Close(0x400098a100) github.com/hashicorp/terraform/internal/plugin/grpc_provider.go:1649 +0x70 fp=0x40008caad0 sp=0x40008caa90 pc=0x29d8cb0 github.com/hashicorp/terraform/internal/schemarepo/loadschemas.(*Plugins).ProviderSchema.deferwrap1() github.com/hashicorp/terraform/internal/schemarepo/loadschemas/plugins.go:120 +0x2c fp=0x40008caaf0 sp=0x40008caad0 pc=0x9565cc runtime.deferreturn() runtime/panic.go:589 +0x60 fp=0x40008cab80 sp=0x40008caaf0 pc=0x529a0 github.com/hashicorp/terraform/internal/schemarepo/loadschemas.(*Plugins).ProviderSchema(0x4000881cb0, {{0x40003d59fa, 0x3}, {0x40003d59f0, 0x9}, {0x3a74be4, 0x15}}) github.com/hashicorp/terraform/internal/schemarepo/loadschemas/plugins.go:217 +0x104c fp=0x40008cb2c0 sp=0x40008cab80 pc=0x955e6c github.com/hashicorp/terraform/internal/schemarepo/loadschemas.loadProviderSchemas.func1({{0x40003d59fa, 0x3}, {0x40003d59f0, 0x9}, {0x3a74be4, 0x15}}) github.com/hashicorp/terraform/internal/schemarepo/loadschemas/load.go:47 +0x210 fp=0x40008cb5f0 sp=0x40008cb2c0 pc=0x9542a0 github.com/hashicorp/terraform/internal/schemarepo/loadschemas.loadProviderSchemas(0x4001a637b8?, 0xbf4be8?, 0x4000ea6000, 0x41a4540?) github.com/hashicorp/terraform/internal/schemarepo/loadschemas/load.go:68 +0x8c fp=0x40008cb6c0 sp=0x40008cb5f0 pc=0x953fcc github.com/hashicorp/terraform/internal/schemarepo/loadschemas.LoadSchemas(0x4000f088c0, 0x4000ea6000, 0x4000881cb0) github.com/hashicorp/terraform/internal/schemarepo/loadschemas/load.go:28 +0xa0 fp=0x40008cb730 sp=0x40008cb6c0 pc=0x953e80 github.com/hashicorp/terraform/internal/terraform.loadSchemas(...) github.com/hashicorp/terraform/internal/terraform/context_plugins.go:31 github.com/hashicorp/terraform/internal/terraform.(*Context).Schemas(0x0?, 0x0?, 0x0?) github.com/hashicorp/terraform/internal/terraform/context.go:176 +0x30 fp=0x40008cb7c0 sp=0x40008cb730 pc=0x9ace40 github.com/hashicorp/terraform/internal/backend/local.(*Local).opApply(0x40002a2dc0, {0x4204d70, 0x40005c5db0}, {0x4204d70, 0x40005c5e00}, 0x400017fa20, 0x4000f2a900) github.com/hashicorp/terraform/internal/backend/local/backend_apply.go:82 +0x378 fp=0x40008cbf00 sp=0x40008cb7c0 pc=0xbf4d78 github.com/hashicorp/terraform/internal/backend/local.(*Local).opApply-fm({0x4204d70?, 0x40005c5db0?}, {0x4204d70?, 0x40005c5e00?}, 0x0?, 0x92984?) :1 +0x54 fp=0x40008cbf50 sp=0x40008cbf00 pc=0xc00694 github.com/hashicorp/terraform/internal/backend/local.(*Local).Operation.func1() github.com/hashicorp/terraform/internal/backend/local/backend.go:349 +0xbc fp=0x40008cbfd0 sp=0x40008cbf50 pc=0xbf3e1c runtime.goexit({}) runtime/asm_arm64.s:1268 +0x4 fp=0x40008cbfd0 sp=0x40008cbfd0 pc=0x92984 created by github.com/hashicorp/terraform/internal/backend/local.(*Local).Operation in goroutine 1 github.com/hashicorp/terraform/internal/backend/local/backend.go:342 +0x388 goroutine 261 gp=0x4000505340 m=nil [select, 318 minutes]: runtime.gopark(0x4000a4db38?, 0x2?, 0x30?, 0x24?, 0x4000a4db34?) runtime/proc.go:460 +0xc0 fp=0x4000a4d9b0 sp=0x4000a4d990 pc=0x8aa10 runtime.selectgo(0x4000a4db38, 0x4000a4db30, 0x50?, 0x0, 0x4218f38?, 0x1) runtime/select.go:351 +0x6bc fp=0x4000a4daf0 sp=0x4000a4d9b0 pc=0x6a34c google.golang.org/grpc/internal/transport.(*ClientStream).waitOnHeader(0x4001a3a680) google.golang.org/grpc@v1.80.0/internal/transport/client_stream.go:96 +0x5c fp=0x4000a4db60 sp=0x4000a4daf0 pc=0x71e47c google.golang.org/grpc/internal/transport.(*ClientStream).RecvCompress(...) google.golang.org/grpc@v1.80.0/internal/transport/client_stream.go:111 google.golang.org/grpc.(*csAttempt).recvMsg(0x400158ee10, {0x36c6ec0, 0x40019e2180}, 0x40000b7e8a?) google.golang.org/grpc@v1.80.0/stream.go:1155 +0xb0 fp=0x4000a4dd00 sp=0x4000a4db60 pc=0x785890 google.golang.org/grpc.(*clientStream).RecvMsg.func1(0x40000b7d98?) google.golang.org/grpc@v1.80.0/stream.go:1020 +0x24 fp=0x4000a4dd30 sp=0x4000a4dd00 pc=0x784d24 google.golang.org/grpc.(*clientStream).withRetry(0x4001669c20, 0x4000a4de80, 0x4000a4de70) save .terraform.lock.hcl ... google.golang.org/grpc@v1.80.0/stream.go:824 +0x4e4 fp=0x4000a4ddb0 sp=0x4000a4dd30 pc=0x783cf4 google.golang.org/grpc.(*clientStream).RecvMsg(0x4001669c20, {0x36c6ec0?, 0x40019e2180?}) google.golang.org/grpc@v1.80.0/stream.go:1019 +0xdc fp=0x4000a4deb0 sp=0x4000a4ddb0 pc=0x784b7c github.com/hashicorp/go-plugin/internal/plugin.(*gRPCBrokerStartStreamClient).Recv(0x400043e060) github.com/hashicorp/go-plugin@v1.7.0/internal/plugin/grpc_broker_grpc.pb.go:68 +0x54 fp=0x4000a4dee0 sp=0x4000a4deb0 pc=0x78e3e4 github.com/hashicorp/go-plugin.(*gRPCBrokerClientImpl).StartStream(0x40019deb00) github.com/hashicorp/go-plugin@v1.7.0/grpc_broker.go:199 +0x16c fp=0x4000a4dfb0 sp=0x4000a4dee0 pc=0x7bdb6c github.com/hashicorp/go-plugin.newGRPCClient.func1() github.com/hashicorp/go-plugin@v1.7.0/grpc_client.go:73 +0x20 fp=0x4000a4dfd0 sp=0x4000a4dfb0 pc=0x7c12b0 runtime.goexit({}) runtime/asm_arm64.s:1268 +0x4 fp=0x4000a4dfd0 sp=0x4000a4dfd0 pc=0x92984 created by github.com/hashicorp/go-plugin.newGRPCClient in goroutine 72 github.com/hashicorp/go-plugin@v1.7.0/grpc_client.go:73 +0x37c goroutine 251 gp=0x4000f12700 m=3 mp=0x40000bb008 [syscall, 318 minutes]: syscall.Syscall(0x3f, 0xa, 0x4000bee000, 0xa000) syscall/syscall_linux.go:74 +0x20 fp=0x4000a49700 sp=0x4000a496a0 pc=0xafbb0 syscall.read(0x40010e6780?, {0x4000bee000?, 0x4000a49700?, 0x239878?}) syscall/zsyscall_linux_arm64.go:736 +0x40 fp=0x4000a49740 sp=0x4000a49700 pc=0xade00 syscall.Read(...) syscall/syscall_unix.go:183 internal/poll.ignoringEINTRIO(...) internal/poll/fd_unix.go:738 internal/poll.(*FD).Read(0x40010e6780, {0x4000bee000, 0xa000, 0xa000}) internal/poll/fd_unix.go:161 +0x204 fp=0x4000a497e0 sp=0x4000a49740 pc=0x109eb4 net.(*netFD).Read(0x40010e6780, {0x4000bee000?, 0x4000a49868?, 0x2a578?}) net/fd_posix.go:68 +0x28 fp=0x4000a49830 sp=0x4000a497e0 pc=0x1de808 net.(*conn).Read(0x4000406c58, {0x4000bee000?, 0x300bee000?, 0xf6c4e9e9af40?}) net/net.go:196 +0x34 fp=0x4000a49880 sp=0x4000a49830 pc=0x1ecd64 crypto/tls.(*atLeastReader).Read(0x4000ea6030, {0x4000bee000?, 0x4000a49928?, 0x241ef4?}) crypto/tls/conn.go:819 +0x38 fp=0x4000a498d0 sp=0x4000a49880 pc=0x241d98 bytes.(*Buffer).ReadFrom(0x4001a39428, {0x41a85e0, 0x4000ea6030}) bytes/buffer.go:217 +0x90 fp=0x4000a49930 sp=0x4000a498d0 pc=0x141510 crypto/tls.(*Conn).readFromUntil(0x4001a39188, {0x41a76a0, 0x4000406c58}, 0xa000?) crypto/tls/conn.go:841 +0xcc fp=0x4000a49970 sp=0x4000a49930 pc=0x241f5c crypto/tls.(*Conn).readRecordOrCCS(0x4001a39188, 0x0) crypto/tls/conn.go:630 +0x340 fp=0x4000a49bf0 sp=0x4000a49970 pc=0x23f880 crypto/tls.(*Conn).readRecord(...) crypto/tls/conn.go:592 crypto/tls.(*Conn).Read(0x4001a39188, {0x4001b8a000, 0x8000, 0x18?}) crypto/tls/conn.go:1397 +0x14c fp=0x4000a49c60 sp=0x4000a49bf0 pc=0x2453bc google.golang.org/grpc/internal/credentials.(*syscallConn).Read(0x4000a49cc8?, {0x4001b8a000?, 0x18?, 0x3493440?}) :1 +0x30 fp=0x4000a49c90 sp=0x4000a49c60 pc=0x6132e0 bufio.(*Reader).Read(0x4001aff200, {0x4001a490e4, 0x9, 0x9cba4?}) bufio/bufio.go:245 +0x188 fp=0x4000a49cd0 sp=0x4000a49c90 pc=0x282f08 io.ReadAtLeast({0x41a45e0, 0x4001aff200}, {0x4001a490e4, 0x9, 0x9}, 0x9) io/io.go:335 +0x98 fp=0x4000a49d20 sp=0x4000a49cd0 pc=0xfda28 io.ReadFull(...) io/io.go:354 golang.org/x/net/http2.readFrameHeader({0x4001a490e4, 0x9, 0x40000bb008?}, {0x41a45e0?, 0x4001aff200?}) golang.org/x/net@v0.55.0/http2/frame.go:250 +0x58 fp=0x4000a49d70 sp=0x4000a49d20 pc=0x6f7fe8 golang.org/x/net/http2.(*Framer).ReadFrameHeader(0x4001a490a0) golang.org/x/net@v0.55.0/http2/frame.go:513 +0x60 fp=0x4000a49e00 sp=0x4000a49d70 pc=0x6f8650 google.golang.org/grpc/internal/transport.(*framer).readFrame(0x40010e6b80) google.golang.org/grpc@v1.80.0/internal/transport/http_util.go:480 +0x44 fp=0x4000a49e40 sp=0x4000a49e00 pc=0x73cb04 google.golang.org/grpc/internal/transport.(*http2Client).reader(0x4000d8efc8, 0x40010f3880) google.golang.org/grpc@v1.80.0/internal/transport/http2_client.go:1663 +0x14c fp=0x4000a49fb0 sp=0x4000a49e40 pc=0x730c7c google.golang.org/grpc/internal/transport.NewHTTP2Client.gowrap4() google.golang.org/grpc@v1.80.0/internal/transport/http2_client.go:411 +0x28 fp=0x4000a49fd0 sp=0x4000a49fb0 pc=0x727f18 runtime.goexit({}) runtime/asm_arm64.s:1268 +0x4 fp=0x4000a49fd0 sp=0x4000a49fd0 pc=0x92984 created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 259 google.golang.org/grpc@v1.80.0/internal/transport/http2_client.go:411 +0x157c goroutine 217 gp=0x4000c24700 m=nil [semacquire, 318 minutes]: runtime.gopark(0x674d2a0?, 0x8a34c?, 0x40?, 0x82?, 0x8a310?) runtime/proc.go:460 +0xc0 fp=0x40000e7b80 sp=0x40000e7b60 pc=0x8aa10 runtime.goparkunlock(...) runtime/proc.go:466 runtime.semacquire1(0x40010e67a8, 0x0, 0x1, 0x0, 0x13) runtime/sema.go:192 +0x204 fp=0x40000e7bd0 sp=0x40000e7b80 pc=0x6b0f4 internal/poll.runtime_Semacquire(0xf6c53086b200?) runtime/sema.go:76 +0x2c fp=0x40000e7c10 sp=0x40000e7bd0 pc=0x8bf6c internal/poll.(*FD).Close(0x40010e6780) internal/poll/fd_unix.go:114 +0x64 fp=0x40000e7c40 sp=0x40000e7c10 pc=0x109a74 net.(*netFD).Close(0x40010e6780) net/fd_posix.go:50 +0x40 fp=0x40000e7c70 sp=0x40000e7c40 pc=0x1de700 net.(*conn).Close(0x4000406c58) net/net.go:220 +0x30 fp=0x40000e7cb0 sp=0x40000e7c70 pc=0x1ed020 crypto/tls.(*Conn).Close(0x4001a39188?) crypto/tls/conn.go:1456 +0xe8 fp=0x40000e7d10 sp=0x40000e7cb0 pc=0x245748 google.golang.org/grpc/internal/credentials.(*syscallConn).Close(0x4000a4bdd8?) :1 +0x2c fp=0x40000e7d30 sp=0x40000e7d10 pc=0x6130dc google.golang.org/grpc/internal/transport.(*http2Client).Close(0x4000d8efc8, {0x41a5640, 0x400070f550}) google.golang.org/grpc@v1.80.0/internal/transport/http2_client.go:1047 +0x3e4 fp=0x40000e7ec0 sp=0x40000e7d30 pc=0x72c8b4 google.golang.org/grpc.(*addrConn).tearDown(0x40019fc008, {0x41a5640, 0x400070f550}) google.golang.org/grpc@v1.80.0/clientconn.go:1705 +0x328 fp=0x40000e7f60 sp=0x40000e7ec0 pc=0x76e538 google.golang.org/grpc.(*ClientConn).Close.func2(0x200a2c22746f6f52?) google.golang.org/grpc@v1.80.0/clientconn.go:1226 +0x54 fp=0x40000e7fb0 sp=0x40000e7f60 pc=0x76b6d4 google.golang.org/grpc.(*ClientConn).Close.gowrap1() google.golang.org/grpc@v1.80.0/clientconn.go:1227 +0x2c fp=0x40000e7fd0 sp=0x40000e7fb0 pc=0x76b64c runtime.goexit({}) runtime/asm_arm64.s:1268 +0x4 fp=0x40000e7fd0 sp=0x40000e7fd0 pc=0x92984 created by google.golang.org/grpc.(*ClientConn).Close in goroutine 72 google.golang.org/grpc@v1.80.0/clientconn.go:1224 +0x184 goroutine 274 gp=0x4000c248c0 m=nil [chan receive, 318 minutes]: runtime.gopark(0x400085e6b8?, 0x2b04c?, 0x10?, 0x0?, 0x33d0660?) runtime/proc.go:460 +0xc0 fp=0x400085e680 sp=0x400085e660 pc=0x8aa10 runtime.chanrecv(0x40002d04d0, 0x400085e780, 0x1) runtime/chan.go:667 +0x428 fp=0x400085e700 sp=0x400085e680 pc=0x22ba8 runtime.chanrecv2(0x4204d70?, 0x4000e3b400?) runtime/chan.go:514 +0x14 fp=0x400085e730 sp=0x400085e700 pc=0x22764 google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x4000a84670, {0x4204d70, 0x4000e3b400}) google.golang.org/grpc@v1.80.0/internal/grpcsync/callback_serializer.go:88 +0xcc fp=0x400085e7a0 sp=0x400085e730 pc=0x646acc google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer.gowrap1() google.golang.org/grpc@v1.80.0/internal/grpcsync/callback_serializer.go:52 +0x2c fp=0x400085e7d0 sp=0x400085e7a0 pc=0x64690c runtime.goexit({}) runtime/asm_arm64.s:1268 +0x4 fp=0x400085e7d0 sp=0x400085e7d0 pc=0x92984 created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 72 google.golang.org/grpc@v1.80.0/internal/grpcsync/callback_serializer.go:52 +0x10c r0 0x6743c50 r1 0x80 r2 0x0 r3 0x0 r4 0x0 r5 0x0 r6 0x3 r7 0x0 r8 0x62 r9 0x0 r10 0x0 r11 0x0 r12 0x0 r13 0x0 r14 0x0 r15 0x0 r16 0xffffffbf4ab0 r17 0x0 r18 0x0 r19 0x400007a0c0 r20 0xffffffc04540 r21 0x6743b00 r22 0x4000004000 r23 0x10 r24 0x4000726310 r25 0x40000a2d08 r26 0x3c979a8 r27 0x668f000 r28 0x67427c0 r29 0xffffffc04528 lr 0x50688 sp 0xffffffc04530 pc 0x93dec fault 0x0 ### References _No response_ ### Generative AI / LLM assisted development? _No response_