Issue

What were you trying to do that didn't work? One of our customer in a supporrt case is facing performance issue in his application once they have tested their product on RHEL 9 , compared to CentOS 7 On RHEL 9 the bash process is consuming high amount of CPU and is taking more time to complete the same task it used to do on CentOS 7. They were further able to isolate the cause to the changes which happened in bash-5.0 and the issue is solved in bash-5.2 according to customer tests They found that this happens because they are using large number of "environment variables". They say that the particular change that's affecting them is a change to increment SHLVL when executing a command in a pipe, which causes bash to rebuild its array of exported environment variables every time that happens. In their case we have ~1.4k such variables so rebuilding the array is a nontrivial amount of work (partly because bash does a bunch of seemingly unnecessary deduplication checking). See the following for discussion of this issue in the bash mailing lists. https://lists.gnu.org/archive/html/help-bash/2020-10/msg00018.html https://lists.gnu.org/archive/html/bug-bash/2022-06/msg00091.html The relevant code is visible at https://github.com/bminor/bash/blob/bash-5.1/execute_cmd.c#L5487. They have checked that making flipping that "#if 0" to "#if 1" makes the performance comparable to bash 4.4 or 5.2. Test perform by customer on their systems ~~~ for exe in bash-*; do duration=$(/usr/bin/time -f "%Us" ./$exe perf-test.sh 1000 2>&1); echo "$exe: $duration"; done bash-4.2: 1.67s bash-4.4: 1.58s bash-5.0: 15.06s bash-5.1: 14.61s bash-5.2: 1.66s ~~~ Customer have provided a reproducer script which can be used to test the performance of the time taken to run bash. See the attached bash-perf-repro.tgz file for the repro scripts. This contains 3 files, perf-test.sh to run some commands in pipeline in a loop env.sh to set the environment variables run-test.sh to launch perf-test.sh in a clean environment and time it. To use run ./run-test.sh <bash_exe> <iterations>, where <bash_exe> is the bash executable to test and <iterations> is the number of times perf-test.sh should launch a pipeline. e.g. to run a test with 1000 iterations using the bash executable on the path run ./run-test.sh bash 1000. I had tested it on different individual installation of RHEL 7 (bash-4.2) , RHEL 8 (bash-4.4) and RHEL 9 (bash-5.1) having the same resource configuration. I see that the old version of RHEL 7 and RHEL 8 ver much faster while the RHEL 9 does take significantly more time. I tested by downloading the latest "bash-5.2" from upstream "https://ftp.gnu.org/gnu/bash/bash-5.2.tar.gz" and can see that version 5.2 does have improvement to performance. RHEL 7 ~~~ ./run-test.sh /bin/bash 1000 Red Hat Enterprise Linux Server release 7.9 (Maipo) 4.2.46(2)-release real 0m2.177s user 0m0.718s sys 0m1.241s ~~~ RHEL 8 ~~~ ./run-test.sh /bin/bash 1000 Red Hat Enterprise Linux release 8.9 (Ootpa) 4.4.20(1)-release real 0m3.565s user 0m1.561s sys 0m1.996s ~~~ RHEL 9 ~~~ ./run-test.sh /bin/bash 1000 Red Hat Enterprise Linux release 9.3 (Plow) 5.1.8(1)-release real 0m7.609s user 0m6.494s sys 0m1.318s ~~~ RHEL 9 (with bash version 5.2 from upstream) ~~~ ./run-test.sh /usr/local/build/bash-5.2/bash 1000 Red Hat Enterprise Linux release 9.3 (Plow) 5.2.0(1)-release real 0m2.137s user 0m1.183s sys 0m1.196s ~~~ Additional information provided by rhn-support-pandrade  about this issue. Problem in bash 5.1: https://github.com/bminor/bash/blame/bash-5.1/execute_cmd.c#L5487 No problem in bash 5.2: https://github.com/bminor/bash/blame/bash-5.2/execute_cmd.c#L5636 Please provide the package NVR for which bug is seen: ~~~ rpm -q bash bash-5.1.8-6.el9_1.x86_64 echo $BASH_VERSION 5.1.8(1)-release ~~~ How reproducible: Every time on bash 5.1 on RHEL 9 Steps to reproduce Download and extract the "bash-repro.tgz" To use run ./run-test.sh <bash_exe> <iterations>, where <bash_exe> is the bash executable to test and <iterations> is the number of times perf-test.sh should launch a pipeline. e.g. to run a test with 1000 iterations using the bash executable on the path run ./run-test.sh bash 1000. Expected results Bash is taking a long time on bash 5.1 on RHEL 9 when executing the script with large number of environment variables. Actual results It should complete with time comparable to old version of bash 4.4 form RHEL 8 and bash 4.2 from RHEL 7

Release Notes

No. of Comments

Resolution

Done-Errata