Loading...
Loading...
Filesystem on datastore server where Postgres is located is almost full. Application service starts successfully but fails to process data and soon crashes. The following message was found in the server.log on the Application server: 2022-11-15 08:41:00,700 WARN [org.jboss.jca.core.connectionmanager.listener.TxConnectionListener] (ReportEngineMessageListener #1:Backup Report Card:22b4e14e-a7a9-4b49-a244-54336df9661f) IJ000305: Connection error occured: org.postgresql.util.PSQLException: ERROR: could not create temporary file "base/pgsql_tmp/pgsql_tmp921.5.sharedfileset/i165of8192.p0.0": No space left on device An error followed the above warning: 2022-11-15 08:41:00,704 ERROR [com.emc.apollo.jdbc.exec.SQLExecutor] (ReportEngineMessageListener #1:Backup Report Card:22b4e14e-a7a9-4b49-a244-54336df9661f) Error executing query: Error. Query was: SELECT t0.f_agent_name as "t0f_agent_name", t0.f_sub_name as "t0f_sub_name", t0.f_group_name as "t0f_group_name", t0.f_schedule_name as "t0f_schedule_name", t0.f_client_name as "t0f_client_name", t0.f_policy_name as "t0f_policy_name", t0.f_workflow_name as "t0f_workflow_name", t0.f_workflow_jobid as "t0f_workflow_jobid", t0.f_action_name as "t0f_action_name", t0.f_job_name as "t0f_job_name", t0.f_domain_name as "t0f_domain_name", t0.f_backup_set as "t0f_backup_set", t0.f_session as "t0f_session", t0.f_proxy as "t0f_proxy", t0.f_queuestart as "t0f_queuestart", t0.f_status as "t0f_status", t0.f_starttime as "t0f_starttime", t0.f_endtime as "t0f_endtime", t0.f_ntries as "t0f_ntries", t0.f_queuestart AS f_reporterstarttime, t0.f_queuestart AS f_reporterendtime, t0.f_id as f_id FROM view_backupjob t0.... In the Datastore logs, similar messages will be found: 2022-11-15 07:29:49.001 GMT [7443] ERROR: could not write to file "base/pgsql_tmp/pgsql_tmp4401.6.sharedfileset/i19081of32768.p1.0": No space left on device Checking on the Datastore server in <install_dir>/dpa/services/datastore/data/base/pgsql_tmp directory, there are numerous folders and files making up many GBs in size. The date on the first files in the pgsql_tmp directory may correspond to a recent upgrade of the DPA software.
During the recent upgrade, the DPA Datastore was not tuned correctly for the amount of RAM available on the server. Running the following command shows the following: # dpa.sh ds tune EMC Data Protection Advisor [INFO] Current Datastore Service tuned expecting 512MB of host memory Command completed successfully. The 512MB value is typical of default settings for PostgreSQL server. This low memory setting caused DPA to write query information to temporary files in pgsql_tmp. When the filesystem became full, the system killed the query which left behind the tmp files that eventually filled up all the free space on the filesystem. This caused the error in the server.log relating to a SELECT query being unable to create temporary file in pgsql_tmp directory.
Run the tune command on the Datastore server for the amount of RAM on the server. For example, if the server is configured with 64GB of RAM, run the command: # dpa.sh ds tune 64GB EMC Data Protection Advisor Set Datastore service memory usage to 65536MB : Confirm [Y|N] y Datastore Service successfully tuned. The new memory utilisation for the datastore will not take effect until the datastore service is restarted. Command completed successfully. Restart the Datastore service. Upon restart, the files in pgsql_tmp will be automatically cleared by postgresql process. Monitor the latest Datastore service log, when the following message appears, the Application can then be started. 2022-11-15 09:19:10.003 GMT [10897] LOG: database system is ready to accept connections With more memory tuned for the Datastore service, there will be less requirement to write query data to temporary files in pgsql_tmp. Please contact Dell Technical Support for further information.
Click on a version to see all relevant bugs
Dell Integration
Learn more about where this data comes from
BugZero Plan
Streamline upgrades with automated vendor bug scrubs
BugZero Prevent
Wish you caught this bug sooner? Get proactive today.