Contents
Hello !
Little trouble yesterday when wanting to start a VM I encountered the following error: A general system error occurred: Connection refused
After a few minutes of research especially in the VMware KB I applied the few recommendations found :
Check the status of the vmware-vpx-workflow service in your vCenter console
First of all, check the status of the vmware-vpx-workflow service by logging in ssh to your vCenter appliance.
|
1 |
vcsa-vdays:~ # service-control --status vmware-vpx-workflow |
|
1 2 3 4 5 6 7 8 9 |
vcsa-vdays:~ # service-control --status vmware-vpx-workflow INFO:root:Service: vmware-vpx-workflow, Action: status Service: vmware-vpx-workflow, Action: status 2018-04-04T12:54:02.019Z Running command: ['/sbin/service', u'vmware-vpx-workflow', 'status'] 2018-04-04T12:54:02.098Z Done running command INFO:root:Stopped: vmware-vpx-workflow (VMware vCenter Workflow Manager) Stopped: vmware-vpx-workflow (VMware vCenter Workflow Manager) |
If the service is stopped, try restarting it:
|
1 |
vcsa-vdays:~ # service-control --start vmware-vpx-workflow |
If the service start fails with this error message:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 |
vcsa-vdays:~ # service-control --start vmware-vpx-workflow INFO:root:Service: vmware-vpx-workflow, Action: start Service: vmware-vpx-workflow, Action: start 2018-04-03T13:00:52.415Z Running command: ['/sbin/chkconfig', u'vmware-vpx-workflow'] 2018-04-03T13:00:52.453Z Done running command 2018-04-03T13:00:52.453Z Running command: ['/sbin/service', u'vmware-vpx-workflow', 'status'] 2018-04-03T13:00:52.513Z Done running command 2018-04-03T13:00:52.513Z Running command: ['/sbin/chkconfig', '--force', u'vmware-vpx-workflow', 'on'] 2018-04-03T13:00:52.549Z Done running command 2018-04-03T13:00:52.549Z Running command: ['/sbin/service', u'vmware-vpx-workflow', 'start'] 2018-04-03T13:01:08.572Z Done running command 2018-04-03T13:01:08.572Z Invoked command: ['/sbin/service', u'vmware-vpx-workflow', 'start'] 2018-04-03T13:01:08.573Z RC = 1 Stdout = VMware vCenter workflow manager is not running. workflow port = 8088 Waiting for vpxd to initialize Starting VMware vCenter workflow manager... wrapper | An encoding declaration is missing from the top of configuration file, /usr/lib/vmware-vpx/workflow/wrapper/conf/wrapper.conf, trying the system encoding. wrapper | An encoding declaration is missing from the top of configuration file, /etc/vmware/wrapper_common.conf, trying the system encoding. wrapper | An encoding declaration is missing from the top of configuration file, /etc/vmware/wrapper_linux.conf, trying the system encoding. wrapper | An encoding declaration is missing from the top of configuration file, /usr/lib/vmware-vpx/workflow/wrapper/conf/wrapper-classpath.conf, trying the system encoding. wrapper | Spawning intermediate process... Waiting for VMware vCenter workflow manager.................. WARNING: VMware vCenter workflow manager may have failed to start. Last login: Tue Apr 3 12:55:57 UTC 2018 Workflow start failed. Error code = 1 Stderr = 2018-04-03T13:01:08.573Z { "resolution": null, "detail": nStderr: " ], "id": "install.ciscommon.command.errinvoke", "localized": "An error occurred while invoking external command : 'Command: nStderr: '", "translatable": "An error occurred while invoking external command : '%(0)s'" } ], "componentKey": null, "problemId": null } ERROR:root:Unable to start service vmware-vpx-workflow, Exception: { "resolution": null, "detail": , "id": "install.ciscommon.service.failstart", "localized": "An error occurred while starting service 'vmware-vpx-workflow'", "translatable": "An error occurred while starting service '%(0)s'" } ], "componentKey": null, "problemId": null } Unable to start service vmware-vpx-workflow, Exception: { "resolution": null, "detail": , "id": "install.ciscommon.service.failstart", "localized": "An error occurred while starting service 'vmware-vpx-workflow'", "translatable": "An error occurred while starting service '%(0)s'" } ], "componentKey": null, "problemId": null } |
There is a problem elsewhere.
Check used space of vCenter partitions
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
vcsa-vdays:/var/log/audit # df -h Filesystem Size Used Avail Use% Mounted on /dev/sda3 11G 5.6G 4.7G 55% / udev 7.9G 164K 7.9G 1% /dev tmpfs 7.9G 40K 7.9G 1% /dev/shm /dev/sda1 128M 41M 81M 34% /boot /dev/mapper/core_vg-core 50G 185M 47G 1% /storage/core /dev/mapper/log_vg-log 9.9G 9.9G 0G 100% /storage/log /dev/mapper/db_vg-db 9.9G 884M 8.5G 10% /storage/db /dev/mapper/dblog_vg-dblog 5.0G 571M 4.2G 12% /storage/dblog /dev/mapper/seat_vg-seat 25G 4.4G 19G 19% /storage/seat /dev/mapper/netdump_vg-netdump 1001M 18M 932M 2% /storage/netdump /dev/mapper/autodeploy_vg-autodeploy 9.9G 151M 9.2G 2% /storage/autodeploy /dev/mapper/invsvc_vg-invsvc 9.9G 1.5G 7.9G 17% /storage/invsvc |
In my case the partition of log was full thus causing problems to generate the various logs necessary for the good operation of the vCenter.
To make some space, you can delete the logs of type “java_error12049.hprof”:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
vcsa-vdays:/var/log/vmware/vsphere-client # ls -l total 280 -rw------- 1 vsphere-client users 0 Mar 30 22:27 java_error12049.hprof drwxr-xr-x 3 vsphere-client root 4096 Apr 4 09:56 logs -rw------- 1 vsphere-client users 0 Apr 2 11:46 vsphere-client-gc.log.0 -rw------- 1 vsphere-client users 102071 Apr 4 13:11 vsphere-client-gc.log.0.current -rw------- 1 vsphere-client users 1957 Apr 4 09:51 vsphere-client-gc.log.1.current -rw------- 1 vsphere-client users 0 Mar 9 08:19 vsphere-client-gc.log.2.current -rw------- 1 vsphere-client users 0 Mar 9 06:10 vsphere-client-gc.log.3 -rw------- 1 vsphere-client users 0 Mar 9 06:25 vsphere-client-gc.log.4 -rw------- 1 vsphere-client users 0 Mar 9 06:40 vsphere-client-gc.log.5 -rw------- 1 vsphere-client users 0 Mar 9 06:56 vsphere-client-gc.log.6 -rw------- 1 vsphere-client users 0 Mar 9 07:11 vsphere-client-gc.log.7 -rw------- 1 vsphere-client users 0 Mar 9 07:26 vsphere-client-gc.log.8 -rw------- 1 vsphere-client users 0 Mar 9 07:42 vsphere-client-gc.log.9 -rw------- 1 vsphere-client users 167185 Apr 4 10:02 wrapper.log -rw------- 1 vsphere-client users 0 Apr 3 12:26 wrapper.log.1 |
|
1 |
vcsa-vdays:/var/log/vmware/vsphere-client # rm -rf *.hprof |
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
vcsa-vdays:/var/log/vmware/vsphere-client # df -h Filesystem Size Used Avail Use% Mounted on /dev/sda3 11G 5.6G 4.7G 55% / udev 7.9G 164K 7.9G 1% /dev tmpfs 7.9G 40K 7.9G 1% /dev/shm /DEV/SDA1 128M 41M 81M 34%/boot /dev/mapper/core_vg-core 50g 185M 47G 1%/storage/core /dev/mapper/log_vg-log 9.9 g 7.9 g 2g 80%/storage/log /dev/mapper/db_vg-db 9.9 g 884M 8.5 g 10%/storage/db /dev/mapper/dblog_vg-dblog 5.0 g 571M 4.2 g 12%/storage/dblog /dev/mapper/seat_vg-seat 25g 4.4 g 19G 19%/storage/seat /dev/mapper/netdump_vg-netdump 1001 m 18m 932M 2%/storage/netdump /dev/mapper/autodeploy_vg-autodeploy 9.9 g 151M 9.2 G 2%/storage/autodeploy /dev/mapper/invsvc_vg-invsvc 9.9 g 1.5 g 7.9 g 17%/storage/invsvc |
To go further, we can also reset to 0 the logs here:
|
1 |
VCSA-vdays:/var/log/VMware/CLOUDVM # :> CLOUDVM-ram-size. Log |
and re-check the available disk space:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
VCSA-vdays:/var/log/VMware/CLOUDVM # DF-H Filesystem Size Used Avail Use% mounted on /dev/sda3 11g 5.6 g 4.7 g 55%/ Udev 7.9 g 164K 7.9 g 1%/dev Tmpfs 7.9 g 40k 7.9 g 1%/DEV/SHM /DEV/SDA1 128M 41M 81M 34%/boot /dev/mapper/core_vg-core 50g 185M 47G 1%/storage/core /dev/mapper/log_vg-log 9.9 G 6.9 g 3g 70%/storage/log /dev/mapper/db_vg-db 9.9 g 884M 8.5 g 10%/storage/db /dev/mapper/dblog_vg-dblog 5.0 g 571M 4.2 g 12%/storage/dblog /dev/mapper/seat_vg-seat 25g 4.4 g 19G 19%/storage/seat /dev/mapper/netdump_vg-netdump 1001 m 18m 932M 2%/storage/netdump /dev/mapper/autodeploy_vg-autodeploy 9.9 g 151M 9.2 G 2%/storage/autodeploy /dev/mapper/invsvc_vg-invsvc 9.9 g 1.5 g 7.9 g 17%/storage/invsvc |
Cool cool! We have won some GB ! :).
Our service vmware-vpx-workflow is always stop I remind you!
By “security” and because there can be dependencies between services, it is better to stop all services and let VMware restart them in the right order, as is necessary. This has no impact on your production, it will just make access to the vSphere Client and vSphere Web Client console inaccessible.
|
1 |
VCSA-vdays:/var/log/VMware/CLOUDVM # Service-Control--stop--all |
|
1 |
VCSA-vdays:/var/log/VMware/CLOUDVM # Service-Control--start--all |
If you are fast enough, you will see the start of the service scroll:):
|
1 2 3 4 5 6 7 8 |
INFO: Root: Service: VMware-vpx-Workflow, Action: status Service: VMware-vpx-Workflow, Action: status 2018-04-04T12:54:02.019 z Running command: ['/sbin/service', u'vmware-vpx-workflow', 'status'] 2018-04-04T12:54:02.098 z Done running command INFO: Root: Running: VMware-VPX-Workflow (VMware vCenter workflow Manager) Running: VMware-VPX-Workflow (VMware vCenter workflow Manager) |
We’re good, with an up & running service! 🙂
Managing Audit logs
We can go a little further. If like me, you have set a root password when installing VCenter, and you never came back on it then it may be expired. In fact, by default, the root password for the VCSA appliance expires every 365 days. If you don’t change it, then it collects a certain amount of error and makes the log audit bigger.
So at first you would have to change your root password if it has expired, or else make sure it does not expire. You can do this by connecting to this address:
- https://yourvcsa.tld:5480
Then we return to the VCenter in SSH and then restart the rotation of the logs that stopped at the end of the validity period of the password root.
|
1 |
VCSA-Vdays:/etc # logrotate-F/etc/logrotate.conf |
Go check that the new logs of the day are well generated and archived:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 |
VCSA-vdays:/var/log/Audit # LS-LH Total 34M drwx------2 root root 4.0 k Mar 21 2017 audispd -RW-------1 root root 5.3 m APR 4 15:01 Audit. Log -RW-------1 root root 649K Jan 10 18:15 audit. Log-20180110. bz2 -RW-------1 root root 652K Jan 11 18:15 audit. Log-20180111. bz2 -RW-------1 root root 653K Jan 12 18:15 audit. Log-20180112. bz2 -RW-------1 root root 652K Jan 13 18:15 audit. Log-20180113. bz2 -RW-------1 root root 652K Jan 14 18:15 audit. Log-20180114. bz2 -RW-------1 root root 651K Jan 15 18:15 audit. Log-20180115. bz2 -RW-------1 root root 652K Jan 16 18:15 audit. Log-20180116. bz2 -RW-------1 root root 647K Jan 17 18:15 audit. Log-20180117. bz2 -RW-------1 root root 650K Jan 18 18:15 audit. Log-20180118. bz2 -RW-------1 root root 650K Jan 19 18:15 audit. Log-20180119. bz2 -RW-------1 root root 647K Jan 20 18:15 audit. Log-20180120. bz2 -RW-------1 root root 648K Jan 21 18:15 audit. Log-20180121. bz2 -RW-------1 root root 648K Jan 22 18:15 audit. Log-20180122. bz2 -RW-------1 root root 648K Jan 23 18:15 audit. Log-20180123. bz2 -RW-------1 root root 0 Apr 4 10:11th audit. Log-20180404 -RW-------1 root root 20m Apr 4 10:07 Audit. Log-20180404. bz2 drwx------2 root root 4.0 k Jan 23 2017 AUDITD |
If you ever have a log too big you can delete it or reset it to zero:
|
1 |
VCSA-vdays:/var/log/Audit #: > audit. log-20180404 |
Hopefully it will have been helpful!
M.
Go further
Here some KB Vmware to go further in setting up the rotation of the logs:
Hi ! I’m Maxime. Founder and independant author of vDays.net. I have worked in service IT since 6 years ago, after a 5 years’ internship. Via this blog, I would like share and discuss with you on new technologies, especially on virtualization and VMware. If you want to know more about me, check out my “about me” page or follow me on Twitter and LinkedIn
Good post, thank you.
You’re welcome !