"503 Service Unavailable" Fault on the vSphere Web Client: What Should You lot Practise?

Subscribe banner

VMware vCenter Server is a centralized vSphere management system that allows yous to manage ESXi hosts, clusters, virtual machines, and other components of a virtual data center. Sometimes, when yous endeavour to open VMware vSphere Client to connect to vCenter, you get the 503 Service Unavailable error. This weblog mail service explains the possible reasons that can cause this error and the methods of fixing the vCenter 503 Service Unavailable error.

General Information

Earlier covering troubleshooting, you should sympathise what the 503 error is. 503 Service Unavailable is ane of the standard HTTP status codes that are returned past a server when a client sends a request to the server. The 5xx class of HTTP status codes is used to notify users most server errors. Based on this, 503 Service Unavailable is a response that describes a server-side error. Usually this error indicates that a server cannot handle a request and yous should look for the result on the web server hosting an application.

Hither'southward the full text of this error for VMware vSphere Client:

503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http20NamedPipeServiceSpecE:0x000055615a4d7a70] _serverNamespace = / activeness = Let _pipeName =/var/run/vmware/vpxd-webserver-pipe)

Full error message vCenter 503 Service Unavailable (Failed to connect to endpoint

In the example of the vCenter 503 Service Unavailable error, the error tin be caused by different factors:

  • Some services accept non loaded yet and are non running (one of the common issues is that the reverse Proxy service or the vSphere web client service is down).
  • vCenter Server is on maintenance
  • vCenter Server is overloaded
  • Disk issues occurred on vCenter Server
  • At that place are problems with a database used by vCenter
  • Incorrect firewall settings (for vCenter installed on Windows) and network issues

Yous should find and gear up the fault on the vCenter server and not on the customer side.

Services Have Not Started

If you accept just started a vCenter case, and all the needed services have not started yet, expect for a few minutes and try once again.

Endeavour connecting to vCenter from another client.

If this doesn't assist, check whether all needed services take started.

Open up the Appliance Management Interface (for vCenter Server Apparatus) in a spider web browser by entering the IP address or the FQDN (a Fully Qualified Domain Proper noun) for vCenter and port 5480, for example:

https://192.168.101.103:5480/ui/services

Check the status of vCenter services in the Services department. For example, if the vSphere Web Customer service is Started, and the vCenter Server service is Stopped, the VMware 503 Service Unavailable error can occur.

Fixing the 503 Service Unavailable error – checking running services

You can cheque the status of vCenter services in the panel and the control line interface (CLI). Enable SSH login, console CLI, and Bash Trounce in the Access section of the vCenter Appliance Management Interface.

SSH login and Bash Shell are enabled in VCSA management settings

Every bit an alternative, you can enable Bash Shell and SSH admission in the DCUI (Straight Panel User Interface) of vCenter Server Appliance (VCSA) in the Troubleshooting Mode options.

Enabling SSH access in Troubleshooting Mode options in the DCUI of VCSA

Ensure that there is a connection betwixt your computer and vCenter server past running the command from your computer (enter the host proper noun or IP address of your vCenter):

telnet vcenter_fqdn 9443

Connect to vCenter via SSH, or connect to the VCSA console (Printing Alt+F1 in the DCUI of vCenter, log in, type trounce, and press Enter).

Check the condition of vCenter services by running the command on the machine running vCenter:

service-command --status --all

Checking status of vCenter services to find a cause of the 503 Service Unavailable error

If there is an external Platform Service Controller (PSC), check vCenter connexion to the PSC. Bank check services on the PSC also.

If some services have been terminated, run all services with the command:

service-control --start --all

Effort to stop and start the vSphere Client service:

service-control --stop vsphere-client

service-control --start vsphere-client

Expect nigh 10–15 minutes until the service starts, then endeavor to access vCenter in vSphere Client.

Check Log Files

Checking log files can assistance you lot notice the cause of errors including the vCenter 503 Service Unavailable error. Beginning, investigate the vsphere_client_virgo.log file.

For the Windows version of vCenter Server, logs are stored in:

C:\ProgamData\VMware\vCenterServer\logs\vsphere-customer\logs\

For vCenter Server Appliance, vCenter logs are stored in:

/var/log/vmware/vsphere-customer/logs/

Besides check the vpxd.log file.

On vCenter running on Windows, vpxd.log is stored in:

C:\ProgramData\VMware\vCenterServer\logs\vmware-vpx\

On vCenter Server Appliance, vpxd.log is stored in:

/var/log/vmware/vpxd/

Non Enough Free Hardware Resources

Insufficient disk space doesn't cause the vCenter 503 Service Unavailable error itself. However, some services cannot start if in that location is no free disk space, and this may cause the 503 Service Unavailable error. Loftier storage latency for vCenter may also be a reason of this error. Kickoff from VCSA 6.5, the apparatus is provided with 12 virtual disks (there were 2 virtual disks for VCSA earlier), and it's possible that something may go wrong with one of the virtual disks. If vCenter Server is overloaded, the busy server cannot handle your request.

Cheque free deejay space, and run the command on the vCenter Server side. If yous use vCenter Server Appliance, run this command:

df -h

The /storage/archive/ partition tin can be full in VCSA 6.seven, and this is an expected (normal) situation based on VMware's pattern of vCenter half dozen.7.

Check gratis memory and processor resources:

top

Check the file organisation on the auto running vCenter. Utilise e2fsck to cheque the file system on VCSA, and run the command e2fsck -y filesystem , for instance:

e2fsck -y /dev/sda1

If errors are found and fixed, reboot the vCenter Server:

reboot

Issues with a vCenter Database

A database is an of import component required for the proper performance of vCenter Server. If a database is not working, vCenter errors, including "vCenter 503 Service Unavailable", may occur. Let's explore possible database issues and methods of fixing them.

Sometimes the following errors are displayed in the vpxd log file:

An unrecoverable trouble has occurred, stopping the VMware VirtualCenter service. Error: Error[VdbODBCError] (-1) "ODBC mistake: (23505) - Mistake: duplicate key value violates unique constraint "pk_vpx_vm_virtual_device";

Error while executing the query" is returned when executing SQL statement "INSERT INTO VPX_VM_VIRTUAL_DEVICE

If your vCenter is installed on Windows, bank check the Event viewer to see awarding event logs.

This is a bug due to duplicated entries in the embedded Postgres database that is used for the proper performance of vCenter.

Open the command line interface of vCenter Server Appliance, and, for example, connect to the server via SSH.

Get to the Postgres log directory:

cd /var/log/vmware/vpostgres

Check the log files. Log files are named in the postgresql-"dayofmonth".log format. For example, the log file for the third day of the month is named "postgresql-03.log". If today is the May three, check the contents of the appropriate log file:

cat postgresql-03.log

2020-05-03 16:05:12.749 UTC 38a12db9.5481 1008636 VCDB vc ERROR:  indistinguishable primal value violates unique constraint "pk_vpx_vm_virtual_device"

2020-05-03 16:05:12.749 UTC 38a12db9.5481 1008636 VCDB vc DETAIL:  Key (id, device_key)=(8101, 4002) already exists.

Recall the ID and DEVICE_KEY. In this case, these values are 8101 and 4002 respectively.

Connect to the embedded Postgres database used past vCenter:

/opt/vmware/vpostgres/electric current/bin/psql -d VCDB -U postgres

Delete the duplicated entry:

DELETE FROM vc.vpx_vm_virtual_device where id='8101′ and device_key='4002′;

"DELETE 1" in the output means that everything is OK.

Exit the database CLI:

\q

Type reboot to reboot your vCenter Server. Look until the vCenter Server and services accept started, and try to open vSphere Customer.

You may go the same error again afterward reboot. Check database logs over again. If the duplicate entries were created again, delete them and reboot vCenter. Yous may demand to repeat deleting duplicate entries and rebooting vCenter Server multiple times until the issue is resolved.

Sometimes you may need to detect the affected VM, remove the affected VM from the inventory, then re-register (add) the VM to the inventory.

If this method doesn't assist prepare the event permanently, try to upgrade your vCenter to vCenter 6.7 or newer.

If you apply the SQL database, check whether you take free disk space to store SQL database logs. If there is no complimentary space on the SQL database logs filesystem, you may get the error.

Bug with Permissions

If your vCenter is installed on Windows Server, sometimes issues with account permissions may occur.

Bank check the "Log on as a batch task" policy in a group policy editor, and make sure that all needed accounts are present. Lack of permissions on Windows Server where vCenter is installed may cause the vSphere 503 Service Unavailable error.

Open the local group policy editor in Windows (Start > Run > gpedit.msc).

In the group policy editor go to Computer Configuration > Windows Settings > Security Settings > Local Policies > User Rights Assignment > Log on as a batch job.

Add these user accounts to this policy if they are missing:

cm

content-library

eam

imagebuilder

mbcs

netdumper

perfcharts

rbd

vapiEndpoint

vmware-vpostgres

vsan-health

vsm

vsphere-client

vsphere-ui

Click Add User or Group to add accounts.

Editing permissions in the group policy editor

Finish and start the vSphere Customer service past using these commands:

service-control --stop vsphere-client

service-control --start vsphere-client

Outset the vSphere user interface service:

service-command --start vsphere-ui

Wait until the service is started.

Reboot vCenter Server, and check whether vCenter is working correctly.

Usually the needed accounts should be added past the installer of vCenter automatically until this policy is set manually in a grouping policy editor.

Network and DNS Problems

Make sure that a DNS name of vCenter is set correctly and the IP address of vCenter Server is resolved. If DNS has a incorrect entry, you may go the 503 Service Unavailable error. When you install vCenter, set up the DNS proper noun in the configuration at step seven (Configure network settings) of stage one.

Set the FQDN when you install vCenter to avoid 503 service unavailable (failed to connect to endpoint

If you lot are using an external PSC, the PSC IP address must be resolved in both directions – frontward lookup and reverse lookup.

Check the hostname in /etc/hosts on the vCenter Server Apparatus. Brand sure that a Fully Qualified Domain Proper name (FQDN) and a short name are present in this file in the following format:

vCenter_IP vCenter Server Appliance FQDN vcenter brusque proper noun

For example:

192.168.101.103 vCenter6-7.vsphere.local vCenter6-7

Subsequently changing the DNS configuration, restart the vpxd service:

service vmware-vpxd restart

Y'all tin check the status of the vpxd service with the command:

vpxd status service vmware-vpxd status

On the PSC run these commands to restart related services:

/etc/init.d/vmware-stsd restart

/etc/init.d/vmware-sts-idmd restart

service vmware-cm restart

Restart the vpxd service on the machine where vCenter is installed:

service vmware-vpxd restart

Wait until all needed services have started, and try to open VMware vSphere Client.

It is recommended that you use a vCenter deployment model with an embedded PSC to have less network issues. The latest versions of vCenter, like vCenter vii.0, merely support installation with an embedded PSC.

If you however get the vCenter 503 Service Unavailable mistake, endeavour the side by side method.

Try to enable or disable IPv6. Open /etc/sysctl.conf and change the net.ipv6.conf.eth0.disable_ipv6 from 1 to 0 to enable IPv6:

#Disabling SLAAC/Link Local addresses

net.ipv6.conf.eth0.disable_ipv6=0

Similarly, set 1 to disable IPv6.

Once you save settings, run the commands:

sysctl -a

sysctl -p

Open VMware vSphere Client now. Cheque whether everything is working properly after reboot.

Effort to disable SMBv1 and enable SMBv2 instead. Then restart vCenter Server and check whether this method has helped.

Password Issues

Check whether your root countersign for vCenter Server Appliance has expired. If it's expired, you won't be able to log in to vCenter to edit configuration and fix errors including the vCenter appliance 503 Service Unavailable error without a workaround. You lot tin can set the root password expiration period to 0 if you want a password to never expire. You can edit the root password expiration settings in the GUI of vCenter Server Apparatus Management:

https://your_vCenter_IP:5480/ui/administration

The root password expiration settings in vCenter Server Appliance Management configuration

If in that location is password mismatch in vmdird for the account specified in vmdird-syslog.log, this cistron may be a reason of getting the post-obit error:

503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http20NamedPipeServiceSpecE:0x7f3d084a60c0] _serverNamespace = / _isRedirect = simulated _pipeName =/var/run/vmware/vpxd-webserver-pipage)"

Check the /var/log/vmware/vmdird/vmdird-syslog.log file to identify which business relationship password is invalid and must be reset.

Support your vCenter earlier editing configuration or at least take a snapshot if your vCenter Server is running on a virtual machine. Do the same for the external Platform Service Controller if y'all are using one.

Log into vCenter Server Appliance equally root via SSH. Type trounce.prepare --enabled true and type Shell to access the Bash console.

Open the vdcadmintool tool by running the command:

/usr/lib/vmware-vmdir/bin/vdcadmintool

Select the third choice:

  1. Reset account password

Enter the name of the account, the password of which yous want to reset, according to the information in the vmdird-syslog.log file.

A new password is generated automatically. Copy and notation this countersign.

Note: If you lot see a blank graphic symbol in the password, generate a new countersign past going back to the previous step. This issue occurs if a password contains special characters, and they are displayed as missing (blank) characters.

Connect to vCenter Server via SSH as explained higher up (if yous are using an external PSC). If yous are using the embedded PSC, become to the next step.

Run the control to update the new countersign:

/opt/likewise/bin/lwregshell

cd HKEY_THIS_MACHINE\services\vmdir\

set_value dcAccountPassword "new password"

quit

Restart the vCenter Server Appliance services by using the commands:

service-control --stop --all

service-command --start --all

Perform vCenter Backup

Information technology is generally recommended that you back up vCenter afterward finishing configuration. You should do that later ensuring that vCenter is working correctly and earlier editing configuration to get in possible to restore vCenter if something goes wrong. The best approach is creating regular vCenter backups because vCenter configuration may change during operation and some items may be added to the vCenter inventory. You cannot predict when a failure will happen.

NAKIVO Backup & Replication is a universal data protection solution that tin back up VMware vSphere VMs and concrete machines running Windows and Linux, including SQL databases. Protect your vCenter Server machines with NAKIVO Fill-in & Replication, and, if something goes wrong with your vCenter, you tin can restore the working configuration in a brusque time. Support of incremental backup and the GFS retentiveness policy helps perform VMware vCenter fill-in more than rationally. The production supports SQL log truncation to save storage infinite and reduce the probability of errors acquired by insufficient deejay space with large database logs.

For seamless however reliable protection of your VMware environments, endeavor the Costless Edition of NAKIVO Backup & Replication, which delivers the Pro edition functionality at zero price for one year!

Conclusion

This blog post explained possible methods of fixing the VMware 503 Service Unavailable error in vCenter. This is a server-side error that can occur due to multiple reasons, such equally insufficient hardware resources, stopped services, database errors, network errors, and wrong permissions. Try to utilize the methods detailed in this blog post to resolve issues related to the 503 Service Unavailable mistake in your vSphere surroundings. You lot can also read these blog posts virtually vCenter installation and all-time practices.