Here’s a quick tech tip; if you have a macOS (or OS X) virtual machine that you can’t seem to set to a reasonably high resolution, it’s probably because scaled Retina resolution is enabled. The telltale sign is that the actual resolution you see, when selecting the Scaled option in Display preferences, is exactly half of what you’re setting with the vmware-resolutionSet utility, followed by “HiDPI”. The other sign is that while you’re able to see an apparently higher resolution in the Scaled resolution list, selecting it has no effect (though you may see the display momentarily go to a higher resolution and then revert).
Disable scaled Retina resolution by opening Terminal.app and running this command:
sudo defaults write /Library/Preferences/com.apple.windowserver.plist DisplayResolutionEnabled -bool false
Reboot, then set your desired resolution with this command (for example):
sudo /Library/Application\ Support/VMware\ Tools/vmware-resolutionSet 1680 1050
If, on the other hand, you’re looking to take advantage of a HiDPI display on your host, simply replace “false” with “true” in the first line and set a resolution of exactly half of the desired resolution (on each axis). Make sure you reboot after changing the DisplayResolutionEnabled property.