I have recently faced a problem with VNC after upgrading my home Ubuntu machine to 20.04. I have started getting below encryption error which I had to troubleshoot.

"Unable to connect to VNC Server using your chosen security setting. Either upgrade VNC Server to a more recent version from RealVNC, or select a weaker level of encryption."

Solution :

While troubleshooting this I found out that the server was suggesting TLS security after which the client closes the connection in wireshark packet capture. Wireshark made it really easy for me to jump to the fix which was to disable TLS/encryption need at server side.

For that I had to download dconf-editor on my ubuntu machine. In dconf-editor we need to disable encryption at server side for that…

Open dconf-editor >>org>>gnome>>desktop>>remote session >> require-encryption

See below :

In case you don’t have dconf-editor them you can download it with below command.

#sudo apt-get install dconf-editor

Please checkout the youtube video to understand this better.