Xorg uses GPU memory
Solution: comment out everything in /usr/share/X11/xorg.conf.d/10-nvidia.conf and then restart the display manager with:
sudo systemctl restart display-managernvidia-smi takes too long to load
One of the reasons could be that persistence mode is disabled. To enable it, run:
sudo nvidia-smi -pm 1