The whole being based on the Server 2003 kernel thing is what really made that annoying. That's not really a problem today between desktop and server releases for most stuff; the differences you find are usually artificial, like Microsoft not supporting consumer-grade wired NICs (wireless are fine!) in Windows Server, and AMD not supporting Windows Server with GPU drivers for their consumer line
at all.
But since the fundamental architecture is exactly the same, and since Microsoft has done a ton of work to keep drivers from being so consequential to the OS in the first place, companies really have to go out of their way to prevent functionality.
XP 64, on the other hand, required one to use stuff supported by Server 2003, and wasn't a development target for a lot of consumer-oriented stuff like games. I applaud your courage for going down that route though
.
What's funny is that 7 and Vista at the second service pack were essentially the same. Some features were gated to 7, but generally speaking the complaints about stability and performance were addressed. And I'm still kind of annoyed that Microsoft abandoned the idea of sidebars and gadgets.
7 existed largely in the same vein that 10 does today; Vista and 8 both earned the ire of consumers, so their successors were largely marketing-driven.
And as I'm typing on Fedora 33 with the default Gnome desktop environment, honestly Microsoft has done a pretty good job with the basic desktop experience, at least with 10 Pro.