It depends on what you're trying to do, and how far apart the releases are, and which product is newer.
Generally a newer Desktop client will be able to interact with an older server instance, provided it isn't too old. The documentation includes compatibility matrix tables; the 10.5 doc includes this FAQ.
Keep in mind that significant implementation changes were made between ArcGIS 10.0sp5 and 10.1.0, so ArcGIS 10.1 is likely to be the limit of reverse compatibility.
It also depends on if you are talking about being installed on the same machine (then it is highly recommended) or on different machines and just interacting....which is more forgiving.
Something to keep in mind, April, if you decide to install Desktop on the server that's hosting ArcGIS Server, you could negatively impact performance when geoprocessing/doing stuff in Desktop.
I am not planning on installing Desktop on the Server machine, but rather looking at installing the GeoEvent Extension. All of our Desktop versions are 10.3.1 and we cannot upgrade those due to a software collaboration. Our Server version is also 10.3.1. Does that mean I should install the 10.3 version of GeoEvent?
Yes, you'll need to stick with Geoevent 10.3.1. GEE needs to be installed on a machine that also hosts ArcGIS Server, and you won't be able to mix versions of ArcGIS Server software on the same machine.
GeoEvent Extension needs to be same version as AGS, it is an AGS extension. There is no need to have Desktop and AGS/Portal as the same version. We publish to 10.5 AGS from different versions of Desktop going back to 10.3.1.
Unless there is a reason you cannot (i.e., in production with ArcGIS Server) I would stay with the latest version of AGS/Portal (aka ArcGIS Enterprise)