Hello Simon,
I know I am a bit late to the discussion here but I wanted to provide a response in case someone else comes across this discussion. The issue with the 406 error looks to be due to the acceptable MIME type included in the HTTP request header.
By default, the "Poll An External Website for XML" input connector defines the default acceptable MIME type as "application/xml". This MIME type isn't always defined on the web server and therefore isn't acceptable for (hence the 406). As a result, it is up to us to change the acceptable MIME type to something the external web server recognizes in order to issue a 'successful' request (I emphasize successful since the request is in fact valid.. it's just not what the web server wants or recognizes). With external XML sources, I have had success with changing the acceptable MIME type to text/xml, application/xhtml+xml, etc. I would recommend reviewing the MIME types available for XML and changing the acceptable MIME type in ArcGIS GeoEvent Server from application/xml as needed.
In situations where we have direct access to the web server hosting the XML, a different solution to this problem would be to define the MIME type on the web server to use application/xml (ArcGIS GeoEvent Server's default). In Microsoft IIS for example (installed with the ArcGIS Web Adaptor), we can see that the default MIME type for .xml extensions is text/xml. If we wanted to, we could change the default MIME type in Microsoft IIS from text/xml to application/xml for .xml extensions. With this change in place, we wouldn't need to adjust the acceptable MIME type default in ArcGIS GeoEvent Server (however we could if we wanted to hence the previous paragraph).
The key takeaway here is to pay close attention to the MIME types being used. Sometimes what ArcGIS GeoEvent Server uses by default isn't what the web server recognizes and vice versa. Hope this helps!
