This may be a bug:
BUG-000113688 - Portal for ArcGIS 10.6 fails to set Raster Analytics server and returns the error 'Error while processing request. Internal error. JSON response not set.', if hosting server protocol has been updated to HTTPS
It's logged against 10.6, but it may occur at 10.5.1. It's also logged against Raster Analytics but the details of the bug indicate it happens with GeoAnalytics as well.