you can create a soe. Here you can see an example: http://blog.geomusings.com/2013/06/10/geojson-from-arcgis-server/
Have you tried using Esri's Terraformer libraries to do the conversion?
Unfortunately, Server Object Extension (SOE) is not helpful because we do not control what extensions are installed on the customer servers. We're interested to get the converted data from server using techniques officially supported by Esri, or convert the data on the client side.