Obviously a web service cannot write a file directly to a user's file system, so ArcGIS Server allows the service to create files in a staging folder on the server and let's the user retrieve it from there.
Here's how to author the tool in a Python Toolbox:
Make an output parameter with datatype = "DEFile" and parameterType = "Derived". In the execute function, create a file in the arcpy.env.scratchFolder. When running the tool in Desktop, this folder will be buried in the user's AppData\Local\Temp folder (it might be helpful to see this location using "print arcpy.env.scratchFolder"). After publishing, each execution of the task will get a unique job folder with its own scratch folder, located here arcgisserver\directories\arcgisjobs\<folder>\<service>_gpserver\<job>\scratch. Return the output parameter as follows: parameters.value = file_path, where file_path is the entire path to the file in the scratch folder. I recommend something like this: file_path = os.path.join(arcpy.env.scratchFolder, "filename.ext")
Here's how to publish the service:
Run the tool, right click on the gp result and and Share As GeoProcessing Service. There's really nothing special you have to set here, other than the normal metadata and parameter descriptions, but I recommend configuring the service to run asynchronously because you never know how long it will take to create the file. When the task is executed, the REST endpoint will have a link to the output parameter in the Results section. That output parameter will include a url to the file that lives in the job's scratch folder. It's up to the client to go and retrieve that file.