Problems trying to get ExtractData to work in V4.x

[MarkSmith]

I’m trying to write some code in ESRI JavaScript V4.32 to export data to shapefile using an ExtractData service that currently works fine in ESRI JavaScript V3.x, but I’m struggling.

So, I’m using an ExtractData service I published from Pro to my ArcGIS Server 11.1.  My old JavaScript V3.x application uses this service just fine using code along the lines of:
gp = new esri.tasks.Geoprocessor("https://<server>/arcgis/rest/services/<extractService>/GPServer/ExtractData");
gp.submitJob(params, completeCallback, statusCallback, function (error) {
});
function completeCallback(jobInfo) {
    if (jobInfo.jobStatus !== "esriJobFailed") {
        gp.getResultData(jobInfo.jobId, "Output_Zip_File", downloadFile);
    }
}

Now I’m trying to migrate this to ESRI JavaScript V4.32 but without success.  I have only found one example online, so I’m following that as a guide.  The crux of the code is as follows:

                let url = "https://<server>//arcgis/rest/services/<service>/GPServer/ExtractData/submitJob";
                let layer = "https://<server>/ArcGIS/rest/services/<layersService>/FeatureServer/0";
                var options = {
                    inputLayers: [{ "url": layer, "filter": "1=1" }],
                    extent: featCollection,
                    dataFormat: "SHAPEFILE",
                    outSR: 27700,
                    outputName: {
                        title: 'TEST'
                    }
                }
                esriRequest(
                    url, options).then((response) => {
                        console.log(response)
                    })

The problem is, when it tries to execute esriRequest, I get a forbidden 403 error, and if I look in the console detail it tells me The Administrator has disabled the Service Directory.  Yes, I have disabled my services directory because I don’t want people browsing it.  The thing is, I’ve got lots of other code making requests to other rest/services on the same server and it all works perfectly, so why is this one an issue, and only for the new V4.32 code?  I did try temporarily enabling the services directory and that solved the 403 error (I did get back another error about invalid token “<”, I think that’s something to do with html vs JSON, but one problem at a time).  However, I shouldn’t have to enable the services directory for this, it’s already working perfectly in good old JS V3.x.

Something else I noted from the one example I could find: https://community.esri.com/t5/arcgis-javascript-maps-sdk-questions/extractdata-in-js-4-x/td-p/130253...  is that this person is making use of the System ExtractData service, e.g.
https://<server>/arcgis/rest/services/System/SpatialAnalysisTools/GPServer/ExtractData/submitJob
And so I gave that a try, but with the same 403 result.  I tried it with and without token security enabled, but I still get 403.  The fact that the above sample uses the System ExtractData service does raise an interesting question however…  If it’s possible to extract data from a FeatureService by making use of the System ExtractData service, then why do I need to publish my own ExtractData service?  Either way, I can’t get my ExtractData service to work, or the System one at present because of the forbidden 403 problem.

I’ve spent a lot of time trying to figure this out, but with the above sample being my only point of reference (together with the ESRI page on the subject that just lists the request parameters) I’m afraid I’m stuck.  I know my ExtractData service works and is accessible because it’s working right now in my old V3.x app, so can anyone please help shed some light on how to get this working in V4.32, perhaps with a code example?

Thank you very much.

Mark.