PrintTask includes renderingRule if that property exists in your layer. Please check yourCustomImageServiceLayer.renderingRule is there, and yourCustomImageServiceLayer.renderingRule.toJson() is a valid function.
Ahh I think you may be on to something. I logged the layer to the console and noticed the renderingRule was null, so I added a this.renderingRule = _self.getRenderingRule() inside the getImageUrl() function, which the getRenderingRule() is a function we have added in to build the custom rendering rule based off some slider and checkbox controls we have in the app. After doing this I did a console.dir(layer.renderingRule.toJson()) and kept getting a blank object. I overwrote the toJson method inside the constructor and was getting the same response, just a blank object, so I then moved it to the getImageUrl() function and was able to get it to retrieve the JSON string but still am unable to see it print. I put in alert in the layer.renderingRule.toJson() and it does not fire when I call the print task. I tried just logging layer.renderingRule.toJson() to the console and this was what it returned.
"{"rasterFunction":"PalmOilSuitabilityNew","rasterFunctionArguments":{"ElevRaster":"$1","SlopeRaster":"$2","WaterRaster":"$3","ConsRaster":"$4","STypeRaster":"$5","SDepthRaster":"$6","PeatRaster":"$7","SAcidRaster":"$8","SDrainRaster":"$9","RainfallRaster":"$10","LCRaster":"$11","ElevInpR":[0,100,100,1000000],"ElevOutV":[1,0],"SlopeInpR":[0,30,30,1000000],"SlopeOutV":[1,0],"WaterInpR":[0,10,10,1000000],"WaterOutV":[0,1],"ConsInpR":[0,10,10,1000000],"ConsOutV":[0,1],"RainfallInpR":[0,150,150,700,700,1000000],"RainfallOutV":[1,1,0],"SDepthInpR":[0,0,1,1,2,2,3,3,4,4,5,5,6,6,7,7],"SDepthOutV":[1,0,0,0,1,1,1,1],"PeatInpR":[0,0,1,1,2,2,3,3,4,4,5,5,6,6],"PeatOutV":[1,0,0,0,0,0,0],"SAcidInpR":[0,0,1,1,2,2,3,3,4,4,5,5,6,6,7,7],"SAcidOutV":[1,1,1,1,1,1,1,1],"SDrainInpR":[0,0,1,1,2,2,3,3,4,4],"SDrainOutV":[1,0,1,1,1],"LCInpR":[0,0,1,1,2,2,3,3,4,4,5,5,6,6,7,7,8,8],"LCOutV":[1,1,1,1,0,0,0,0,0],"STypeInpR":[1,1,2,2,3,3,4,4,5,5,6,6,7,7,8,8,9,9,10,10],"STypeOutV":[1,1,1,0,1,1,1,1,1,0]}}"
does this look like a valid renderingRule and is setting the this.renderingRule to the newly created custom renderingRule in getImageUrl() the best way to keep it up to date and setting the toJson() function in the getImageUrl() as well the right way to override it ?