Hi Don, good to know I'm not alone, though being a bug is alarming...
It seems to me every js app that needs to turn on/off change querydefs of layers will not work with a 10.5 server. That's a huge oversight if you ask me...
You can convert your example to json but it will only work using a direct request to the rest api, not using the js api.
Your example would be:
{0:"POPULATION > 5000000", 5:"AREA > 100000"}
You can see what's going on by copying the request in the browser's console. The request will end with a parameter like f=image. If you change this last parameter to f=html you will see a page where you can change all parameters including the querydefs.
This however it does not solve the problem: js api 3.9 does not comply with 10.5 server requiring a json string in querydefs. It's the js api responsibility to turn your programming into correct rest requests.
The only way out I see is to roll back 10.5 to 10.4, and wait for a compliant release of js api.
Rolling back has to be done manually: you have to write down every service configuration, every server configuration, and then recreate everything by hand. Unless you made a backup of your last configuration before upgrading to 10.5.
Anyway, a huge oversight from esri, and a huge loss of time and amount of problems for us.
If you hear anything positive from esri please let me know.
EDIT: it just now occurred to me it could be a bug with 10.5 server, accepting the wrong syntax in the layerdefs parameter, and not with the js api... also, I seem to remember an old solution where there was a way in the js api to edit the request before it was sent, correcting whatever was needed to make it work.