Stewart,
I don't use app builder, so I'm not sure. I believe app builder only works with compiled apps, so it shouldn't be re-compiling it for you, especially with only .xml changes.
Not sure if app builder will re-write the xml file every time you configure, but know there are certain configs that you can't do within the app builder.
However, if I change the xml(s) on my server (for that FV app) for a widget, the changes are reflected the next time I do a clean load into the browser (can do this with any text editor, so is independant of the "build"). So, you should be able to make the config changes on your server folder and have them take effect. Not sure, however, if app builder will "remove" your customizations if you open/configure that widget again or not. You would have to test.
With the identifywidget, I find it much easier to just put ONE layer in there and get it configured so that it is reporting as you want, then I add my other layers to the xml.
R_