OK, problem found, and work-around implemented.
What you need to know is that SelectXY only works on the features that are currently on the screen.
I figured that out early on, and then implemented a "zoom to full extent" before running the selectXY.
This created a different problem:
SelectXY uses a tolerance of x pixels, so if you zoom out too far, the point will be within this tolerance, even though it is outside of the polygon.
You can set the tolerance to zero, but then it becomes virtually impossible to click on any points (if you need to do that elsewhere), so it is not really a sollution.
What I did, was to zoom to a specific scale programatically (i.e. 1:10 000 in my case), and then centre the map on the X,Y of the point that I am working with.
Not the cleanest or quickest, but it seems to work OK.