AGOL appears to give inconsistent results

[Vidar]

This is an urgent request for help as I am trying to release my addin in the next week.

I am absolutely pulling my hair out with the seemingly random results and duplicate entities from from AGOL (ArcGIS Online).

I have 1700 layer files published in our ArcGIS Online account. There are no duplicates of our layer files when I look in AGOL. I query for them using the following code:

var query = PortalQueryParameters.CreateForItemsOfTypes(new List<PortalItemType>() { PortalItemType.Layer }, "", PortalUtils.GroupID, "");

var portalItems = new List<PortalItem>();
while (query != null)
{
   //run the search
   PortalQueryResultSet<PortalItem> results = await portal.SearchForContentAsync(query);
   portalItems.AddRange(results.Results);
   query = results.NextQueryParameters;
}

foreach (var pi in portalItems)
{
   //Do something with the portal items
   System.Diagnostics.Debug.WriteLine($"{pi.Id}, {pi.Name}");
}

When I look at the debug output above - I get 1700 results back, which on the face of it seems correct, but when I look closely at the results (I have code to look for dupes) there appears to be some objects that have duplicate item ID's and of course names. Not only this - it's NOT THE SAME DUPLICATES EVERY TIME I QUERY! They are different in the number of duplicates and the actual layers that are duplicated. 

I have also manually looked at the names and sorted the results in Excel - so I know for sure duplicates are occurring - if nothing else it appears in my UI ultimately - hence why I am trying to debug and get to the bottom of this problem.

I am at a loss as to what to do here. I cannot understand it at all and it's making me pretty nervous.  If you can help I would really appreciate it.

[SteveVan_Esch]

I could reproduce this.

I increased my query limit from10 to 100 and moved the operation to a background thread. My HashSet sees different duplicates every time I run it. I'll investigate more.

internal async Task<bool> Fetch()
{
var query = PortalQueryParameters.CreateForItemsOfTypes(new List<PortalItemType>() { PortalItemType.Layer }, "", "", "");
query.Limit = 100; // max

ArcGISPortal portal = ArcGIS.Desktop.Core.ArcGISPortalManager.Current.GetActivePortal();

HashSet<string> ids = new HashSet<string>();

await BackgroundTask.Run(TaskPriority.normal, () =>
{
var portalItems = new List<PortalItem>();
while (query != null)
{
//run the search
PortalQueryResultSet<PortalItem> results = ArcGIS.Desktop.Core.ArcGISPortalExtensions.SearchForContentAsync(portal, query).Result;
portalItems.AddRange(results.Results);
query = results.NextQueryParameters;
}

//portalItems.Sort((x, y) => x.Title.Trim().CompareTo(y.Title.Trim()));

portalItems.Sort((x, y) => x.ID.CompareTo(y.ID));

foreach (var pi in portalItems)
{
if (!ids.Add(pi.ID))
System.Diagnostics.Debug.WriteLine("duplicate id");
}

}, BackgroundProgressor.None);

return true;
}

[Vidar]

Good to know it can be reproduced.

I have been speaking to a colleague - we thought it may be something to do with ordering the paging?   It's not.

I'm out of ideas thats for sure.

[SteveVan_Esch]

Can you try adding this bit of code? It looks like there's a known issue on the Server relating to paging lots of data (index is constantly changing) without a sort order.

var query = PortalQueryParameters.CreateForItemsOfTypes(new List<PortalItemType>() { PortalItemType.Layer }, "", "", "");
query.Limit = 100; // max
query.Query = query.Query + "&sortField=modified&sortOrder=desc";

[Vidar]

It gave almost full unique results - but it took 3 goes.

1699/1700

1698/1700

1700/1700

I don't have that much confidence this is going to work 100% of the time really.

[SteveVan_Esch]

In my case the response says there are 2797 items total and I always get this many back after all my fetches, no duplicates. I tried about 10 times. So you're still getting a duplicate sometimes and that's why your count is off?

[Vidar]

Yes - I seem to be getting duplicates still - but the count is almost complete.

In order to get a complete set I have to re-issue the query request multiple times.

The following is list of the amount of re-queries required to get a full result set. Each row is a new session of Pro:

2 queries

3 queries

2 queries

2 queries

1 - first complete result set!

5 - queries

2 - queries

So 1 in 7 attempts I managed to get a full result set back without duplicates.

Hope that makes sense - I could do more more tests - but I think the trend is obvious.

[Vidar]

Hi Steve,

You say the issue looks like it's "know" - do you have a NIM number for this bug - and any idea when it will be fixed? The NIM number would be good for us to track at the very least.  

Cheers

Simon.