This is an urgent request for help as I am trying to release my addin in the next week.
I am absolutely pulling my hair out with the seemingly random results and duplicate entities from from AGOL (ArcGIS Online).
I have 1700 layer files published in our ArcGIS Online account. There are no duplicates of our layer files when I look in AGOL. I query for them using the following code:
var query = PortalQueryParameters.CreateForItemsOfTypes(new List<PortalItemType>() { PortalItemType.Layer }, "", PortalUtils.GroupID, ""); var portalItems = new List<PortalItem>(); while (query != null) { //run the search PortalQueryResultSet<PortalItem> results = await portal.SearchForContentAsync(query); portalItems.AddRange(results.Results); query = results.NextQueryParameters; } foreach (var pi in portalItems) { //Do something with the portal items System.Diagnostics.Debug.WriteLine($"{pi.Id}, {pi.Name}"); }
When I look at the debug output above - I get 1700 results back, which on the face of it seems correct, but when I look closely at the results (I have code to look for dupes) there appears to be some objects that have duplicate item ID's and of course names. Not only this - it's NOT THE SAME DUPLICATES EVERY TIME I QUERY! They are different in the number of duplicates and the actual layers that are duplicated.
I have also manually looked at the names and sorted the results in Excel - so I know for sure duplicates are occurring - if nothing else it appears in my UI ultimately - hence why I am trying to debug and get to the bottom of this problem.
I am at a loss as to what to do here. I cannot understand it at all and it's making me pretty nervous. If you can help I would really appreciate it.
I could reproduce this.
I increased my query limit from10 to 100 and moved the operation to a background thread. My HashSet sees different duplicates every time I run it. I'll investigate more.
internal async Task<bool> Fetch()
{
var query = PortalQueryParameters.CreateForItemsOfTypes(new List<PortalItemType>() { PortalItemType.Layer }, "", "", "");
query.Limit = 100; // max
ArcGISPortal portal = ArcGIS.Desktop.Core.ArcGISPortalManager.Current.GetActivePortal();
HashSet<string> ids = new HashSet<string>();
await BackgroundTask.Run(TaskPriority.normal, () =>
{
var portalItems = new List<PortalItem>();
while (query != null)
{
//run the search
PortalQueryResultSet<PortalItem> results = ArcGIS.Desktop.Core.ArcGISPortalExtensions.SearchForContentAsync(portal, query).Result;
portalItems.AddRange(results.Results);
query = results.NextQueryParameters;
}
//portalItems.Sort((x, y) => x.Title.Trim().CompareTo(y.Title.Trim()));
portalItems.Sort((x, y) => x.ID.CompareTo(y.ID));
foreach (var pi in portalItems)
{
if (!ids.Add(pi.ID))
System.Diagnostics.Debug.WriteLine("duplicate id");
}
}, BackgroundProgressor.None);
return true;
}
Good to know it can be reproduced.
I have been speaking to a colleague - we thought it may be something to do with ordering the paging? It's not.
I'm out of ideas thats for sure.
Can you try adding this bit of code? It looks like there's a known issue on the Server relating to paging lots of data (index is constantly changing) without a sort order.
var query = PortalQueryParameters.CreateForItemsOfTypes(new List<PortalItemType>() { PortalItemType.Layer }, "", "", "");
query.Limit = 100; // max
query.Query = query.Query + "&sortField=modified&sortOrder=desc";
It gave almost full unique results - but it took 3 goes.
1699/1700
1698/1700
1700/1700
I don't have that much confidence this is going to work 100% of the time really.
In my case the response says there are 2797 items total and I always get this many back after all my fetches, no duplicates. I tried about 10 times. So you're still getting a duplicate sometimes and that's why your count is off?
Yes - I seem to be getting duplicates still - but the count is almost complete.
In order to get a complete set I have to re-issue the query request multiple times.
The following is list of the amount of re-queries required to get a full result set. Each row is a new session of Pro:
2 queries
3 queries
2 queries
2 queries
1 - first complete result set!
5 - queries
2 - queries
So 1 in 7 attempts I managed to get a full result set back without duplicates.
Hope that makes sense - I could do more more tests - but I think the trend is obvious.
Hi Steve,
You say the issue looks like it's "know" - do you have a NIM number for this bug - and any idea when it will be fixed? The NIM number would be good for us to track at the very least.
Cheers
Simon.