Missing features in ESRI REST services: Query 1=1 does not return all records

[SeanMiller2]

We are running ArcGIS for Server 10.4.1 on linux (AWS) and encountering issues where certain features in our map services will not appear. Upon querying in the rest services directory, the missing object will show an error code "Invalid or missing input parameters." where it typically shows the fields in that features attributes. 

We are working with a 2 machine cluster and serving up data dynamically to an AWS s3 bucket using standard NSF4 mounts for  the data store. Is there an issue with the way we are serving up our data and /or our site configuration/settings that could be causing the issues we are encountering? 

[RandallWilliams]

Weird. A few questions:

a. How many GIS Servers are in the site?

b. What's the data source (looks like a shape file?)

c. Any usual characters in that record?

d. Is data copied to your server when you publish, or are you referencing it on a share somewhere?

[SeanMiller2]

Hi  Randall,

a. We are running 2 servers on AWS, we sync our datasets to Amazon S3 bucket that we have mounted to each host using NFS4.

b. Correct, this is shape file data

c. There is nothing unusual in the records of those particular layers that have issues 

d. We have a registered data store on the server, so that the services can be updated with out dynamic data (it is frequently updated weather forecast information). 

As  a side note -- ESRI tech support thinks it may have something to do with caching related to the mounted file store, but nothing we've done (e.g. turn on the automated caching 'noac' NFS mount option) as resulted in any fix.

Thanks for your response!

Sean

[RandallWilliams]

OK. So that S3 bucket is mounted on both machines?

Another question: Is it always the same record(s) that have this problem, or is it random?

This almost seems like a concurrency or locking file issue of some sort, but that wouldn't make much sense on a per-record basis.  

Have you tested with other data sources, like FGDB or a PGSQL database?

[SeanMiller2]

Yes, the S3 bucket is mounted on both and it is always the same records that are problematic (not returning any queries). 

Based on the issues we’ve been encountering we’ve decided to redesign our AGIS for Server configuration and move our systems directory path to refer to local storage. 

We haven't tested with any of data sources other than SHP file content. 

Thanks!

Sean