topic Re: Collector App Sync - SEVERE ERROR. in ArcGIS Collector Questions

Collector App Sync - SEVERE ERROR.

BledarBirbo1 — Sat, 11 Dec 2021 20:45:26 GMT

HI i have been investigating this for 3 days but haven't been able to find a solution.

We have published 2 Feature Services on our own ArcGIS Server 10.2.2

We have a Web Adapter 10.2.2 in front of the ArcGIS Server. (HTTPS)

We have created a Map on AGOL

The users are in the field collecting data offline and taking photos as attachments with the iOS Collector 10.3

During our tests and for some users in the beginning there were no syncing issues at all.

The problem is now that all the users are now reporting " illegal start of token (<) " and cannot sync their work back to the server.

a-) On Arcgis Server (DEBUG Logs) we are getting the following ERROR:

com.esri.rf.RException: Error parsing multi-part request at 
com.esri.rf.multipart.DefaultMultipartRequestHandler.parseRequest(DefaultMultipartRequestHandler.java:72) at 
com.esri.rf.RFactory.newMultipartRequestHandler(RFactory.java:203) at 
com.esri.rf.RRequest.newInstance(RRequest.java:976) at 
com.esri.rf.RServlet.service(RServlet.java:86) at 
javax.servlet.http.HttpServlet.service(HttpServlet.java:728) at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305) at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at 
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222) at 
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123) at 
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502) at 
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171) at 
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:100) at 
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118) at 
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408) at 
org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1041) at 
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:603) at 
org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310) at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at 
java.lang.Thread.run(Thread.java:744) 
Caused by: org.apache.commons.fileupload.FileUploadException: Read timed out at org.apache.commons.fileupload.FileUploadBase.parseRequest(FileUploadBase.java:381) at 
org.apache.commons.fileupload.servlet.ServletFileUpload.parseRequest(ServletFileUpload.java:126) at 
com.esri.rf.multipart.DefaultMultipartRequestHandler.parseRequest(DefaultMultipartRequestHandler.java:65) ... 19 
more Caused by: java.net.SocketTimeoutException: Read timed out at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:152) at java.net.SocketInputStream.read(SocketInputStream.java:122) at sun.security.ssl.InputRecord.readFully(InputRecord.java:442) at sun.security.ssl.InputRecord.read(InputRecord.java:480) at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:927) at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:884) at sun.security.ssl.AppInputStream.read(AppInputStream.java:102) at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:532) at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:501) at org.apache.coyote.http11.InternalInputBuffer$InputStreamInputBuffer.doRead(InternalInputBuffer.java:563) at org.apache.coyote.http11.filters.IdentityInputFilter.doRead(IdentityInputFilter.java:124) at org.apache.coyote.http11.AbstractInputBuffer.doRead(AbstractInputBuffer.java:346) at org.apache.coyote.Request.doRead(Request.java:422) at org.apache.catalina.connector.InputBuffer.realReadBytes(InputBuffer.java:290) at org.apache.tomcat.util.buf.ByteChunk.substract(ByteChunk.java:449) at org.apache.catalina.connector.InputBuffer.read(InputBuffer.java:315) at org.apache.catalina.connector.CoyoteInputStream.read(CoyoteInputStream.java:200) at org.apache.commons.fileupload.MultipartStream$ItemInputStream.makeAvailable(MultipartStream.java:977) at org.apache.commons.fileupload.MultipartStream$ItemInputStream.read(MultipartStream.java:887) at java.io.InputStream.read(InputStream.java:101) at org.apache.commons.fileupload.util.Streams.copy(Streams.java:94) at org.apache.commons.fileupload.util.Streams.copy(Streams.java:64) at org.apache.commons.fileupload.MultipartStream.readBodyData(MultipartStream.java:593) at org.apache.commons.fileupload.MultipartStream.discardBodyData(MultipartStream.java:619) at org.apache.commons.fileupload.MultipartStream.skipPreamble(MultipartStream.java:638) at org.apache.commons.fileupload.FileUploadBase$FileItemIteratorImpl.findNextItem(FileUploadBase.java:961) at org.apache.commons.fileupload.FileUploadBase$FileItemIteratorImpl.<init>(FileUploadBase.java:942) at org.apache.commons.fileupload.FileUploadBase.getItemIterator(FileUploadBase.java:331) at org.apache.commons.fileupload.FileUploadBase.parseRequest(FileUploadBase.java:349) ... 21 more

b-) On Arcgis Server (DEBUG Logs) we are also getting "Unable to process request. Error parsing multi-part request" as a high level error.

c-) Here are the IIS logs from the Web-Adpator site. (Please note that although some ports made it to the arcgis server the last upload part is failing with code 500)

2015-05-07 07:09:36 141.xxx.xxx.xxx POST /server/tokens/ f=json 443 - 82.148.xxx.xxx Collector-iOS-10.3:ArcGISiOS-10.2.4+Collector/8.1.2/iPad5,4 200 0 0 2979
2015-05-07 07:09:44 141.xxx.xxx.xxx POST /server/rest/services/TEAM2100/Collector_Active_Assets/FeatureServer/uploads/register - 443 - 82.148.44.205 Collector-iOS-10.3:ArcGISiOS-10.2.4+Collector/8.1.2/iPad5,4 200 0 0 920
2015-05-07 07:09:44 141.xxx.xxx.xxx POST /server/rest/services/TEAM2100/Collector_Active_Assets_Redline/FeatureServer/synchronizeReplica - 443 - 82.148.44.205 Collector-iOS-10.3:ArcGISiOS-10.2.4+Collector/8.1.2/iPad5,4 200 0 0 8002
2015-05-07 07:09:46 141.xxx.xxx.xxx GET /server/rest/services/TEAM2100/Collector_Active_Assets_Redline/FeatureServer/jobs/jaa5f0db584ea4053acbdd6ef428390d5 f=json&token=C3-2a7tyzySX4nq1NMOsL2Y2j89rGiXmR68hT7bnOuyr6I9UrafswZP6WmLcbERz 443 - 82.148.44.205 Collector-iOS-10.3:ArcGISiOS-10.2.4+Collector/8.1.2/iPad5,4 200 0 0 1419
2015-05-07 07:09:54 141.xxx.xxx.xxx GET /server/rest/services/TEAM2100/Collector_Active_Assets_Redline/FeatureServer/jobs/jaa5f0db584ea4053acbdd6ef428390d5 f=json&token=C3-2a7tyzySX4nq1NMOsL2Y2j89rGiXmR68hT7bnOuyr6I9UrafswZP6WmLcbERz 443 - 82.148.44.205 Collector-iOS-10.3:ArcGISiOS-10.2.4+Collector/8.1.2/iPad5,4 200 0 0 967
2015-05-07 07:10:01 141.xxx.xxx.xxx GET /server/rest/services/TEAM2100/Collector_Active_Assets_Redline/FeatureServer/jobs/jaa5f0db584ea4053acbdd6ef428390d5 f=json&token=C3-2a7tyzySX4nq1NMOsL2Y2j89rGiXmR68hT7bnOuyr6I9UrafswZP6WmLcbERz 443 - 82.148.44.205 Collector-iOS-10.3:ArcGISiOS-10.2.4+Collector/8.1.2/iPad5,4 200 0 0 1872
2015-05-07 07:10:45 141.xxx.xxx.xxx POST /server/rest/services/TEAM2100/Collector_Active_Assets/FeatureServer/uploads/i8aa3ef1e-fec4-48b2-bc9f-5d697178d051/uploadPart token=C3-2a7tyzySX4nq1NMOsL2Y2j89rGiXmR68hT7bnOuyr6I9UrafswZP6WmLcbERz 443 - 82.148.44.205 Collector-iOS-10.3:ArcGISiOS-10.2.4+Collector/8.1.2/iPad5,4 200 0 0 60824
2015-05-07 07:11:56 141.xxx.xxx.xxx POST /server/rest/services/TEAM2100/Collector_Active_Assets/FeatureServer/uploads/i8aa3ef1e-fec4-48b2-bc9f-5d697178d051/uploadPart token=C3-2a7tyzySX4nq1NMOsL2Y2j89rGiXmR68hT7bnOuyr6I9UrafswZP6WmLcbERz 443 - 82.148.44.205 Collector-iOS-10.3:ArcGISiOS-10.2.4+Collector/8.1.2/iPad5,4 200 0 0 71292
2015-05-07 07:13:01 141.xxx.xxx.xxx POST /server/rest/services/TEAM2100/Collector_Active_Assets/FeatureServer/uploads/i8aa3ef1e-fec4-48b2-bc9f-5d697178d051/uploadPart token=C3-2a7tyzySX4nq1NMOsL2Y2j89rGiXmR68hT7bnOuyr6I9UrafswZP6WmLcbERz 443 - 82.148.44.205 Collector-iOS-10.3:ArcGISiOS-10.2.4+Collector/8.1.2/iPad5,4 500 0 0 64225
#Software: Microsoft Internet Information Services 7.5

Things we have tried with no success :

NIM099731 - The error message, "Error parsing multi-part reque..

Re: Sync error: Illegal start of token [<]

Any ideas on why this happens ?

Thanks

Re: Collector App Sync - SEVERE ERROR.

MiaogengZhang — Mon, 11 May 2015 18:00:36 GMT

Bledar,

Do you mean you have tried to increase the uploadReadAheadSize value on IIS?

Here are the steps:

1. Launch “Internet Information Services (IIS) Manager”

2. Select the site that you are hosting your web application under it.

3. In the Features section, double click “Configuration Editor”

4. Under “Section” select: system.webServer then serverRuntime

5. Modify the “uploadReadAheadSize” section to ~20MB

Re: Collector App Sync - SEVERE ERROR.

MichaelDavis3 — Mon, 11 May 2015 19:18:41 GMT

FYI - the illegal start of token errors are what we see when our users are trying to check in a database that is too large (somewhere in the neighborhood of ~800mb - 1gb). We see this on our projects with lots of image attachments.

Re: Collector App Sync - SEVERE ERROR.

BledarBirbo1 — Tue, 12 May 2015 15:38:28 GMT

Yes i have made these changes and the problem is not resolved yet.

Here is more information on this issue.

a-) I tested the offline editing with about 100MB of attachments with ArcMap and the sync failed with the same reason.

b-) I saw that the error was coming from one of the 4 clustered arcgis server machines and i stopped the other 3 and left the default cluster with only one machine.

c-) After that the ArcMap offline map synced ok but the Collector app was still getting an error.

Here is the error on the server side.

SEVERE

May 12, 2015, 6:23:37 PM

Unable to process request. Error parsing multi-part request

Rest

Re: Collector App Sync - SEVERE ERROR.

MiaogengZhang — Wed, 13 May 2015 21:03:45 GMT

Bledar,

The development team has confirmed that there is a bug in ArcGIS Server that prevents huge uploads (greater than a few GB on fast networks and greater than 100 MB on slower networks). We are sorry about this problem and this issue is under consideration in the 10.4 release. I would recommend you to contact the Support to see if they can come up with a work around.

Re: Collector App Sync - SEVERE ERROR.

BledarBirbo1 — Wed, 13 May 2015 21:18:48 GMT

Thank you Zhang for the information.

Can you post here the Bug Number for this issue, so i can track it ?

Re: Collector App Sync - SEVERE ERROR.

BradySmith — Wed, 13 May 2015 23:09:58 GMT

We have been having this problem somewhat sporadically for a couple months with Collector and service that is used to take notes/pictures in the field. It appears that your information has helped us identify what is causing our errors so we can work on finding a workaround.

Re: Collector App Sync - SEVERE ERROR.

MattCooper — Wed, 13 May 2015 23:14:52 GMT

Yep, this has been causing us a lot of issues. Thanks for confirming it was a bug. Any idea if this is an issue when using hosted feature services in AGOL as well?

Re: Collector App Sync - SEVERE ERROR.

ScottPrindle — Thu, 14 May 2015 21:21:46 GMT

Hey Bledar,

I noticed in your initial post that the following patch was not mentioned:

ArcGIS 10.2.2 for (Desktop, Engine, Server) Geodatabase and Feature Service Sync Optimization Patch | Samples and Utilit…

This may improve sync performance since you are at ArcGIS for Server 10.2.2, in addition to the other modifications outlined byMiaogeng Zhang previously.

Re: Collector App Sync - SEVERE ERROR.

AdrianMarsden — Mon, 25 Jan 2016 12:26:00 GMT

I've still got this issue, and I am running on 10.3.1 Server and 10.3.6 Collector and connecting via a pretty fast internet connection.

I have 10 features with 20 photos (taken before the option of setting the size, so will be at full iPad size) I have a call in with local support (Esri(UK) but my main goal here is just to get these features off.

Cheers

ACM

Re: Collector App Sync - SEVERE ERROR.

AdrianMarsden — Mon, 25 Jan 2016 16:50:50 GMT

Some more testing and a single feature work no attachments syncs, a single feature with a single photo worked. But a single feature with 9 photos failed. I'll try numbers between 1 and 9 later.

Also, on full debug, the server shows a huge java message just at the same time as the SRVERE error thrown on the sync.

So is there a java memory setting that needs increasing?