Collector App Sync - SEVERE ERROR.

7305
10
05-07-2015 05:17 AM
BledarBirbo1
Occasional Contributor

HI i have been investigating this for 3 days but haven't been able to find a solution.

We have published 2 Feature Services on our own ArcGIS Server 10.2.2

We have a Web Adapter 10.2.2 in front of the ArcGIS Server. (HTTPS)

We have created a Map on AGOL

The users are in the field collecting data offline and taking photos as attachments with the iOS Collector 10.3

During our tests and for some users in the beginning there were no syncing issues at all.

The problem is now that all the users are now reporting " illegal start of token (<) " and cannot sync their work back to the server.

a-) On Arcgis Server (DEBUG Logs) we are getting the following ERROR:

com.esri.rf.RException: Error parsing multi-part request at 
com.esri.rf.multipart.DefaultMultipartRequestHandler.parseRequest(DefaultMultipartRequestHandler.java:72) at 
com.esri.rf.RFactory.newMultipartRequestHandler(RFactory.java:203) at 
com.esri.rf.RRequest.newInstance(RRequest.java:976) at 
com.esri.rf.RServlet.service(RServlet.java:86) at 
javax.servlet.http.HttpServlet.service(HttpServlet.java:728) at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305) at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at 
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222) at 
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123) at 
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502) at 
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171) at 
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:100) at 
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118) at 
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408) at 
org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1041) at 
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:603) at 
org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310) at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at 
java.lang.Thread.run(Thread.java:744) 
Caused by: org.apache.commons.fileupload.FileUploadException: Read timed out at org.apache.commons.fileupload.FileUploadBase.parseRequest(FileUploadBase.java:381) at 
org.apache.commons.fileupload.servlet.ServletFileUpload.parseRequest(ServletFileUpload.java:126) at 
com.esri.rf.multipart.DefaultMultipartRequestHandler.parseRequest(DefaultMultipartRequestHandler.java:65) ... 19 
more Caused by: java.net.SocketTimeoutException: Read timed out at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:152) at java.net.SocketInputStream.read(SocketInputStream.java:122) at sun.security.ssl.InputRecord.readFully(InputRecord.java:442) at sun.security.ssl.InputRecord.read(InputRecord.java:480) at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:927) at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:884) at sun.security.ssl.AppInputStream.read(AppInputStream.java:102) at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:532) at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:501) at org.apache.coyote.http11.InternalInputBuffer$InputStreamInputBuffer.doRead(InternalInputBuffer.java:563) at org.apache.coyote.http11.filters.IdentityInputFilter.doRead(IdentityInputFilter.java:124) at org.apache.coyote.http11.AbstractInputBuffer.doRead(AbstractInputBuffer.java:346) at org.apache.coyote.Request.doRead(Request.java:422) at org.apache.catalina.connector.InputBuffer.realReadBytes(InputBuffer.java:290) at org.apache.tomcat.util.buf.ByteChunk.substract(ByteChunk.java:449) at org.apache.catalina.connector.InputBuffer.read(InputBuffer.java:315) at org.apache.catalina.connector.CoyoteInputStream.read(CoyoteInputStream.java:200) at org.apache.commons.fileupload.MultipartStream$ItemInputStream.makeAvailable(MultipartStream.java:977) at org.apache.commons.fileupload.MultipartStream$ItemInputStream.read(MultipartStream.java:887) at java.io.InputStream.read(InputStream.java:101) at org.apache.commons.fileupload.util.Streams.copy(Streams.java:94) at org.apache.commons.fileupload.util.Streams.copy(Streams.java:64) at org.apache.commons.fileupload.MultipartStream.readBodyData(MultipartStream.java:593) at org.apache.commons.fileupload.MultipartStream.discardBodyData(MultipartStream.java:619) at org.apache.commons.fileupload.MultipartStream.skipPreamble(MultipartStream.java:638) at org.apache.commons.fileupload.FileUploadBase$FileItemIteratorImpl.findNextItem(FileUploadBase.java:961) at org.apache.commons.fileupload.FileUploadBase$FileItemIteratorImpl.<init>(FileUploadBase.java:942) at org.apache.commons.fileupload.FileUploadBase.getItemIterator(FileUploadBase.java:331) at org.apache.commons.fileupload.FileUploadBase.parseRequest(FileUploadBase.java:349) ... 21 more 

b-) On Arcgis Server (DEBUG Logs) we are also getting "Unable to process request. Error parsing multi-part request" as a high level error.

c-) Here are the IIS logs from the Web-Adpator site. (Please note that although some ports made it to the arcgis server the last upload part is failing with code 500)

2015-05-07 07:09:36 141.xxx.xxx.xxx POST /server/tokens/ f=json 443 - 82.148.xxx.xxx Collector-iOS-10.3:ArcGISiOS-10.2.4+Collector/8.1.2/iPad5,4 200 0 0 2979
2015-05-07 07:09:44 141.xxx.xxx.xxx POST /server/rest/services/TEAM2100/Collector_Active_Assets/FeatureServer/uploads/register - 443 - 82.148.44.205 Collector-iOS-10.3:ArcGISiOS-10.2.4+Collector/8.1.2/iPad5,4 200 0 0 920
2015-05-07 07:09:44 141.xxx.xxx.xxx POST /server/rest/services/TEAM2100/Collector_Active_Assets_Redline/FeatureServer/synchronizeReplica - 443 - 82.148.44.205 Collector-iOS-10.3:ArcGISiOS-10.2.4+Collector/8.1.2/iPad5,4 200 0 0 8002
2015-05-07 07:09:46 141.xxx.xxx.xxx GET /server/rest/services/TEAM2100/Collector_Active_Assets_Redline/FeatureServer/jobs/jaa5f0db584ea4053acbdd6ef428390d5 f=json&token=C3-2a7tyzySX4nq1NMOsL2Y2j89rGiXmR68hT7bnOuyr6I9UrafswZP6WmLcbERz 443 - 82.148.44.205 Collector-iOS-10.3:ArcGISiOS-10.2.4+Collector/8.1.2/iPad5,4 200 0 0 1419
2015-05-07 07:09:54 141.xxx.xxx.xxx GET /server/rest/services/TEAM2100/Collector_Active_Assets_Redline/FeatureServer/jobs/jaa5f0db584ea4053acbdd6ef428390d5 f=json&token=C3-2a7tyzySX4nq1NMOsL2Y2j89rGiXmR68hT7bnOuyr6I9UrafswZP6WmLcbERz 443 - 82.148.44.205 Collector-iOS-10.3:ArcGISiOS-10.2.4+Collector/8.1.2/iPad5,4 200 0 0 967
2015-05-07 07:10:01 141.xxx.xxx.xxx GET /server/rest/services/TEAM2100/Collector_Active_Assets_Redline/FeatureServer/jobs/jaa5f0db584ea4053acbdd6ef428390d5 f=json&token=C3-2a7tyzySX4nq1NMOsL2Y2j89rGiXmR68hT7bnOuyr6I9UrafswZP6WmLcbERz 443 - 82.148.44.205 Collector-iOS-10.3:ArcGISiOS-10.2.4+Collector/8.1.2/iPad5,4 200 0 0 1872
2015-05-07 07:10:45 141.xxx.xxx.xxx POST /server/rest/services/TEAM2100/Collector_Active_Assets/FeatureServer/uploads/i8aa3ef1e-fec4-48b2-bc9f-5d697178d051/uploadPart token=C3-2a7tyzySX4nq1NMOsL2Y2j89rGiXmR68hT7bnOuyr6I9UrafswZP6WmLcbERz 443 - 82.148.44.205 Collector-iOS-10.3:ArcGISiOS-10.2.4+Collector/8.1.2/iPad5,4 200 0 0 60824
2015-05-07 07:11:56 141.xxx.xxx.xxx POST /server/rest/services/TEAM2100/Collector_Active_Assets/FeatureServer/uploads/i8aa3ef1e-fec4-48b2-bc9f-5d697178d051/uploadPart token=C3-2a7tyzySX4nq1NMOsL2Y2j89rGiXmR68hT7bnOuyr6I9UrafswZP6WmLcbERz 443 - 82.148.44.205 Collector-iOS-10.3:ArcGISiOS-10.2.4+Collector/8.1.2/iPad5,4 200 0 0 71292
2015-05-07 07:13:01 141.xxx.xxx.xxx POST /server/rest/services/TEAM2100/Collector_Active_Assets/FeatureServer/uploads/i8aa3ef1e-fec4-48b2-bc9f-5d697178d051/uploadPart token=C3-2a7tyzySX4nq1NMOsL2Y2j89rGiXmR68hT7bnOuyr6I9UrafswZP6WmLcbERz 443 - 82.148.44.205 Collector-iOS-10.3:ArcGISiOS-10.2.4+Collector/8.1.2/iPad5,4 500 0 0 64225
#Software: Microsoft Internet Information Services 7.5

Things we have tried  with no success :

NIM099731 - The error message, "Error parsing multi-part reque..

Re: Sync error: Illegal start of token [&lt;]

Any ideas on why this happens ?

Thanks

0 Kudos
10 Replies
MiaogengZhang
Esri Regular Contributor

Bledar,

Do you mean you have tried to increase the uploadReadAheadSize value on IIS?

Here are the steps:

1. Launch “Internet Information Services (IIS) Manager” 

2. Select the site that you are hosting your web application under it.

3. In the Features section, double click “Configuration Editor”

4. Under “Section” select: system.webServer then serverRuntime

5. Modify the “uploadReadAheadSize” section to ~20MB

BledarBirbo1
Occasional Contributor

Yes i have made these changes and the problem is not resolved yet.

Here is more information on this issue.

a-) I tested the offline editing with about 100MB of attachments with ArcMap and the sync failed with the same reason.

b-) I saw that the error was coming from one of the 4 clustered arcgis server machines  and i stopped the other 3 and left the default cluster with only one machine.

c-) After that the ArcMap offline map synced ok but the Collector app was still getting an error.

Here is the error on the server side.

SEVEREMay 12, 2015, 6:23:37 PMUnable to process request. Error parsing multi-part requestRest
0 Kudos
MiaogengZhang
Esri Regular Contributor

Bledar,

The development team has confirmed that there is a bug in ArcGIS Server that prevents huge uploads (greater than a few GB on fast networks and greater than 100 MB on slower networks).  We are sorry about this problem and this issue is under consideration in the 10.4 release.  I would recommend you to contact the Support to see if they can come up with a work around.

BledarBirbo1
Occasional Contributor

Thank you Zhang for the information.

Can you post here the Bug Number for this issue, so i can track it ?

0 Kudos
BradySmith
Occasional Contributor

We have been having this problem somewhat sporadically for a couple months with Collector and service that is used to take notes/pictures in the field. It appears that your information has helped us identify what is causing our errors so we can work on finding a workaround.

0 Kudos
MattCooper
Occasional Contributor

Yep, this has been causing us a lot of issues. Thanks for confirming it was a bug.  Any idea if this is an issue when using hosted feature services in AGOL as well?

0 Kudos
MichaelDavis3
Occasional Contributor III

FYI - the illegal start of token errors are what we see when our users are trying to check in a database that is too large (somewhere in the neighborhood of ~800mb - 1gb).  We see this on our projects with lots of image attachments.

ScottPrindle
Esri Regular Contributor

Hey Bledar,

I noticed in your initial post that the following patch was not mentioned:

ArcGIS 10.2.2 for (Desktop, Engine, Server) Geodatabase and Feature Service Sync Optimization Patch ...

This may improve sync performance since you are at ArcGIS for Server 10.2.2, in addition to the other modifications outlined byMiaogeng Zhang previously.

0 Kudos
AdrianMarsden
Occasional Contributor III

Hi

I've still got this issue, and I am running on 10.3.1 Server and 10.3.6 Collector and connecting via a pretty fast internet connection.

I have 10 features with 20 photos (taken before the option of setting the size, so will be at full iPad size)  I have a call in with local support (Esri(UK) but my main goal here is just to get these features off.

Cheers


ACM

0 Kudos