HI i have been investigating this for 3 days but haven't been able to find a solution.
We have published 2 Feature Services on our own ArcGIS Server 10.2.2
We have a Web Adapter 10.2.2 in front of the ArcGIS Server. (HTTPS)
We have created a Map on AGOL
The users are in the field collecting data offline and taking photos as attachments with the iOS Collector 10.3
During our tests and for some users in the beginning there were no syncing issues at all.
The problem is now that all the users are now reporting " illegal start of token (<) " and cannot sync their work back to the server.
a-) On Arcgis Server (DEBUG Logs) we are getting the following ERROR:
com.esri.rf.RException: Error parsing multi-part request at com.esri.rf.multipart.DefaultMultipartRequestHandler.parseRequest(DefaultMultipartRequestHandler.java:72) at com.esri.rf.RFactory.newMultipartRequestHandler(RFactory.java:203) at com.esri.rf.RRequest.newInstance(RRequest.java:976) at com.esri.rf.RServlet.service(RServlet.java:86) at javax.servlet.http.HttpServlet.service(HttpServlet.java:728) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:100) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1041) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:603) at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) Caused by: org.apache.commons.fileupload.FileUploadException: Read timed out at org.apache.commons.fileupload.FileUploadBase.parseRequest(FileUploadBase.java:381) at org.apache.commons.fileupload.servlet.ServletFileUpload.parseRequest(ServletFileUpload.java:126) at com.esri.rf.multipart.DefaultMultipartRequestHandler.parseRequest(DefaultMultipartRequestHandler.java:65) ... 19 more Caused by: java.net.SocketTimeoutException: Read timed out at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:152) at java.net.SocketInputStream.read(SocketInputStream.java:122) at sun.security.ssl.InputRecord.readFully(InputRecord.java:442) at sun.security.ssl.InputRecord.read(InputRecord.java:480) at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:927) at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:884) at sun.security.ssl.AppInputStream.read(AppInputStream.java:102) at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:532) at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:501) at org.apache.coyote.http11.InternalInputBuffer$InputStreamInputBuffer.doRead(InternalInputBuffer.java:563) at org.apache.coyote.http11.filters.IdentityInputFilter.doRead(IdentityInputFilter.java:124) at org.apache.coyote.http11.AbstractInputBuffer.doRead(AbstractInputBuffer.java:346) at org.apache.coyote.Request.doRead(Request.java:422) at org.apache.catalina.connector.InputBuffer.realReadBytes(InputBuffer.java:290) at org.apache.tomcat.util.buf.ByteChunk.substract(ByteChunk.java:449) at org.apache.catalina.connector.InputBuffer.read(InputBuffer.java:315) at org.apache.catalina.connector.CoyoteInputStream.read(CoyoteInputStream.java:200) at org.apache.commons.fileupload.MultipartStream$ItemInputStream.makeAvailable(MultipartStream.java:977) at org.apache.commons.fileupload.MultipartStream$ItemInputStream.read(MultipartStream.java:887) at java.io.InputStream.read(InputStream.java:101) at org.apache.commons.fileupload.util.Streams.copy(Streams.java:94) at org.apache.commons.fileupload.util.Streams.copy(Streams.java:64) at org.apache.commons.fileupload.MultipartStream.readBodyData(MultipartStream.java:593) at org.apache.commons.fileupload.MultipartStream.discardBodyData(MultipartStream.java:619) at org.apache.commons.fileupload.MultipartStream.skipPreamble(MultipartStream.java:638) at org.apache.commons.fileupload.FileUploadBase$FileItemIteratorImpl.findNextItem(FileUploadBase.java:961) at org.apache.commons.fileupload.FileUploadBase$FileItemIteratorImpl.<init>(FileUploadBase.java:942) at org.apache.commons.fileupload.FileUploadBase.getItemIterator(FileUploadBase.java:331) at org.apache.commons.fileupload.FileUploadBase.parseRequest(FileUploadBase.java:349) ... 21 more
b-) On Arcgis Server (DEBUG Logs) we are also getting "Unable to process request. Error parsing multi-part request" as a high level error.
c-) Here are the IIS logs from the Web-Adpator site. (Please note that although some ports made it to the arcgis server the last upload part is failing with code 500)
2015-05-07 07:09:36 141.xxx.xxx.xxx POST /server/tokens/ f=json 443 - 82.148.xxx.xxx Collector-iOS-10.3:ArcGISiOS-10.2.4+Collector/8.1.2/iPad5,4 200 0 0 2979 2015-05-07 07:09:44 141.xxx.xxx.xxx POST /server/rest/services/TEAM2100/Collector_Active_Assets/FeatureServer/uploads/register - 443 - 82.148.44.205 Collector-iOS-10.3:ArcGISiOS-10.2.4+Collector/8.1.2/iPad5,4 200 0 0 920 2015-05-07 07:09:44 141.xxx.xxx.xxx POST /server/rest/services/TEAM2100/Collector_Active_Assets_Redline/FeatureServer/synchronizeReplica - 443 - 82.148.44.205 Collector-iOS-10.3:ArcGISiOS-10.2.4+Collector/8.1.2/iPad5,4 200 0 0 8002 2015-05-07 07:09:46 141.xxx.xxx.xxx GET /server/rest/services/TEAM2100/Collector_Active_Assets_Redline/FeatureServer/jobs/jaa5f0db584ea4053acbdd6ef428390d5 f=json&token=C3-2a7tyzySX4nq1NMOsL2Y2j89rGiXmR68hT7bnOuyr6I9UrafswZP6WmLcbERz 443 - 82.148.44.205 Collector-iOS-10.3:ArcGISiOS-10.2.4+Collector/8.1.2/iPad5,4 200 0 0 1419 2015-05-07 07:09:54 141.xxx.xxx.xxx GET /server/rest/services/TEAM2100/Collector_Active_Assets_Redline/FeatureServer/jobs/jaa5f0db584ea4053acbdd6ef428390d5 f=json&token=C3-2a7tyzySX4nq1NMOsL2Y2j89rGiXmR68hT7bnOuyr6I9UrafswZP6WmLcbERz 443 - 82.148.44.205 Collector-iOS-10.3:ArcGISiOS-10.2.4+Collector/8.1.2/iPad5,4 200 0 0 967 2015-05-07 07:10:01 141.xxx.xxx.xxx GET /server/rest/services/TEAM2100/Collector_Active_Assets_Redline/FeatureServer/jobs/jaa5f0db584ea4053acbdd6ef428390d5 f=json&token=C3-2a7tyzySX4nq1NMOsL2Y2j89rGiXmR68hT7bnOuyr6I9UrafswZP6WmLcbERz 443 - 82.148.44.205 Collector-iOS-10.3:ArcGISiOS-10.2.4+Collector/8.1.2/iPad5,4 200 0 0 1872 2015-05-07 07:10:45 141.xxx.xxx.xxx POST /server/rest/services/TEAM2100/Collector_Active_Assets/FeatureServer/uploads/i8aa3ef1e-fec4-48b2-bc9f-5d697178d051/uploadPart token=C3-2a7tyzySX4nq1NMOsL2Y2j89rGiXmR68hT7bnOuyr6I9UrafswZP6WmLcbERz 443 - 82.148.44.205 Collector-iOS-10.3:ArcGISiOS-10.2.4+Collector/8.1.2/iPad5,4 200 0 0 60824 2015-05-07 07:11:56 141.xxx.xxx.xxx POST /server/rest/services/TEAM2100/Collector_Active_Assets/FeatureServer/uploads/i8aa3ef1e-fec4-48b2-bc9f-5d697178d051/uploadPart token=C3-2a7tyzySX4nq1NMOsL2Y2j89rGiXmR68hT7bnOuyr6I9UrafswZP6WmLcbERz 443 - 82.148.44.205 Collector-iOS-10.3:ArcGISiOS-10.2.4+Collector/8.1.2/iPad5,4 200 0 0 71292 2015-05-07 07:13:01 141.xxx.xxx.xxx POST /server/rest/services/TEAM2100/Collector_Active_Assets/FeatureServer/uploads/i8aa3ef1e-fec4-48b2-bc9f-5d697178d051/uploadPart token=C3-2a7tyzySX4nq1NMOsL2Y2j89rGiXmR68hT7bnOuyr6I9UrafswZP6WmLcbERz 443 - 82.148.44.205 Collector-iOS-10.3:ArcGISiOS-10.2.4+Collector/8.1.2/iPad5,4 500 0 0 64225 #Software: Microsoft Internet Information Services 7.5
Things we have tried with no success :
NIM099731 - The error message, "Error parsing multi-part reque..
Re: Sync error: Illegal start of token [<]
Any ideas on why this happens ?
Thanks
Bledar,
Do you mean you have tried to increase the uploadReadAheadSize value on IIS?
Here are the steps:
1. Launch “Internet Information Services (IIS) Manager”
2. Select the site that you are hosting your web application under it.
3. In the Features section, double click “Configuration Editor”
4. Under “Section” select: system.webServer then serverRuntime
5. Modify the “uploadReadAheadSize” section to ~20MB
Yes i have made these changes and the problem is not resolved yet.
Here is more information on this issue.
a-) I tested the offline editing with about 100MB of attachments with ArcMap and the sync failed with the same reason.
b-) I saw that the error was coming from one of the 4 clustered arcgis server machines and i stopped the other 3 and left the default cluster with only one machine.
c-) After that the ArcMap offline map synced ok but the Collector app was still getting an error.
Here is the error on the server side.
SEVERE | May 12, 2015, 6:23:37 PM | Unable to process request. Error parsing multi-part request | Rest |
Bledar,
The development team has confirmed that there is a bug in ArcGIS Server that prevents huge uploads (greater than a few GB on fast networks and greater than 100 MB on slower networks). We are sorry about this problem and this issue is under consideration in the 10.4 release. I would recommend you to contact the Support to see if they can come up with a work around.
Thank you Zhang for the information.
Can you post here the Bug Number for this issue, so i can track it ?
We have been having this problem somewhat sporadically for a couple months with Collector and service that is used to take notes/pictures in the field. It appears that your information has helped us identify what is causing our errors so we can work on finding a workaround.
Yep, this has been causing us a lot of issues. Thanks for confirming it was a bug. Any idea if this is an issue when using hosted feature services in AGOL as well?
FYI - the illegal start of token errors are what we see when our users are trying to check in a database that is too large (somewhere in the neighborhood of ~800mb - 1gb). We see this on our projects with lots of image attachments.
Hey Bledar,
I noticed in your initial post that the following patch was not mentioned:
This may improve sync performance since you are at ArcGIS for Server 10.2.2, in addition to the other modifications outlined byMiaogeng Zhang previously.
Hi
I've still got this issue, and I am running on 10.3.1 Server and 10.3.6 Collector and connecting via a pretty fast internet connection.
I have 10 features with 20 photos (taken before the option of setting the size, so will be at full iPad size) I have a call in with local support (Esri(UK) but my main goal here is just to get these features off.
Cheers
ACM