Reconstruct tracks failed with "Request Entity Too Large" during stage "mapPartitions" using AIS Data.

Jan Tschada

We are trying to reconstruct tracks using AIS data and our GeoAnalytics  Server fail with some "Request Entity Too Large" during mapping the partitions. As far as we can judge it, sparks wants to write into the BDFS being based on Elasticsearch. Maybe we need to define some spark configuration to adjust the size of the data being written?

{"messageCode":"BD_101029","message":"193/372 distributed tasks completed.","params":{"completedTasks":"193","totalTasks":"372"}}
[ERROR|09:01:47] Stage 'runJob at EsSpark.scala:108' failed in GeoAnalytics job (attempt=0): Job aborted due to stage failure: Task 9 in stage 1.0 failed 4 times, most recent failure: Lost task 9.3 in stage 1.0 (TID 207, ##.##.##.###:####, executor 1): org.elasticsearch.hadoop.rest.EsHadoopInvalidRequest: [PUT] on [/_bulk] failed; server[##.##.##.###:####] returned [413|Request Entity Too Large:]
	at org.elasticsearch.hadoop.rest.RestClient.checkResponse(RestClient.java:477)
	at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:434)
	at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:428)
	at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:408)
	at org.elasticsearch.hadoop.rest.RestClient.bulk(RestClient.java:226)
	at org.elasticsearch.hadoop.rest.bulk.BulkProcessor.tryFlush(BulkProcessor.java:215)
	at org.elasticsearch.hadoop.rest.bulk.BulkProcessor.flush(BulkProcessor.java:518)
	at org.elasticsearch.hadoop.rest.bulk.BulkProcessor.add(BulkProcessor.java:132)
	at org.elasticsearch.hadoop.rest.RestRepository.doWriteToIndex(RestRepository.java:192)
	at org.elasticsearch.hadoop.rest.RestRepository.writeToIndex(RestRepository.java:172)
	at org.elasticsearch.spark.rdd.EsRDDWriter.write(EsRDDWriter.scala:77)
	at org.elasticsearch.spark.rdd.EsSpark$.$anonfun$doSaveToEs$1(EsSpark.scala:108)
	at org.elasticsearch.spark.rdd.EsSpark$.$anonfun$doSaveToEs$1$adapted(EsSpark.scala:108)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
	at org.apache.spark.scheduler.Task.run(Task.scala:127)
	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:446)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:449)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

Driver stacktrace:.
{"messageCode":"BD_101030","message":"Failed to execute ReconstructTracks.  Please contact your administrator with job ID 'jedad0ac11f654d64bc31ecc732c295be'.","params":{"toolName":"ReconstructTracks","jobID":"jedad0ac11f654d64bc31ecc732c295be"}}
[ERROR|09:01:47] Job 'jedad0ac11f654d64bc31ecc732c295be' for tool 'ReconstructTracks' failed: Job aborted due to stage failure: Task 9 in stage 1.0 failed 4 times, most recent failure: Lost task 9.3 in stage 1.0 (TID 207, ##.##.##.###, executor 1): org.elasticsearch.hadoop.rest.EsHadoopInvalidRequest: [PUT] on [/_bulk] failed; server[##.##.##.###:####] returned [413|Request Entity Too Large:]
	at org.elasticsearch.hadoop.rest.RestClient.checkResponse(RestClient.java:477)
	at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:434)
	at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:428)
	at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:408)
	at org.elasticsearch.hadoop.rest.RestClient.bulk(RestClient.java:226)
	at org.elasticsearch.hadoop.rest.bulk.BulkProcessor.tryFlush(BulkProcessor.java:215)
	at org.elasticsearch.hadoop.rest.bulk.BulkProcessor.flush(BulkProcessor.java:518)
	at org.elasticsearch.hadoop.rest.bulk.BulkProcessor.add(BulkProcessor.java:132)
	at org.elasticsearch.hadoop.rest.RestRepository.doWriteToIndex(RestRepository.java:192)
	at org.elasticsearch.hadoop.rest.RestRepository.writeToIndex(RestRepository.java:172)
	at org.elasticsearch.spark.rdd.EsRDDWriter.write(EsRDDWriter.scala:77)
	at org.elasticsearch.spark.rdd.EsSpark$.$anonfun$doSaveToEs$1(EsSpark.scala:108)
	at org.elasticsearch.spark.rdd.EsSpark$.$anonfun$doSaveToEs$1$adapted(EsSpark.scala:108)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
	at org.apache.spark.scheduler.Task.run(Task.scala:127)
	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:446)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:449)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

Driver stacktrace:. [PUT] on [/_bulk] failed; server[##.##.##.###:####] returned [413|Request Entity Too Large:]. 
[DEBUG|09:01:47] Job 'jedad0ac11f654d64bc31ecc732c295be' for tool 'ReconstructTracks' failed: Job aborted due to stage failure: Task 9 in stage 1.0 failed 4 times, most recent failure: Lost task 9.3 in stage 1.0 (TID 207, ##.##.##.###, executor 1): org.elasticsearch.hadoop.rest.EsHadoopInvalidRequest: [PUT] on [/_bulk] failed; server[##.##.##.###:####] returned [413|Request Entity Too Large:]
	at org.elasticsearch.hadoop.rest.RestClient.checkResponse(RestClient.java:477)
	at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:434)
	at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:428)
	at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:408)
	at org.elasticsearch.hadoop.rest.RestClient.bulk(RestClient.java:226)
	at org.elasticsearch.hadoop.rest.bulk.BulkProcessor.tryFlush(BulkProcessor.java:215)
	at org.elasticsearch.hadoop.rest.bulk.BulkProcessor.flush(BulkProcessor.java:518)
	at org.elasticsearch.hadoop.rest.bulk.BulkProcessor.add(BulkProcessor.java:132)
	at org.elasticsearch.hadoop.rest.RestRepository.doWriteToIndex(RestRepository.java:192)
	at org.elasticsearch.hadoop.rest.RestRepository.writeToIndex(RestRepository.java:172)
	at org.elasticsearch.spark.rdd.EsRDDWriter.write(EsRDDWriter.scala:77)
	at org.elasticsearch.spark.rdd.EsSpark$.$anonfun$doSaveToEs$1(EsSpark.scala:108)
	at org.elasticsearch.spark.rdd.EsSpark$.$anonfun$doSaveToEs$1$adapted(EsSpark.scala:108)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
	at org.apache.spark.scheduler.Task.run(Task.scala:127)
	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:446)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:449)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

Driver stacktrace:. [PUT] on [/_bulk] failed; server[##.##.##.###:####] returned [413|Request Entity Too Large:]. 
Job aborted due to stage failure: Task 9 in stage 1.0 failed 4 times, most recent failure: Lost task 9.3 in stage 1.0 (TID 207, 10.23.35.141, executor 1): org.elasticsearch.hadoop.rest.EsHadoopInvalidRequest: [PUT] on [/_bulk] failed; server[##.##.##.###:####] returned [413|Request Entity Too Large:]
	at org.elasticsearch.hadoop.rest.RestClient.checkResponse(RestClient.java:477)
	at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:434)
	at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:428)
	at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:408)
	at org.elasticsearch.hadoop.rest.RestClient.bulk(RestClient.java:226)
	at org.elasticsearch.hadoop.rest.bulk.BulkProcessor.tryFlush(BulkProcessor.java:215)
	at org.elasticsearch.hadoop.rest.bulk.BulkProcessor.flush(BulkProcessor.java:518)
	at org.elasticsearch.hadoop.rest.bulk.BulkProcessor.add(BulkProcessor.java:132)
	at org.elasticsearch.hadoop.rest.RestRepository.doWriteToIndex(RestRepository.java:192)
	at org.elasticsearch.hadoop.rest.RestRepository.writeToIndex(RestRepository.java:172)
	at org.elasticsearch.spark.rdd.EsRDDWriter.write(EsRDDWriter.scala:77)
	at org.elasticsearch.spark.rdd.EsSpark$.$anonfun$doSaveToEs$1(EsSpark.scala:108)
	at org.elasticsearch.spark.rdd.EsSpark$.$anonfun$doSaveToEs$1$adapted(EsSpark.scala:108)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
	at org.apache.spark.scheduler.Task.run(Task.scala:127)
	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:446)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:449)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

Driver stacktrace:
[DEBUG]org.apache.spark.SparkException: Job aborted due to stage failure: Task 9 in stage 1.0 failed 4 times, most recent failure: Lost task 9.3 in stage 1.0 (TID 207, ##.##.##.###, executor 1): org.elasticsearch.hadoop.rest.EsHadoopInvalidRequest: [PUT] on [/_bulk] failed; server[##.##.##.###:####] returned [413|Request Entity Too Large:]
	at org.elasticsearch.hadoop.rest.RestClient.checkResponse(RestClient.java:477)
	at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:434)
	at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:428)
	at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:408)
	at org.elasticsearch.hadoop.rest.RestClient.bulk(RestClient.java:226)
	at org.elasticsearch.hadoop.rest.bulk.BulkProcessor.tryFlush(BulkProcessor.java:215)
	at org.elasticsearch.hadoop.rest.bulk.BulkProcessor.flush(BulkProcessor.java:518)
	at org.elasticsearch.hadoop.rest.bulk.BulkProcessor.add(BulkProcessor.java:132)
	at org.elasticsearch.hadoop.rest.RestRepository.doWriteToIndex(RestRepository.java:192)
	at org.elasticsearch.hadoop.rest.RestRepository.writeToIndex(RestRepository.java:172)
	at org.elasticsearch.spark.rdd.EsRDDWriter.write(EsRDDWriter.scala:77)
	at org.elasticsearch.spark.rdd.EsSpark$.$anonfun$doSaveToEs$1(EsSpark.scala:108)
	at org.elasticsearch.spark.rdd.EsSpark$.$anonfun$doSaveToEs$1$adapted(EsSpark.scala:108)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
	at org.apache.spark.scheduler.Task.run(Task.scala:127)
	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:446)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:449)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

Driver stacktrace:
	at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2059)
	at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:2008)
	at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:2007)
	at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
	at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
	at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2007)
	at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:973)
	at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:973)
	at scala.Option.foreach(Option.scala:407)
	at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:973)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2239)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2188)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2177)
	at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
	at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:775)
	at org.apache.spark.SparkContext.runJob(SparkContext.scala:2099)
	at org.apache.spark.SparkContext.runJob(SparkContext.scala:2120)
	at org.apache.spark.SparkContext.runJob(SparkContext.scala:2152)
	at org.elasticsearch.spark.rdd.EsSpark$.doSaveToEs(EsSpark.scala:108)
	at org.elasticsearch.spark.rdd.EsSpark$.saveToEs(EsSpark.scala:79)
	at org.elasticsearch.spark.rdd.EsSpark$.saveToEs(EsSpark.scala:76)
	at org.elasticsearch.spark.package$SparkRDDFunctions.saveToEs(package.scala:56)
	at com.esri.arcgis.bds.spark.BDSSpark$.saveFeaturesToBDS(BDSSpark.scala:390)
	at com.esri.arcgis.bds.spark.BDSSpark$.saveFeaturesToBDS(BDSSpark.scala:238)
	at com.esri.arcgis.gae.gp.ags.SpatiotemporalDataStoreWriter.$anonfun$writeOnce$1(SpatiotemporalDataStoreWriter.scala:342)
	at scala.util.Try$.apply(Try.scala:213)
	at com.esri.arcgis.gae.gp.ags.SpatiotemporalDataStoreWriter.writeOnce(SpatiotemporalDataStoreWriter.scala:300)
	at com.esri.arcgis.gae.ags.datastore.managed.ManagedDataStoreWriter.writeAll(ManagedDataStoreWriter.scala:13)
	at com.esri.arcgis.gae.ags.datastore.managed.ManagedDataStoreWriter.writeAll$(ManagedDataStoreWriter.scala:8)
	at com.esri.arcgis.gae.gp.ags.SpatiotemporalDataStoreWriter.writeAll(SpatiotemporalDataStoreWriter.scala:29)
	at com.esri.arcgis.gae.ags.layer.output.ManagedLayerResultManager.addResultLayer(ManagedLayerResultManager.scala:214)
	at com.esri.arcgis.gae.gp.adapt.param.LayerOutputConverter.writeToManagedDataStore(LayerOutputConverter.scala:85)
	at com.esri.arcgis.gae.gp.adapt.param.LayerOutputConverter.write(LayerOutputConverter.scala:31)
	at com.esri.arcgis.gae.gp.adapt.param.FeatureRDDOutputConverter.write(FeatureRDDOutputConverter.scala:17)
	at com.esri.arcgis.gae.gp.adapt.param.FeatureRDDOutputConverter.write(FeatureRDDOutputConverter.scala:9)
	at com.esri.arcgis.gae.gp.adapt.param.GPParameterAdapter$$anon$2.writeParam(GPParameterAdapter.scala:77)
	at com.esri.arcgis.gae.fn.api.v1.GAFunctionArgs.update(GAFunctionArgs.scala:53)
	at com.esri.arcgis.gae.fn.FnReconstructTracks$.execute(FnReconstructTracks.scala:165)
	at com.esri.arcgis.gae.fn.api.v1.GAFunction.executeFunction(GAFunction.scala:25)
	at com.esri.arcgis.gae.fn.api.v1.GAFunction.executeFunction$(GAFunction.scala:15)
	at com.esri.arcgis.gae.fn.FnReconstructTracks$.executeFunction(FnReconstructTracks.scala:25)
	at com.esri.arcgis.gae.gp.GPToGAFunctionAdapter.$anonfun$execute$4(GPToGAFunctionAdapter.scala:157)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at com.esri.arcgis.st.util.ScopeManager.withScope(ScopeManager.scala:20)
	at com.esri.arcgis.st.spark.ExecutionContext.withScope(ExecutionContext.scala:23)
	at com.esri.arcgis.st.spark.ExecutionContext.withScope$(ExecutionContext.scala:23)
	at com.esri.arcgis.st.spark.GenericExecutionContext.withScope(ExecutionContext.scala:52)
	at com.esri.arcgis.gae.gp.GPToGAFunctionAdapter.execute(GPToGAFunctionAdapter.scala:155)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.esri.arcgis.interop.NativeObjRef.nativeVtblInvokeNative(Native Method)
	at com.esri.arcgis.interop.NativeObjRef.nativeVtblInvoke(Unknown Source)
	at com.esri.arcgis.interop.NativeObjRef.invoke(Unknown Source)
	at com.esri.arcgis.interop.Dispatch.vtblInvoke(Unknown Source)
	at com.esri.arcgis.system.IRequestHandlerProxy.handleStringRequest(Unknown Source)
	at com.esri.arcgis.discovery.servicelib.impl.SOThreadBase.handleSoapRequest(SOThreadBase.java:574)
	at com.esri.arcgis.discovery.servicelib.impl.DedicatedSOThread.handleRequest(DedicatedSOThread.java:310)
	at com.esri.arcgis.discovery.servicelib.impl.SOThreadBase.startRun(SOThreadBase.java:430)
	at com.esri.arcgis.discovery.servicelib.impl.DedicatedSOThread.run(DedicatedSOThread.java:166)
Caused by: org.elasticsearch.hadoop.rest.EsHadoopInvalidRequest: [PUT] on [/_bulk] failed; server[##.##.##.###:####] returned [413|Request Entity Too Large:]
	at org.elasticsearch.hadoop.rest.RestClient.checkResponse(RestClient.java:477)
	at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:434)
	at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:428)
	at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:408)
	at org.elasticsearch.hadoop.rest.RestClient.bulk(RestClient.java:226)
	at org.elasticsearch.hadoop.rest.bulk.BulkProcessor.tryFlush(BulkProcessor.java:215)
	at org.elasticsearch.hadoop.rest.bulk.BulkProcessor.flush(BulkProcessor.java:518)
	at org.elasticsearch.hadoop.rest.bulk.BulkProcessor.add(BulkProcessor.java:132)
	at org.elasticsearch.hadoop.rest.RestRepository.doWriteToIndex(RestRepository.java:192)
	at org.elasticsearch.hadoop.rest.RestRepository.writeToIndex(RestRepository.java:172)
	at org.elasticsearch.spark.rdd.EsRDDWriter.write(EsRDDWriter.scala:77)
	at org.elasticsearch.spark.rdd.EsSpark$.$anonfun$doSaveToEs$1(EsSpark.scala:108)
	at org.elasticsearch.spark.rdd.EsSpark$.$anonfun$doSaveToEs$1$adapted(EsSpark.scala:108)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
	at org.apache.spark.scheduler.Task.run(Task.scala:127)
	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:446)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:449)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

[DEBUG|09:01:47] Running cleanup tasks for conditions (Error,Complete)
[DEBUG|09:01:47] Running cleanup task [? @ DebugUtil.scala:106].

 

NoahSlocum

Hi, from the logs you shared it looks like your GeoAnalytics Server is failing to write a feature service to the Spatiotemporal Big Data Store (different from a BDFS [Big Data File Share], which you can also write to). This can happen when the lines created by the Reconstruct Tracks tool are too large. One way to avoid this is by splitting your tracks into smaller segments using a time, distance, or attribute criteria when running Reconstruct Tracks. Can you please try using one of the four track splitting methods described here to break up your AIS tracks into smaller segments and see if the tool succeeds?