We are trying to use GeoEvent server to pull in updated data from an external system using the "Watch a Folder for new GeoJSON Files" input connector.
Right now, the set-up is that the server machine housing GeoEvent calls an external URL to download a GeoJSON each hour and saves it in a folder on the machine. Then GeoEvent watches that folder for new GeoJSONs. This process was working as expected with the initial GeoJSON pull that it was tried with. We then realized there was a different GeoJSON we should be pulling, and updated the call on the server machine to pull that GeoJSON. The only difference between these GeoJSONs is their size (due to an increase in features included) - the original pull was about 3.5 MB and the new one is about 25.5 MB.
Is there a limit for the size of a GeoJSON that GeoEvent can handle? Or is there something additional we should look at?
For reference:
Solved! Go to Solution.
For those who may come upon this post, here is how we've finally resolved this problem.
When downloaded from the source, the GeoJSON in question is about 28 million characters on one line. We learned through a support ticket with Esri that GeoEvent does not process one-line files well and that there is an existing bug about GeoEvent having trouble reading exceptionally large files (in terms of characters) - BUG-000124967.
With these two things in mind, as well as the help received from @EricIronside above (thanks again!), we followed these steps to pre-process the GeoJSON before letting GeoEvent read it:
I also changed the number of lines per batch in the "Watch a Folder for new GeoJSON Files" input connector from the default to 2 (due to the length of each line), with a wait time of 500 milliseconds (the default), which slowly and methodically reads in all approx. 1700 features, 4 per second. Since we are updating this data daily, this longer processing time works for us, but this process may cause problems if you need to update the data on a much shorter timescale.
Hey,
Seems like you've done everything right. Two things to check:
The first is the filename. GeoEvent may ignore a file it thinks it has already processed, so try to append a datetime to the end of the filename as it is saved.
For the second, please see the following blog on wathching folders for files. Sometimes the file is large enough that the file system takes some time to write the file; if GeoEvent tries to grab the file while it is still being written, it can cause issues.
@EricIronsideThanks for the tips! I will look into both of those, I could see both items contributing to the problem.
@EricIronsideThanks again for the tips - we finally got a chance to come back to this project and test out the suggestions in the blog you linked, and it appears our GeoEvent server is now able to read the GeoJSON file!
Unfortunately, we are now receiving rather generic errors in the GeoEvent server logs that indicate a problem reading in the GeoJSON format; below is the text of one of these errors:
com.esri.ges.adapter.geojson.GeoJsonInboundAdapter Failed to create a GeoEvent from incoming JSON feature. Unexpected error occurred. Apr 28, 2021, 3:34:23 PM ERROR
I have also grabbed and made generic a chunk of the GeoJSON in question; based on my understanding, it follows the appropriate GeoJSON formatting, and I can't identify what the problem might be:
{
"features": [
{
"geometry": {
"coordinates": [
[
[
[0, 0],
[0, 1],
[1, 1],
[1, 0],
[0, 0],
]
]
],
"type": "MultiPolygon"
},
"id": 7,
"properties": {
"author": "user",
"changeset_comment": "#comments here as text",
"created": "2016-02-09T15:45:38.567808",
"done": 0,
"lbl_list": [
null
],
"link": "https://url is here",
"pid": 7,
"priority": 2,
"project_type": "mapping",
"status": 0,
"title": "Example Data",
"updated": "2016-04-27T15:54:27.425713",
"validated": 0
},
"type": "Feature"
},
"type": "FeatureCollection"
}
Do you have any insight into what might be wrong or have any tips on how we can get GeoEvent to properly process this GeoJSON now that it is able to find it?
I will take a look and see if I can get GeoEvent to bring this in. I assume the following are just oversight when constructing the example GeoJSON:
1. Line 12: '[0,0],' should be '[0,0]' (remove the extra comma)
2. 3rd Line from the bottom: '},' should be '}],' (close the feature array)
3. The geometry should follow the 'right-hand' rule (change the order to '[0, 0],[1, 0],[1, 1],[0, 1],[0, 0]')
{
"features": [{
"geometry": {
"coordinates": [
[
[
[0, 0],
[0, 1],
[1, 1],
[1, 0],
[0, 0]
]
]
],
"type": "MultiPolygon"
},
"id": 7,
"properties": {
"author": "user",
"changeset_comment": "#comments here as text",
"created": "2016-02-09T15:45:38.567808",
"done": 0,
"lbl_list": [null],
"link": "https://nome.vgihub.geointservices.io/tasks/project/7",
"pid": 7,
"priority": 2,
"project_type": "mapping",
"status": 0,
"title": "Example Data",
"updated": "2016-04-27T15:54:27.425713",
"validated": 0
},
"type": "Feature"
}],
"type": "FeatureCollection"
}
Putting my modified GeoJSON into a file and reading that file in using a 10.8.1 GES worked and it was able to create the event with a proper geometry. So I would check your data (maybe use an online validator such as https://geojsonlint.com/ ).
Thank you very much for the help! Based on the online validator, it does seem that there may be a formatting problem with the original geojson file that I did not catch.
For those who may come upon this post, here is how we've finally resolved this problem.
When downloaded from the source, the GeoJSON in question is about 28 million characters on one line. We learned through a support ticket with Esri that GeoEvent does not process one-line files well and that there is an existing bug about GeoEvent having trouble reading exceptionally large files (in terms of characters) - BUG-000124967.
With these two things in mind, as well as the help received from @EricIronside above (thanks again!), we followed these steps to pre-process the GeoJSON before letting GeoEvent read it:
I also changed the number of lines per batch in the "Watch a Folder for new GeoJSON Files" input connector from the default to 2 (due to the length of each line), with a wait time of 500 milliseconds (the default), which slowly and methodically reads in all approx. 1700 features, 4 per second. Since we are updating this data daily, this longer processing time works for us, but this process may cause problems if you need to update the data on a much shorter timescale.