nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Having trouble with the GeoEvent input connector "Watch a Folder for new GeoJSON Files"

AbigailStephens

We are trying to use GeoEvent server to pull in updated data from an external system using the "Watch a Folder for new GeoJSON Files" input connector.

Right now, the set-up is that the server machine housing GeoEvent calls an external URL to download a GeoJSON each hour and saves it in a folder on the machine. Then GeoEvent watches that folder for new GeoJSONs. This process was working as expected with the initial GeoJSON pull that it was tried with. We then realized there was a different GeoJSON we should be pulling, and updated the call on the server machine to pull that GeoJSON. The only difference between these GeoJSONs is their size (due to an increase in features included) - the original pull was about 3.5 MB and the new one is about 25.5 MB.

Is there a limit for the size of a GeoJSON that GeoEvent can handle? Or is there something additional we should look at?

For reference:

We are using ArcGIS Enterprise 10.7.1 on Linux
All the data in both is in one line, so changing the settings for line batch processing hasn't had any effect
The Input Folder DataStore + Input Directory parameters give the full path to the folder in which the file is saved, and we have checked that the pulled and saved GeoJSON is in the folder indicated. The file filter is set to ".*\.geojson".
Based on preliminary review, there doesn't appear to be anything wrong with the format of the GeoJSON, and it properly converts to a feature class in ArcGIS Pro.

Trouver plus de publications taguées avec

Accepted answers

AbigailStephens

For those who may come upon this post, here is how we've finally resolved this problem.

When downloaded from the source, the GeoJSON in question is about 28 million characters on one line. We learned through a support ticket with Esri that GeoEvent does not process one-line files well and that there is an existing bug about GeoEvent having trouble reading exceptionally large files (in terms of characters) - BUG-000124967.

With these two things in mind, as well as the help received from @EricIronside above (thanks again!), we followed these steps to pre-process the GeoJSON before letting GeoEvent read it:

Processed the file to place each feature on a line, bringing the total number of lines to around 2,000
- We also considered breaking the file into even more lines, but ended up not needing to do this.
Make sure all polygons follow the right-hand rule (originally, some did, but not all)
Use mapshaper to simplify the polygons to 30% of previous vertices (this reduces the number of characters significantly, as one extremely detailed feature goes from 219,000 vertices to about 70,000)
- A note: for our use, simplifying the polygons to this degree does not impact the usefulness of the data. You may want to consider different simplification parameters if the accuracy of your data needs to be high.

I also changed the number of lines per batch in the "Watch a Folder for new GeoJSON Files" input connector from the default to 2 (due to the length of each line), with a wait time of 500 milliseconds (the default), which slowly and methodically reads in all approx. 1700 features, 4 per second. Since we are updating this data daily, this longer processing time works for us, but this process may cause problems if you need to update the data on a much shorter timescale.

All comments

EricIronside

Hey,

Seems like you've done everything right. Two things to check:

The first is the filename. GeoEvent may ignore a file it thinks it has already processed, so try to append a datetime to the end of the filename as it is saved.

For the second, please see the following blog on wathching folders for files. Sometimes the file is large enough that the file system takes some time to write the file; if GeoEvent tries to grab the file while it is still being written, it can cause issues.

GeoEvent: Watch a folder for new files

AbigailStephens

@EricIronsideThanks for the tips! I will look into both of those, I could see both items contributing to the problem.

AbigailStephens

@EricIronsideThanks again for the tips - we finally got a chance to come back to this project and test out the suggestions in the blog you linked, and it appears our GeoEvent server is now able to read the GeoJSON file!

Unfortunately, we are now receiving rather generic errors in the GeoEvent server logs that indicate a problem reading in the GeoJSON format; below is the text of one of these errors:

com.esri.ges.adapter.geojson.GeoJsonInboundAdapter  Failed to create a GeoEvent from incoming JSON feature. Unexpected error occurred.  Apr 28, 2021, 3:34:23 PM  ERROR

I have also grabbed and made generic a chunk of the GeoJSON in question; based on my understanding, it follows the appropriate GeoJSON formatting, and I can't identify what the problem might be:

{
  "features": [
    {
      "geometry": {
        "coordinates": [
          [
            [
              [0, 0],
              [0, 1],
              [1, 1],
              [1, 0],
              [0, 0],
            ]
          ]
        ],
        "type": "MultiPolygon"
      },
      "id": 7,
      "properties": {
        "author": "user",
        "changeset_comment": "#comments here as text",
        "created": "2016-02-09T15:45:38.567808",
        "done": 0,
        "lbl_list": [
          null
        ],
        "link": "https://url is here",
        "pid": 7,
        "priority": 2,
        "project_type": "mapping",
        "status": 0,
        "title": "Example Data",
        "updated": "2016-04-27T15:54:27.425713",
        "validated": 0
      },
      "type": "Feature"
    },
  "type": "FeatureCollection"
}

Do you have any insight into what might be wrong or have any tips on how we can get GeoEvent to properly process this GeoJSON now that it is able to find it?

EricIronside

I will take a look and see if I can get GeoEvent to bring this in. I assume the following are just oversight when constructing the example GeoJSON:

1. Line 12: '[0,0],' should be '[0,0]' (remove the extra comma)

2. 3rd Line from the bottom: '},' should be '}],' (close the feature array)

3. The geometry should follow the 'right-hand' rule (change the order to '[0, 0],[1, 0],[1, 1],[0, 1],[0, 0]')

{
"features": [{
"geometry": {
"coordinates": [
[
[
[0, 0],
[0, 1],
[1, 1],
[1, 0],
[0, 0]
]
]
],
"type": "MultiPolygon"
},
"id": 7,
"properties": {
"author": "user",
"changeset_comment": "#comments here as text",
"created": "2016-02-09T15:45:38.567808",
"done": 0,
"lbl_list": [null],
"link": "https://nome.vgihub.geointservices.io/tasks/project/7",
"pid": 7,
"priority": 2,
"project_type": "mapping",
"status": 0,
"title": "Example Data",
"updated": "2016-04-27T15:54:27.425713",
"validated": 0
},
"type": "Feature"
}],
"type": "FeatureCollection"
}

EricIronside

Putting my modified GeoJSON into a file and reading that file in using a 10.8.1 GES worked and it was able to create the event with a proper geometry. So I would check your data (maybe use an online validator such as https://geojsonlint.com/ ).

AbigailStephens

Thank you very much for the help! Based on the online validator, it does seem that there may be a formatting problem with the original geojson file that I did not catch.

AbigailStephens

For those who may come upon this post, here is how we've finally resolved this problem.

With these two things in mind, as well as the help received from @EricIronside above (thanks again!), we followed these steps to pre-process the GeoJSON before letting GeoEvent read it:

Processed the file to place each feature on a line, bringing the total number of lines to around 2,000
- We also considered breaking the file into even more lines, but ended up not needing to do this.
Make sure all polygons follow the right-hand rule (originally, some did, but not all)
Use mapshaper to simplify the polygons to 30% of previous vertices (this reduces the number of characters significantly, as one extremely detailed feature goes from 219,000 vertices to about 70,000)
- A note: for our use, simplifying the polygons to this degree does not impact the usefulness of the data. You may want to consider different simplification parameters if the accuracy of your data needs to be high.

Rejoindre la conversation

Les membres connectés peuvent publier, suivre les mises à jour, et plus encore. Nouveau ici ? Inscrivez-vous gratuitement.

Liens rapides

Community Help
Find useful guides, FAQs, and documents to help you navigate and make the most of Esri Community.