Latest Contributions by BruceHarold

A recent blog of mine discussed how ArcGIS Data Interoperability relates to FME but also clarified the licensing situation when FME Flow is being used to access ArcGIS executables, particularly for writing to an Enterprise geodatabase.  In that situation, other than for the file geodatabase exception noted,  the ArcGIS software to be used must be ArcGIS Server or Enterprise.  This isn't always convenient for an FME Flow site, so in this blog I'll give you an alternative - namely using a web tool without a local ArcGIS install. Here is my example, a model tool: Web Tool Source Model The whole point here is to give FME Server users a way to write Enterprise geodatabase objects using a web service and not an executable.  If your interest is in reading Enterprise geodatabases with FME Flow you can already do this with the ArcGIS Server Feature Service reader provided you publish a feature service from an Enterprise geodatabase object.  Let's get started. The important bits in the above Model are: The input workspace parameter is the target output Enterprise geodatabase The input data parameter is a file-based format FME writes and ArcGIS reads At writing, the teams at Safe Software and Esri are putting effort into enhancing the FME EsriJSON reader/writer to fully support a schema object, but we're not done yet.  Regardless, I'm going to use EsriJSON as my data protocol.  I made test data with the Features to JSON geoprocessing tool, not FME yet. There are other options like GeoPackage you might choose, and more coming in the future, but I want to show you how to use EsriJSON for now, not least because you will be able to trust the geometry and schema to be reliable in a destination geodatabase. Is there anything special about the Enterprise geodatabase workspace parameter?  When you add it to your model it is just a database connection file with ".sde" extension but put a little thought into it.  I made a new connection file with the Manage Registered Data Stores  dialog, and indeed registered my Enterprise geodatabase as a data store.  When publishing a web tool, datasets may be copied to the server or referenced from data stores.  If your web tool needs to leverage any data that isn't static then do what I did.  Making the Enterprise workspace a parameter protects it from being copied but any other datasets need to have this behavior managed. Why is the EsriJSON file zipped?  Obviously this helps file upload speed but it is also necessary because of how we use the web tool - i.e. via its REST endpoint and calling from FME Server.  When a web tool supports a file input parameter the file must be supplied as a URL or server or portal item and not a POST upload.  Any JSON item uploaded unzipped will be recognized (incorrectly in our case) as GeoJSON, found to be invalid and rejected.  Zipping it protects it from this behavior.  Even then, EsriJSON isn't a supported portal item type and we have to tell the system it's a Code Sample item.  This is a bit ironic, given I'm the no-code guy in the office, but I'll take the leave pass. My web tool doesn't do any fancy geoprocessing, it only serves to demonstrate the EGDB writing approach, but we'll step through it for completeness. The first Calculate Value model tool downstream of the workspace variable generates a unique output  feature class name for the target geodatabase.  The only notable thing is I make sure to use raw string processing for the workspace variable (r"%DataStoreDBA%") because on the server it is a complex path with metacharacters like "\a" that would otherwise be interpreted.  For example: C:\arcgisserver\directories\arcgissystem\arcgisinput\EsriJSON2EGDB.GPServer\extracted\p30\DataStoreDBA1.sde\FromEsriJSON0 First Calculate Value Tool   The second Calculate Value model tool downstream of the data file input unzips the data to the scratch folder. Second Calculate Value Tool  On the server an example path in the job folder looks like: C:\arcgisserver\directories\arcgisjobs\esrijson2egdb_gpserver\jf37903613c4f4e5685efc5e0861fdbbd\scratch\FeaturesToJSON_OutJsonFile_Nelson.json   Otherwise the only processing is the write we want - the JSON to Features geoprocessing tool sends data to where we want it (your processing may be much more complex).  After a couple of executions I have data in my EGDB: Data in the Enterprise GDB     After publishing a tool result as a web tool I have a portal tool item with a REST submitJob endpoint: SubmitJob Endpoint   Now we have something FME can use via HTTP to write data to the Enterprise geodatabase.  Before we dig into that there are a couple of things to consider.  In this workflow you are responsible for managing timeouts and contention in the data going up to the portal (where the REST endpoint gets its input data), processing, and in the target database.  I went into my Server Manager console and made sure to limit my web tool to a single instance with a 15 minute timeout.  That way I can handle large files and always one at a time. Drum roll please - here is the FME workspace (Main canvas) that calls the web tool: FME Workspace   ...and the looping custom transformer in the middle looks like this: Looping Custom Transformer Everything is in the blog download (authored in Pro 3.0) but let's step through the FME workspace.  In your real world, FME Flow will be cooking up data to be written to the EGDB.  I don't do that in my sample but you will have to make file-based data to send.  My input parameters are a user-defined zipped JSON file and the web tool home URL (not published, as it is static): https://dev0015493.esri.com/server/rest/services/EsriJSON2EGDB/GPServer/EsriJSON2EGDB The first step is to get a portal token, that's the first (green) transformer.  Then the input zip file is uploaded as a portal Code Sample item.  Code Sample items contain a file that must have a unique name per folder, regardless of whether the Code Sample item name is itself unique.  Just in case of a name collision I delete any existing items with the same content name (found by rejection), then upload the input data. After that it's just HTTP to send each job, a looping custom transformer to wait for job completion, and then I delete the portal item we are finished with and finally email a result message to any interested parties. Now I'm off to the races, any time my FME job runs I get data written to my EGDB, and without installing any ArcGIS software on my FME machine! More Output Data Don't forget EsriJSON is a work in progress at Esri & Safe Software but you can get going now with other file-based formats. ... View more

We do our best to automate things but sometimes you need to wire up FME-technology parameters with Esri geoprocessing for yourself.  ModelBuilder is your friend. Here is my example, an ETL tool process needs help from a couple of Calculate Value model tools.  Let's see why. Modelbuilder making it all work A customer asked how to build a web tool to power a Web AppBuilder geoprocessing widget, it had to work a certain way.  Excel files needed to be uploaded and zipped CSV data had to be returned, not just one zipped CSV file per Excel input file but multiple CSV files in each zipfile.  Okay.....no map in sight, just my kind of job! You can pretty quickly stub out a Spatial ETL tool that applies any business logic you need to go from Excel to CSV (the two most popular non-spatial dataset types in the ETL world by the way).   But, what about zipping them up and returning the zipfile as an output? FME technology has some paradigm differences with ArcGIS.  FME Workbench tends to work with what the Esri world thinks of as workspaces - folders and databases.  When you configure an ETL tool you supply input and output datasets accordingly, but to the ArcGIS geoprocessing environment everything looks like an input parameter.  How do you get data out for downstream geoprocessing or as a result in an app like Pro?  The simplest way is with ModelBuilder as part of the web tool. If your workflow is writing to a geodatabase then put your ETL tool in a model and add a Select Data model tool to extract the features you want.  You'll then have geoprocessing outputs available. For my example I had to get a little more creative.  The first challenge wasn't around parameter types, it was zipping multiple CSV outputs per input.  Many formats in FME technology support conversion to a zipfile.  The way to do this with CSV data is supply a zipfile name as the last part of the output dataset "folder".  Here is my ETL tool workspace: Excel to CSV With Feature Type Fanout   And here is my output "folder": CSV Writer Folder Path With Zipfile Path   If you look closely at the workspace you'll see I trigger a feature type fanout based on an attribute value - see the CSV File Name is an expression @Value(Environment).  This gives me one output CSV file per unique attribute value.  In the writer parameters I specify a path ending in "Output.zip".  In operating system terms it is a file but to Workbench (and ArcGIS when calling the ETL tool) it is still a folder. To make everything work as a web tool I need correct ArcGIS parameter types, folder input and file output.  Folders do not work as outputs.  I also want to specify a scratch file location for the output so that the job environment can see and return the data.  The trick is to wrap the ETL tool in a model and set paths and parameter types explicitly. The two Calculate Value model tools are key. The first one creates a zipfile path in the job scratch environment but casts it to Folder: Create Scratch Folder Zipfile Path   The second one takes the same path and casts it to File: Folder Cast To File   Then everyone is happy.  I can run the tool in Pro, it uses the local scratch environment, then I can share the result as a web tool and it will use the ArcGIS Server job scratch environment at run time. I get my data back in Pro and use Web AppBuilder against the web tool. Web Tool Output In Pro Simple, isn't it!  Now you can build your Spatial ETL web tools into your enterprise geoprocessing workflows. ... View more

My first flight since the pandemic hit was a trip to Vancouver, Canada last week, to attend our partner Safe Software's 2022 FME User Conference, where the no-code integration & ETL community could connect and learn.  I took a question at the event that prompted this blog - why does my change detection workspace think every feature has changed when I know only a few features really changed? In case you're new to this, ArcGIS Data Interoperability inherits a very powerful function from FME, namely change detection, embodied in the ChangeDetector transformer, or less frequently, the Matcher transformer.  Change detection is usually done between a source dataset and a derived dataset that is maintained as a mirror on a schedule, and the ability to write only data changes (inserts, updates, deletes) without downtime is a great advantage, especially for targets like feature services, which can remain in use during maintenance. This blog is about avoiding surprises when performing change detection. Back to my story, I had a ticketing problem that caused a delay at the gate for my northbound flight; the airline offered me a seat upgrade, so I chose a port side window seat to enjoy the view do some ground truthing on some data I was working on, namely fire locations in California.  My flight was in daytime but here is a dark theme map of fire locations during my trip. Fires in California   There are a few places to get near real time fire data, the source isn't relevant to the topic and what I relate isn't tied to any data format.  I happen to be reading JSON and maintaining a hosted feature service.  Let's tackle the #1 cause of fake change detection - geometry storage and retrieval. First I will illustrate the issue.  The blog attachment contains a CSV file with a single row, add the CSV table to a map then use the table context menu Display XY Data tool to create a layer from the XY fields, using WGS84 coordinate system.  Throw a little Python at the resulting layer and you will see what happens to the XY coordinates used to create the layer:         with arcpy.da.SearchCursor('Fires_XYTableToPoint',"*") as cursor: for row in cursor: pass row (1, (-120.70132999999998, 39.331022000000075), 1169, -120.70133, 39.331022)         So, what went in as -120.70133, 39.331022 was stored as -120.70132999999998, 39.331022000000075.  The data isn't being randomly shifted, it's just an artefact of how ArcGIS manages coordinates when it has to make some assumptions, and it has no practical implications except for our change detection case, because unless you say otherwise, coordinate changes will be detected with total strictness. I'm maintaining a mirror of the source fire event JSON feed as a feature service and I have a Spatial ETL workspace built to do it.  I'm reading the JSON on a schedule and writing only changes to the target feature service, the process uses a ChangeDetector.  Below is a screenshot of my ChangeDetector properties, take a note of the geometry handling.  I am checking for 2D differences, my data are points so lenient point order isn't relevant, I am not checking coordinate system names (note identical coordinate systems on inputs using  different names will cause "fake change") but most importantly I use a vector geometry tolerance of 0.000001.  If I used the default of zero any difference right down to subatomic particle size will be detected as a change, and we have seen above how that kind of number change can be introduced. Change Detector   How did I arrive at a vector tolerance of 0.000001?  My "personal defaults" for what are real geometry changes are at 3 decimal places for projected data and 6 decimal places for geographic data.  You may work to tighter tolerances but I have never seen such precision in production use.  My data are in WGS84 geographic coordinates so I use 6 decimal place tolerance.  If you prefer to make this aspect more visible in a workspace you can use a CoordinateRounder in each input data stream with the same tolerances and use a vector tolerance of 0 in the ChangeDetector. So that lets you avoid the #1 change detection pitfall, fake geometry "changes".  Are there similar precision-based ones out there?  Yes!  You'll see in my ChangeDetector I have set a name for the Generate Detailed Changes list parameter.  This causes the transformer to output a list that exposes the actual data values driving change.  I have a field "date_created" in my data and I notice every feature seems to change, but only at the microsecond level - definitely fake change.  Here is a log excerpt: Attribute(string: UTF-8) : `delta{1}.action' has value `modified' Attribute(string: UTF-8) : `delta{1}.attributeName' has value `date_created' Attribute(string: UTF-8) : `delta{1}.originalValue' has value `20220816155935.732000' Attribute(string: UTF-8) : `delta{1}.revisedValue' has value  `20220816155935.732935' I have no idea why this timestamp field is moving around, and I cannot know, you'll find yourself in this situation occasionally too, so just figure out ways to work around it. The simplest approach is to drop the fractional seconds from each date_created value.  It may be tempting to round the value but datetimes are tricky things, they are not always valid if rounded up (think 59.999999 seconds rounding up to 60 seconds, which should be 0 seconds in the next minute). So I chop off the fractional seconds with a StringReplacer defined as below in both input streams to my ChangeDetector: Remove fractional seconds   Sorry to throw regex at you but sometimes it's necessary. So that was another precision-based issue we avoided! There are other things to remember when performing change detection; the detailed change list is your friend, it will help you track down what is really changing.  Be sparing with the fields you use to detect changes too, accidentally leaving in format attributes for example will cause bulk change to be found when in fact the changes aren't really in your data. Once you have mastered change detection you'll be able to author very efficient workspaces, for example here is mine finding real changes in the fire data and writing only the updates.  You may spot that all features changed in this run (29 out of 29) which is a hint I may have more work to do... Fire feature service maintenance with change detection One more key point.  The ChangeDetector transformer outputs a format attribute fme_db_operation with values of INSERT, UPDATE or DELETE; when writing to feature services or databases this row-level attribute can be used by the writer to determine feature handling, set this behavior by setting the Feature Operation property in the writer - you don't have to have multiple writers for each mode. Using fme_db_operation   Don't put up with fake change!   ... View more

I'm coining a new term here - Total Integration - meaning the continuous, automated integration of any data and any ArcGIS service, this concept combines data movement with refreshing the web services the data powers. No manual ETL, no disconnected processes after the ETL, no downtime, just continuous Enterprise Service Bus maintenance the way it should be done, in one process. It has become trite to say you're "excited" to share details on functionality, but I really do feel using a Notebook as a lightweight scheduler will bring web tool automation to the masses. My example scenario happens to be about maintaining a portal locator (i.e. geocode service), a very common foundational service, but read between the lines and picture your full suite of services under continuous management no matter how the data powering them changes.  Of course, you don't have to be maintaining a service, your web tool may be doing any sort of geoprocessing. San Francisco Locator Data From Socrata There are a few parts to total integration, the trick is making them work together.  The logical steps are: Publish a web tool that moves your data and refreshes your target web service (if applicable) Automate the web tool on a schedule In my case I use ArcGIS Data Interoperability in the web tool but that's just an artifact of my data source.  The lightbulb moments for me were realising I could combine ETL with core geoprocessing to include service refresh in my web tool and also use a hosted ArcGIS Online hosted Notebook as a lightweight scheduler for complete automation. While I'm usually a no-code guy, there is a little Python involved, but you can relax, I'm sharing the code patterns in the blog download.  My blog scenario is at the high end in its usage of ArcPy but only because ArcGIS currently lacks a geoprocessing tool that rebuilds a portal locator (note to self: get one coming), much of the time you might build your web tool using Data Interoperability called within ModelBuilder, so without ArcPy code. So with some ArcPy and using the ArcGIS Python API in a Notebook we can put it all together.  Let's get started! My source dataset to integrate is the city and county of San Francisco Enterprise Addressing System Addresses with Units.  The data changes daily and I want a portal locator refreshed each day before working hours.  Here is my Data Interoperability workspace authored in Pro and which does the job interactively: Refresh Portal Locator The ETL tool reads a CSV file at a URL provided by Socrata, writes the features to a scratch geodatabase, performs change detection between the latest data and its previous state (maintained in a portal feature service) then writes the changes both to the feature service and an Edits feature class in the scratch geodatabase.  It also emails me a handy summary of new or changed addresses.  The tool is in the blog download and requires Pro 3.0+ and Data Interoperability. Notes on the scratch environment: Since publishing this article I have noticed my web tool occasionally failing at run time due to the scratch geodatabase becoming corrupt, the fix was to refactor how the scratch environment is used, with a unique name generated at run time.  It is not possible to set the file geodatabase writer to automatically overwrite the target geodatabase because it is locked at run time. But how do changes get pushed to the portal locator by the ETL tool, you ask?  By using a shutdown script that calls ArcPy in the geoprocessing scratch environment!  Data Interoperability ETL tools are geoprocessing tools and the scratch geodatabase in this case is found by this scripted parameter:   import arcpy scratchGDB = arcpy.CreateScratchName("xx",".gdb","Workspace",arcpy.env.scratchFolder) arcpy.AddMessage('Using scratch GDB {}'.format(scratchGDB)) return scratchGDB   The scratch folder is found similarly:   import arcpy scratchFolder = str(arcpy.env.scratchFolder) arcpy.AddMessage('Using scratch folder {}'.format(scratchFolder)) return scratchFolder   When shared as a web tool the log shows these messages: Using scratch GDB C:\Users\arcgis\AppData\Local\Temp\scratch\xx0.gdb Using scratch folder C:\Users\arcgis\AppData\Local\Temp\scratch Now the powerful bit, using ArcPy in a shutdown script to recreate the locator service.  The locator creation code originated as a copy from a history item after a Create Locator tool run, the rest just from reading the help.   # If there are edits then recreate the portal locator import arcpy import datetime import fme import os import pytz scratchGDB = fme.macroValues['ScratchGDB'] scratchFolder = fme.macroValues['ScratchFolder'] portalURL = fme.macroValues['PortalURL'] serverURL = fme.macroValues['ServerURL'] portalUser = fme.macroValues['PortalUser'] portalPassword = fme.macroValues['PortalPassword'] arcpy.env.workspace = scratchGDB arcpy.env.overwriteOutput = True if arcpy.Exists('Edits') and arcpy.Exists('AddressesWithUnits'): # Create or recreate the locator if there are data changes arcpy.geocoding.CreateLocator("USA", r"{}\AddressesWithUnits PointAddress".format(scratchGDB), "'PointAddress.HOUSE_NUMBER AddressesWithUnits.Full_Address_Number';"+ "'PointAddress.STREET_NAME AddressesWithUnits.Street_Name';"+ "'PointAddress.STREET_SUFFIX_TYPE AddressesWithUnits.Street_Type';"+ "'PointAddress.SUB_ADDRESS_UNIT AddressesWithUnits.Unit_Number';"+ "'PointAddress.NEIGHBORHOOD AddressesWithUnits.Neighborhood';"+ "'PointAddress.CITY AddressesWithUnits.City';"+ "'PointAddress.SUBREGION AddressesWithUnits.County';"+ "'PointAddress.REGION AddressesWithUnits.Region';"+ "'PointAddress.POSTAL AddressesWithUnits.ZIP_Code';"+ "'PointAddress.COUNTRY AddressesWithUnits.Country';", r"{}\SanFrancisco".format(scratchFolder), "ENG",None,None,None,"GLOBAL_HIGH") # Create the SD draft and SD locator_path = os.path.join(scratchFolder,'SanFrancisco') sddraft_file = os.path.join(scratchFolder,'SanFrancisco.sddraft') sd_file = os.path.join(scratchFolder,'SanFrancisco.sd') service_name = 'SanFrancisco' pst = pytz.timezone('US/Pacific') sfNow = datetime.datetime.now(pst) sfNowStr = sfNow.strftime('%Y:%m:%d %H:%M:%S') summary = 'Point Address With Units locator for the City of San Francisco updated at {} PST'.format(sfNowStr) summary += '\nBuilt from the Socrata data source: https://data.sfgov.org/Geographic-Locations-and-Boundaries/Addresses-with-Units-Enterprise-Addressing-System/ramy-di5m' tags = 'San Francisco,EAS' analyze_messages = arcpy.CreateGeocodeSDDraft(locator_path, sddraft_file, service_name, copy_data_to_server = True, summary = summary, tags = tags, max_result_size=50, max_batch_size=1000, suggested_batch_size=150, overwrite_existing_service=True) # Upload the locator if analyze_messages['errors'] == {}: arcpy.SignInToPortal(portalURL,portalUser,portalPassword) arcpy.server.StageService(sddraft_file,sd_file) arcpy.server.UploadServiceDefinition(sd_file,serverURL) # Clean up the scratch GDB try: arcpy.management.Delete(scratchGDB) except: pass   So that's the ETL tool that I published as a web tool.  Before moving on to how we orchestrate the web tool I have a couple of tips on authoring and publication. The ETL tool is parameterless as far as the geoprocessing environment is concerned, because I set all its inputs to not be published.  This works around a behavior with an input CSV file that is at a URL:  ArcGIS Enterprise cannot register a URL as a data store or copy the data to the server at run time so we just hide it by not publishing the parameter. The ETL tool uses a couple of web connections to access the portal and authenticate to an email service.  You get these onto the server by copying them out as XML files from the Tools>Options>Web Connections menu, copying them to the server, logging onto the server as the arcgis service owner, starting Workbench from fmeworkbench.exe and importing them from the same interface.  Make sure to re-authenticate each connection after importing them. Because we're using the scratch environment on the server, which is shared between processes (this behavior may change), the web tool's geoprocessing service is set to run in 1 instance maximum, this prevents us from accidentally getting contention between instances writing to the scratch file geodatabase (multiple jobs will queue).  If you need to publish multiple web tools that write into the server environment then refactor them to use temporary workspaces. At this point we have a web tool that does what we want, now we need to orchestrate it on a schedule.  For this we use a hosted Notebook in ArcGIS Online.  This is the first notebook I ever authored (my no-code background shows) so if I can do it you can too.  The web tool runs on my portal but my notebook runs in Online.  That is no problem for the notebook, the web tool isn't executing in Online, under the covers the notebook is just authenticating to the portal and sending a submitJob REST request.  Online notebooks know how to connect to portal GIS instances.  Here is the single-cell notebook (Standard runtime): Online hosted Notebook  How hard is that?  Crazy simple.  The hardest part was getting a certificate from the IT folks that the portal would accept when hearing from ArcGIS Online (my colleague Renato S. gets the credit for that).  All I did was read the help and replace help examples with my paths and names.  To find the function name to use after importing the web toolbox I just put the code into IDLE so intellisense could give me the methods. Then while editing the notebook I went to the Tasks editor and set up my notebook to run weekdays at 6am.  Note that the task editor time picker uses local time values from your browser client. Now I can let the system(s) run and each day my organization has an up-to-date locator maintained in a portal! Locator item details plus today's address changes email     ... View more

The Big Picture   The ArcGIS Data Interoperability team at Esri get this question a lot, especially at events like the Esri International User Conference.  This article will equip you with some background and details we find most helpful in purchasing and usage contexts. ArcGIS Data Interoperability In the above screen capture you can see Data Interoperability in a few places; the Analysis ribbon provides access to the Workbench, Data Inspector and Quick Translator apps (also available in the ArcGIS program group start menu), the project has custom Spatial ETL tools defined in a project toolbox, and ModelBuilder is calling a Spatial ETL tool.  Not shown - the Quick Import and Quick Export system tools delivered with the extension in the Data Interoperability Tools system toolbox - these use default settings to translate between any non-raster format and file geodatabase (or other format already understood by ArcGIS) in either direction - very convenient for quick inspection or quick sharing of data. Now, some details. Firstly, ArcGIS Data interoperability is built using FME technology.  Originally, Data Interoperability was conceived as a way for ArcGIS users to take advantage of Safe Software's industry-leading format support, and FME's no-code approach to handling geospatial data at row level granularity, plus of course the transformational ability of FME technology when working with attributes and geometry.  The technology was originally exposed as browsable connections to data and as geoprocessing tools in ArcMap and geoprocessing services in ArcGIS Server. None of that has gone away, but much new has been added, and how it is exposed has changed.  If the ancient Greeks had invented computers and studied ETL as a discipline, Heraclitus might have said something like "You cannot step twice into the same data flow". Let's cover format support.  Data Interoperability is built from FME and therefore uses an FME 'engine' of a particular release, just like the FME product itself.  We build Data Interoperability with the latest engine we can but  FME has a different release cycle and can get ahead.  If you're interested in what the differences between engines are you can go to this page and click the link for the FME Form change log.  How all this relates to formats is that format support is bound to either an engine or an FME Hub package version, and FME packages themselves have minimum engine (and build) requirements. Data Interoperability gets every format we can obtain, so is comparable to FME Form; here are all the formats that are in FME Form but not in Data Interoperability for ArcGIS Pro 3.7. The Format Delta   FME Formats Not in Data Interoperability @ Pro 3.7 APT Aircom ENTERPRISE Map Data/ASSET Data Autodesk AutoCAD Map 3D Object Data Autodesk AutoCAD RealDWG DWG/DXF Autodesk Navisworks Autodesk ReCap Bentley Map XFM Design (V8) Bentley Microstation Design Bentley Pointools POD Bentley i-model Interchange Format CARIS NTX CARIS Spatial Archive (CSAR) Point Cloud CARIS Spatial Archive (CSAR) Raster DES EDIGéO ER Mapper ECW FalconView File GeoConcept Map Graphic Technologies, Inc. (GTI) GTViewer Intergraph FRAMME Standard Exchange Format (SEF) HDF5 Raster JPEG 2000 (GeoJP2/GMLJP2)* Marconi PlaNet Metria AutoKa Transfer File (FF) Microsoft MapPoint Web XML Northrop Grumman C2PC Magic (Tech Preview) Precisely MapInfo Extended TAB* Precisely MapInfo TAB (EFAL- Tech Preview)* Precisely MapInfo TAB (MAPINFO)* Precisely Multi-Resolution Raster (MRR) (GDAL) Smallworld 4/5 Tableau Hyper Tele Atlas MultiNet Interchange format VoxelGeo OpenInventor (VOIV)   * Note that JPEG2000 data is supported with the OpenJPEG2000 reader/writer and MapInfo TAB is supported with the MITAB reader/writer. Let's cover transformer support.  Below is a list of all transformers in FME Form 2026.1 that are not in Data Interoperability at Pro 3.7.  They are 3rd party transformers available for purchase and use in FME - and hence out of scope for Data Interoperability. The Transformer Delta   Transformers in FME 2026.1 not in Data Interoperability @ Pro 3.7 Curvefitter GtransAttributeReprojector GtransReprojector ReframeReprojector   Other Considerations   Why would you buy FME or Data Interoperability over the other?  If the above formats or transformers are critical to your work then you will need to buy FME.  Otherwise there are other criteria you may balance. ArcGIS Data Interoperability is sold and supported by Esri.  FME is sold through Safe Software's channel. Data Interoperability extends ArcGIS Pro and ArcGIS Enterprise; in the case of Pro in the Analysis ribbon and as geoprocessing tools, in the case of Enterprise as geoprocessing services or web tools (although you can use the Workbench application on an Enterprise machine to do things like import credentials, or even process data). Notes on FME Form   It is intended that FME Form and Data Interoperability can be used 'together' in terms of sharing Workbench workspace files but if the products are at different FME engine releases incompatibilities, while rare, can occur and are not bugs.  For example, transformers may fail to 'arrive' in Data Interoperability when opening a workspace authored with FME at a later release because it has a newer version of a transformer.  In this situation re-insert the missing transformer(s) using Data Interoperability with parameter values taken from the FME original workspace.  If this is a common occurrence at your office it can be useful to keep a workspace with any old releases of transformers on your FME machine to copy/paste into workspaces destined for Data Interoperability. FME Form uses ArcGIS Pro software for Esri binary format support such as geodatabase (enterprise, file, mobile) and Knowledge Graph.  An exception is the Esri Geodatabase (OpenFile Geodb) reader/writer that does not need Pro.  Pro can be licensed in any of the usual modes.  Enable offline use if you will be using standalone Python scripts that call an ETL tool. Notes on FME Flow   FME Flow must use ArcGIS Enterprise software for any Esri binary format support such as file, mobile or enterprise geodatabase, with the above Esri Geodatabase (OpenFile Geodb) format exception, or Knowledge Graph.  ArcGIS Server Basic is sufficient for enterprise, file and mobile geodatabase support.  ArcGIS Enterprise must be licensed for all cores on machines running FME engines that call ArcGIS executables, but you may virtualize the cores on a machine to tune the core count to your actual requirements, see pp 4 & 5 in this document..  ArcGIS Pro is licensed as a single-user app and may not be used on an FME Flow machine to enable Esri formats.  However, FME Flow can write to an enterprise geodatabase without a local ArcGIS Enterprise installation if the the target feature class or table is published by reference from a registered data store and FME Flow has the Esri ArcGIS Connector package installed and uses an  Enterprise feature service reader/writer.  This approach uses REST calls and not any Esri executable.  Please see Safe's article here, specifically the topic "Connecting to Esri Applications from FME Flow without a license." Installing ArcGIS Pro and ArcGIS Data Interoperability on the same machine as ArcGIS Server requires version compatibility, please see this article. Can you trade in Data Interoperability for a copy of FME or vice versa?  There is no formal program for any sort of cross grade.  Esri and Safe have tried to make the products interoperable and many sites have both products. Does Data Interoperability have any FME Flow capability and does FME have any ArcGIS Enterprise capability?  Data Interoperability can be used to publish workspaces to FME Flow.  There is a dependency that a standalone workspace (fmw file) be used and that the workspace cannot have been used as the source for an ArcGIS Pro ETL tool.  Support for the workflow is from Safe Software.  FME has no ability to publish web tools to ArcGIS Enterprise but Data Interoperability can embed an FME workspace file (subject to fixing any incompatibility issues) into an ETL tool and publish it as a web tool. Does Data Interoperability have the AI Assist features of FME Form?  FME Form 2025.1+ includes AI Assist, which supports general workflow help, including using workspace context, and within various dialogs that help with generating Python code, regular expressions and SQL statements.  This feature requires the user be logged in to Safe Software's FME Account identity platform.  See also this article.  As ArcGIS users may not have FME accounts, the AI Assist features in FME Form are not currently exposed in ArcGIS Data Interoperability.  Please note however that the various AI transformers available in FME Hub, such as AmazonBedrockConnector, OpenAIConnector, OllamaConnector etc. do work in Data Interoperability. Does Data Interoperability support language localization like FME Form?  Yes.  At the current release of ArcGIS Pro 3.7 the compatible language packs are available from this page.  Follow the instructions for FME Form 2026.1. We hope the above gives you confidence that Esri and Safe have anticipated customer use of our products in a cooperative manner and with the minimum of friction.  If you have further technical questions please post them in this forum or for business-related questions though your Esri account representative. To obtain a free trial of Data Interoperability extension work with your Esri account representative to obtain a copy of the software install.   A license for Data Interoperability is included in ArcGIS Pro trials which you can access at this page. ... View more

Hi Everybody If you made it to the 2022 Esri International user Conference and attended any of the Data Interoperability focused sessions then you may remember some of the material but perhaps not all, and I didn't share the workspaces and data at the time, so here they are! https://pm.maps.arcgis.com/home/item.html?id=1ce84fa8ef06456fb77016e4ac77f69a For example the earthquake feed map: Data Interoperability In Action   The package includes attachments, note the Word doc (DemoDescriptions.docx) with some brief notes on each demo, the workspaces files (*.fmw) used and some sample CSV, GeoJSON and zipped Shapefile data used  in one demo. You'll need Pro 3.0 and Data Interoperability extension to use the package. See you next year! ... View more