topic Re: ArcGIS Pro Ignores Changes to CSV data in ArcGIS Pro Questions

ArcGIS Pro Ignores Changes to CSV data

MattAnderson2 — Wed, 17 Apr 2019 19:20:27 GMT

I added a csv datasource to my project, but after adding it I found two problems. The first is that a column filled with numeric data had a "Grand Total" entry in the bottom row, so ArcGIS Pro intrepreted the column as text instead. The same column also had a non-descriptive heading label. I removed the datasource from ArcGIS Pro, open the csv file in Excel, remove the grand total row, and change the column heading to something more descriptive, then save the file. I go back into ArcGIS Pro and re-add the datasource. To my frustration, ArcGIS Pro is still calling the column by its old label and still interpreting it as text. Why is this application so frustrating??? How can I get it to read the file without deleting everything and starting over with a new project?

Re: ArcGIS Pro Ignores Changes to CSV data

DanPatterson_Retired — Wed, 17 Apr 2019 23:55:07 GMT

You removed the file from the project, then saved the project?

Close and restart does nothing?

Try a simple rename of the *.csv, then re-add

Add the file to an empty project.

Any of those work?

Re: ArcGIS Pro Ignores Changes to CSV data

MattAnderson2 — Wed, 24 Apr 2019 20:25:49 GMT

Thanks for the reply Dan.

If I remove it from the project, save the project, then restart and reopen the project, it still reads the field as text.

So I went into Windows Explorer and changed the filename. I tried to re-add the file, but the ArcGIS Pro "Add Data" dialog still shows me the old file name. If I try to open that, I get an error ("The file doesn't exist"...no $%#&*, I just renamed it). I have to restart ArcGIS Pro for it to see the file with the new name. Unbelievable.

After I have restarted the application, reopened my project, and re-added the renamed file, the application FINALLY reinterprets the data, realizes that there is no longer a call that says "Grand Total" and correctly interprets the column as numeric. At this point I've only torn out half the hair on my head.

Just for kicks I try making a new project, but in the dialog to set the path for my new project, I cannot create a new folder. I have to go back to Windows Explorer to create a new folder for my project. That's why my company is paying thousands of dollars for an ArcGIS license....because we want to have to open a whole separate program to create a folder (*sarcasm*).

ESRI software makes me reconsider my career choice as an urban planner.

Re: ArcGIS Pro Ignores Changes to CSV data

KoryKramer — Wed, 24 Apr 2019 20:32:43 GMT

Matt, what version of Pro are you using?

In the current version, 2.3.2, it is possible to create a new folder when I'm pathing to where I want to create a new project:

Re: ArcGIS Pro Ignores Changes to CSV data

JoshuaFlickinger — Fri, 10 Jul 2020 23:59:17 GMT

Did anyone figure out what's happening here? I'm running Pro 2.5.1 and noticing some similarly strange behavior where ArcGIS Pro appears to be 'remembering' something from old csv files.

Basically I've got a jupyter notebook that generates a csv file from a pandas dataframe and then imports the csv into a geodatabase using the table to table tool. (As an aside, it would be nice if the arcgis API could directly export pandas dataframes to geodatabase tables so I wouldn't have to use this workaround to begin with.)

My notebook worked as expected the FIRST time. However, after tweaking my code and re-running, my notebook spits out a csv that has been re-arranged. In other words, the order of columns is not the same. When this newer version of the csv is added to a map in Pro, I can open it and see that it doesn't update correctly. Unfortunately it maintains some of the old csv file structure. But not all of it. It's a bit jumbled up. Certain fields that I added to my dataframe/csv have been 'overwritten' by the fields which were originally in that position in the csv. I check the csv file itself in Excel and am seeing exactly what I expect... so I know the csv generation part of the process is fine.

Of course then the subsequent table to table tool produces wonky results as well, as shown below, where those 'overwritten' fields are trying to read into the field map twice.

I'm a bit confused as to why this should happen, as my notebook totally erases the geodatabase workspace and reconstructs it each time I run the notebook. The problem DOES resolve itself if I close everything down and rename the csv and re-add it... but this still seems like a bug and could be a big "gotcha" for people. Any feedback on why it's happening?

Re: ArcGIS Pro Ignores Changes to CSV data

ValerieZimmer — Thu, 18 Feb 2021 03:23:17 GMT

Me too, and this happens in ArcMap and ArcGIS Pro, for both .csv and for .xls files. Seriously, ESRI, why is this a thing? A year later, and why is there still no solution or response from ESRI on this?

Re: ArcGIS Pro Ignores Changes to CSV data

DeanChiang — Thu, 29 Jun 2023 03:22:20 GMT

It is June 2023 and this is still happening (v2.9.2). Remove the CSV file from map. Save, close, and open APRX. Click Refresh on Project, Folders, the data source folder. Dozens of times. Still the old *ghost* data field names persist in the latest CSV file (view by Open or under Fields in Table To Table conversion tool). If it was just the field name, I can change that after converting to file geodatabase table, but the data type is interpreted wrong too for me not to be able to import the data.

How is it a native Windows Desktop software program cannot read a text file on the system dynamically correctly? Boggles the mind. Not to mention hours of time wasted.

It's a question for ChatGPT4 - but I don't have time to do fancy GIS stuff. Cannot even import a text file into ArcGIS *Pro*.

Re: ArcGIS Pro Ignores Changes to CSV data

DeanChiang — Thu, 29 Jun 2023 17:04:41 GMT

A workaround is to rename the CSV file differently every time you refresh the source CSV or change its data structure, but that is no way to build a reliable, repeatable workflow or automate a task.

Here is a potential solution (need others to verify):

Trying to figure out why/where ArcGIS Pro is persisting data format for these external files, I noticed there was a file called schema.ini in the source folder for the CSV files that I tried to load into Pro. I deleted that file (which I did not create). Open the APRX project again - Voila! The CSV file now loads with the correct field name and type, and converted to FGDB table correctly.

I cannot believe it took me this long to figure this out, or that Esri does not have this documented somewhere.

Re: ArcGIS Pro Ignores Changes to CSV data

KellyDunn — Wed, 02 Aug 2023 15:17:49 GMT

A better question is why are all these steps necessary, even if they work? Why can't saving tne csv and refreshing cause it to update? Would love if this could be improved.

Re: ArcGIS Pro Ignores Changes to CSV data

Nuria — Fri, 04 Aug 2023 09:28:24 GMT

It has also happened to me using ArcGIS Pro v3.1.

I have 2 csv files I generated with Jupyter Notebooks. Basically the two files are derived from the same dataframe and when I loaded them to ArcGIS pro, to plot the lat/lon locations, one of them show 2 new columns which are not in the original csv; i.e. if I open the csv file in excel these two columns are not there however, when I open the same csv file from the Contents in my project, the two columns are there.

I have rerun the Jupyter Notebook again to generate the code and still the same behaviour. I have tried by creating new ArcGIS project in various network locations and the problem persist. I have also rename the file, copied and rename it; it has not work.

My working folder is Google Drive so I firstly thought it had to do with Google Drive versioning control or something related but after trying in a in a network drive (creating a new ArcGIS project) and getting the same behaviour I am out of ideas.

How is it possible that ArcGIS pro keeps adding (?) these two new columns to the file after starting afresh. And it is always on the same one of the two I am working with. I would like to understand what's going on here.

Any thoughts on this?

Thank you

Re: ArcGIS Pro Ignores Changes to CSV data

DeanChiang — Tue, 08 Aug 2023 19:44:00 GMT

Nuria,

Your experience seems even more pervasive than I have encountered. I would first try to find all files named schema.ini in all your folders and delete them if you find any. My experience has been that ArcGIS Pro can add it to any folder it has permission, including the main AGP folder parallel to the APRX file, under any subfolders, under Users, separate data folders on another drive, even under OneDrive (in the cloud). The schema.ini is a plain text file you can open in any text editor.

Re: ArcGIS Pro Ignores Changes to CSV data

Ben_U — Thu, 07 Sep 2023 02:51:00 GMT

This situation is just infuriating.

I added CSV file into Arc Pro and noticed it had a "-" in the Title ID field that I needed to remove.

I removed the file from Arc Pro, did a find and replace in the CSV file to remove "-" symbols in the title ID field, and re added the CSV file to my project.

It was still showing the dash in the Title ID field.

OK no problem, I found this thread, followed all of the suggested work around's.

I have deleted the schema.ini file a dozen times.

I have renamed the CSV file.

I have deleted the Arc Pro cache.

I have saved the CSV in a new folder location.

I have re started Arc Pro more than a dozen times.

I have removed all Database connections in the project file and re established them.

Every single time I load the CSV into Arc Pro it still has the "-" symbol in the title ID numbers. Even though with a new name, in a new folder location, the CSV file does not have any "-" symbols in the Title ID field.

Arc Pro WILL NOT recognise the change I have made to the Title ID field in the CSV file. The only solution I have found so far is to open it in QGIS and it instantly works perfectly. This is a major issue and it needs fixing ASAP.

Re: ArcGIS Pro Ignores Changes to CSV data

JonasJansson — Thu, 16 Nov 2023 14:17:21 GMT

I'm running Pro 3.1 and I've encountered the same problem as everyone else in this thread. Even tough I resave changes to a csv reload it to ArcGIS it merges some of the columns taking the updated data but placing it under an old heading. Despicable that this is still a bug so many years later.

Re: ArcGIS Pro Ignores Changes to CSV data

DanyaLittlefield — Fri, 05 Jan 2024 22:50:17 GMT

I'm also having this issue, and I'm running ArcGIS Pro 3.2.1

I was repeatedly exporting a csv from R, refreshing and reloading it into ArcGIS, but a few of my columns were continually not appearing correctly in ArcGIS Pro (column names not loading, data showing up as Null) despite my efforts to correct them at the source in R. After reading this thread, I opened the csv files I had been trying to load and they looked totally fine in Excel. I renamed the corrected csv, restarted ArcGIS, and the problem was solved.

I wasted a ton of time on this today thinking that I had an error in my code, when in fact it was a problem with ArcGIS! ESRI - please address this.

Re: ArcGIS Pro Ignores Changes to CSV data

EricaNova — Tue, 19 Mar 2024 15:24:00 GMT

Thanks @DeanChiang , that worked for me.

Re: ArcGIS Pro Ignores Changes to CSV data

JeffAdams — Mon, 22 Apr 2024 15:52:09 GMT

Yes?! So much wasted time trying to figure out what I am doing wrong. For crying out loud, fix this ESRI?!

Re: ArcGIS Pro Ignores Changes to CSV data

FU_ESRI — Wed, 10 Jul 2024 00:12:38 GMT

This issue has been going on for over a decade. It happens with dbf tables and csv. I complained to ESRI about this in 2012 trying to add dbf tables, and 14 years later they haven't fixed it.

Re: ArcGIS Pro Ignores Changes to CSV data

PeterSchultz — Mon, 10 Feb 2025 21:51:28 GMT

I heartily agree, although the issue is possibly a converse to the threads in here - I download a csv file with a field containing mainly numeric data interspersed with some records containing text e.g. 8B, 5A and the like. Attempts to import this file invariably lead to ArcPro designating this field as numeric and any text values in this field are ignored and replaced with a null value. The schema.ini file reads:

[property_01_02_2025.csv]
Col7=BLDG_SUBUNIT_NO Text
If schema.ini were to work or be recognised, Column 7 would correctly report as text.
The situation was worse in ArcGis Desktop - the conversion attempt in ArcCatalog would simply fall over due to a data type mismatch, the ESRI program having first deduced Field 7 as Numeric (Long) then crashing when a text value was encountered. In my experience I agree, this has been going on for over a decade. Enough of the nice new models - time to fix the nuts and bolts.

Re: ArcGIS Pro Ignores Changes to CSV data

PeterSchultz — Mon, 10 Feb 2025 22:01:02 GMT

FME has no issues with it either.

Re: ArcGIS Pro Ignores Changes to CSV data

PeterSchultz — Mon, 10 Feb 2025 23:03:43 GMT

This seems to be an acceptable workaround (as if a quality product should need a workaround). Having opened the "offending" csv file in Excel, Format the "problem" to the required data type and Save As csv once again. That seemed to fix the problem.