ARC.WRITE converts NAs to -1

[AndrewGilbert1]

Thanks for putting together the arcgisbinding package, it is very useful. 

It seems that when converting a spatial dataframe to a file geodatabase, NAs in R are converted to -1 in the file geodatabase. This is a problem on many different levels as -1 may represent a real value that can't be removed if real -1 values are also present. ArcGIS does not appear to use NAs, but they do use the Null value as equivalent. While these are slightly different things in R, it would be much better to convert NAs to real Nulls in the file geodatabase than a -1 which is problematic at best. Can this be fixed in an update to this package?

Thanks

Andrew

[ShaunWalbridge]

Hello Andrew,

Glad you've found the package useful in your work. Could you provide a little detail on the workflow you're using, and what data types you've hit this issue with? I just tried some quick tests with my own data, and can successfully round-trip ArcGIS <Null> values with R (where they are represented as NA -- agree this isn't a perfect replacement for what NA means in the R context). I tried this with table, File Geodatabase, and Shapefile outputs and could get round tripping to work fine there. I tested string fields and numeric fields.

Cheers,

Shaun

[FrancescoTonini2]

Shaun, I actually had a similar but slightly different experience. In my case, the I had an R SpatialPolygonDataFrame with some NAs in R and when I used arc.write() it converted those values to 0 when saved to a shapefile and opened in ArcGIS. If you want I can send you the R code and data to replicate it...

[ShaunWalbridge]

Hi Franceco,

Sure, an example script would be great. Historically, the dBASE format didn't support NULL values, so it may be defaulting to writing these out as 0 for backward compatibility reasons (and a single space for string fields).  Modern software does support NULL values in DBFs, and we can look at how we write the values to include these.

[ShaunWalbridge]

OK, I did a little more digging. Basically, some software like OGR will write NULL values into a DBF. It does this by writing out 0x2a in the place where a value would normally exist, which indicates the particular record is "deleted". OGR, and software based on OGR do support this, but this is a deviation from the Shapefile whitepaper, and as such ArcGIS doesn't directly support this, independently of using the bridge. NULL values work fine to Geodatabase outputs.

[AndrewGilbert1]

Hi Shaun

I'm not in the office today but I'm trying to write a spatialpointdataframe from R to a point featureclass in a File Geodatabase. I have not tried pulling data the other way. I think both text and integer fields in R are being translated to -1 in the gdb. I need to confirm the integer issue when I get back to my desk. Thanks for the response on this. I've been desperate for a good way to write out to File geodatabase from R and just discovered this.

Cheers

Andrew

Andrew Gilbert

Biodiversity Research Institute

276 Canco Rd.

Portland, ME 04103

207-839-7600 X205

Sent from my iPhone

[AndrewGilbert1]

Shaun,

I double checked what was going on and it's only fields that are listed

as logical. I have a number of fields that default to logical in R when

I import them from a CSV file. These fields have no data for a

particular dataset so it creates them as logical fields which if NA

convert to -1 in arc.write. Checking numeric fields these do as you say

convert to successfully. So I was not correct. I can of course

make sure there are no logical fields, except that if you have a true

logical field then -1 may not make sense.

I did discover another issue I can't seem to determine a good work

around to if you have happen to have a suggestion. When I try to write a

character field with more than 255 characters to the File Geodatabase

featureclass it fails as it seems to default to field length 255 for a

text field in Arc, despite the fact that they can be much larger. Is

there a way to either default to a much larger size or specify this in

arc.write? The only workaround is to truncate or split among multiple

fields which isn't desirable.

Thanks for your help.

Cheers, Andrew Gilbert

[ShaunWalbridge]

Andrew,

I can confirm the issue when working with Logical values. None of the ArcGIS Field Types directly map onto a boolean, so by default its mapping these to values it knows how to encode. I've created a new issue for this here:

 Support mapping logical values into the closest supported type · Issue #13 · R-ArcGIS/r-bridge · Git... 

I also can confirm that writing long text values into a new output file doesn't work. I've filed a second issue for that here:

Support writing text fields which are longer than 255 characters. · Issue #14 · R-ArcGIS/r-bridge · ... 

Both of these are things we intend to fix, I'll update this thread and let you know when we've made a release which fixes these issues.

Cheers,

Shaun

[AndrewGilbert1]

That's great Shaun. It's a really necessary piece of software so I'm

glad you're actively supporting it.

I hate to do it to you, but I've discovered two further issues

surrounding date fields. When you use a date type in R, which I can

successfully transfer dates to date type from POSIXct, but not POSIXlt,

which causes an error. A simple translation to POSIXct prior to writing

should fix this.

Also, ArcGIS does not seem to honor time zone info, so I actually have

to trick Arc when local times are given with time zones by telling it's

a GMT time, otherwise it appears to strip the time zone info and present

the GMT time only even if it's New York time or whatever. Of course you

can convert to text and export, but it then will not be a date field.

Sorry to bring you more issues, I think these two probably will be minor

issues to deal with, but can be problematic if folks don't realize this

is happening.

Thanks again, Cheers, Andrew