Handling blank values not converting string to float with NumPy

Question

A web-service is returning blank values in lat/long causing ValueError: could not convert string to float: Output: 34.0746400246 -118.37200592 34.0968797927 -118.449631186 34.0635760772 -118.327509298 34.0635760772 -118.327509298 34.0746400246 -118.37200592 34.0828880733 -118.320862797 34.0828880733 Code: import requests import json import jsonpickle f2 =open('C:\Users\GeoffreyWest\Desktop\Request.json') data2 = jsonpickle.decode((f2.read())) data2 = jsonpickle.encode(data2) fc= (aFC) print data2 url2 = "url" headers2 = {'Content-type': 'text/plain', 'Accept': '/'} r2 = requests.post(url2, data=data2, headers=headers2) decoded2 = json.loads(r2.text) print json.dumps(decoded2, sort_keys=True, indent=4) if arcpy.Exists(fc): arcpy.Delete_management(fc) try: r2 except requests.exceptions.ConnectTimeout as e: print "Too slow Mojo!" items = [] for sr in decoded2['Response']['ListOfServiceRequest']['ServiceRequest']: SRAddress = sr['SRAddress'] lat = "3" y = sr['Latitude'] x = sr['Longitude'] str_list = filter(None, y) str_list2 = filter(None, x) print str_list print str_list2 dt = np.dtype([('LatitudeShape', '), ('LongitudeShape', '), ('Latitude', '), ('Longitude', '), ]) items.append((str_list2, str_list, str_list2, str_list, )) sr = arcpy.SpatialReference(4326) arr = np.array(items,dtype=dt) NumPyArray = arcpy.da.NumPyArrayToFeatureClass(arr, fc, ['longitudeshape', 'latitudeshape'], sr) print "success"

DanPatterson_Retired · Answer

In order to use numpy arrays with null values, you have to implement masked arrays and not typical ndarrays.  The documentation on masked arrays is quite good but there are some tricks particularly if your input data contains a mix of numbers and text, in which case, your dtype should be 'object'.  A couple of illustrations follow:

>>> import numpy as np
>>> x = np.array([1, 2, 3, np.nan, 5, np.nan, 7,8, np.nan])
>>> y = np.ma.masked_array(x, np.isnan(x))
>>> y
masked_array(data = [1.0 2.0 3.0 -- 5.0 -- 7.0 8.0 --],
            mask = [False False False  True False  True False False  True],
      fill_value = 1e+20)
>>> u = np.array(['a', 2, 'c', "", 'e', "", 7, 'g', ""],dtype=object)
>>> u = np.ma.masked_values(u,"")
>>> u
masked_array(data = ['a' 2 'c' -- 'e' -- 7 'g' --],
            mask = [False False False  True False  True False False  True],
      fill_value = ?)
>>>

may be of help depending upon your data structure.

In the x,y example, the array 'null' values were represented as np.nan's this could be other numbers such as -9999 etc.

In the u,v example, the array is a mix of text and numbers and an empty string is used to represent null values.

bixb0012 · Answer

After a quick check, I am guessing it is Line 53 of your code snippet that is throwing the error.  I guess my first question is whether you want to rely on implicit type conversion?  Python doesn't natively support implicit type conversion, and the Zen of Python states "explicit is better than implicit."  Even though NumPy's array constructor supports implicit type conversion, I would argue relying on it is not very Pythonic.

What are you trying to achieve with Lines 32 & 33?  If you explicitly convert strings to numbers and empty strings to None, I think the code will stop throwing errors at Line 53.

bixb0012 · Answer

Can you paste the specific error message, the actual text returned?  It would be helpful to know what function or method is actually raising the error.