Handling blank values not converting string to float with NumPy

[GeoffreyWest1]

A web-service is returning blank values in lat/long causing ValueError: could not convert string to float:

Output:

34.0746400246
-118.37200592


34.0968797927
-118.449631186
34.0635760772
-118.327509298
34.0635760772
-118.327509298
34.0746400246
-118.37200592
34.0828880733
-118.320862797
34.0828880733

Code:

import requests
import json
import jsonpickle

f2 =open('C:\Users\GeoffreyWest\Desktop\Request.json')
data2 = jsonpickle.decode((f2.read()))
data2 = jsonpickle.encode(data2)
fc= (aFC)



print data2
url2 = "url"
headers2 = {'Content-type': 'text/plain', 'Accept': '/'}

r2 = requests.post(url2, data=data2, headers=headers2)
decoded2 = json.loads(r2.text)
print json.dumps(decoded2, sort_keys=True, indent=4)
if arcpy.Exists(fc):
  arcpy.Delete_management(fc)
try:
  r2
except requests.exceptions.ConnectTimeout as e:
   print "Too slow Mojo!"

items = []
for sr in decoded2['Response']['ListOfServiceRequest']['ServiceRequest']:
   SRAddress = sr['SRAddress']
  lat =  "3"
  y = sr['Latitude']
  x = sr['Longitude']
  str_list = filter(None, y)
  str_list2 = filter(None, x)
   print str_list
   print str_list2



  dt = np.dtype([('LatitudeShape', '<f8'),
   ('LongitudeShape', '<f8'),
   ('Latitude', '<f8'),
   ('Longitude', '<f8'),
])

  items.append((str_list2,
  str_list,
  str_list2,
  str_list,
   ))

  sr = arcpy.SpatialReference(4326)

arr = np.array(items,dtype=dt)

NumPyArray = arcpy.da.NumPyArrayToFeatureClass(arr, fc, ['longitudeshape', 'latitudeshape'], sr)
print "success"

[JoshuaBixby]

Can you paste the specific error message, the actual text returned?  It would be helpful to know what function or method is actually raising the error.

[JoshuaBixby]

After a quick check, I am guessing it is Line 53 of your code snippet that is throwing the error.  I guess my first question is whether you want to rely on implicit type conversion?  Python doesn't natively support implicit type conversion, and the Zen of Python states "explicit is better than implicit."  Even though NumPy's array constructor supports implicit type conversion, I would argue relying on it is not very Pythonic.

What are you trying to achieve with Lines 32 & 33?  If you explicitly convert strings to numbers and empty strings to None, I think the code will stop throwing errors at Line 53.

[DanPatterson_Retired]

In order to use numpy arrays with null values, you have to implement masked arrays and not typical ndarrays.  The documentation on masked arrays is quite good but there are some tricks particularly if your input data contains a mix of numbers and text, in which case, your dtype should be 'object'.  A couple of illustrations follow:

>>> import numpy as np
>>> x = np.array([1, 2, 3, np.nan, 5, np.nan, 7,8, np.nan])
>>> y = np.ma.masked_array(x, np.isnan(x))
>>> y
masked_array(data = [1.0 2.0 3.0 -- 5.0 -- 7.0 8.0 --],
            mask = [False False False  True False  True False False  True],
      fill_value = 1e+20)
>>> u = np.array(['a', 2, 'c', "", 'e', "", 7, 'g', ""],dtype=object)
>>> u = np.ma.masked_values(u,"")
>>> u
masked_array(data = ['a' 2 'c' -- 'e' -- 7 'g' --],
            mask = [False False False  True False  True False False  True],
      fill_value = ?)
>>>

may be of help depending upon your data structure.

In the x,y example, the array 'null' values were represented as np.nan's this could be other numbers such as -9999 etc.

In the u,v example, the array is a mix of text and numbers and an empty string is used to represent null values.