topic Handling blank values not converting string to float with NumPy in Python Questions

Handling blank values not converting string to float with NumPy

GeoffreyWest1 — Sat, 11 Dec 2021 21:24:47 GMT

A web-service is returning blank values in lat/long causing ValueError: could not convert string to float:

Output:

34.0746400246
-118.37200592


34.0968797927
-118.449631186
34.0635760772
-118.327509298
34.0635760772
-118.327509298
34.0746400246
-118.37200592
34.0828880733
-118.320862797
34.0828880733

Code:

import requests
import json
import jsonpickle

f2 =open('C:\Users\GeoffreyWest\Desktop\Request.json')
data2 = jsonpickle.decode((f2.read()))
data2 = jsonpickle.encode(data2)
fc= (aFC)



print data2
url2 = "url"
headers2 = {'Content-type': 'text/plain', 'Accept': '/'}

r2 = requests.post(url2, data=data2, headers=headers2)
decoded2 = json.loads(r2.text)
print json.dumps(decoded2, sort_keys=True, indent=4)
if arcpy.Exists(fc):
  arcpy.Delete_management(fc)
try:
  r2
except requests.exceptions.ConnectTimeout as e:
   print "Too slow Mojo!"

items = []
for sr in decoded2['Response']['ListOfServiceRequest']['ServiceRequest']:
   SRAddress = sr['SRAddress']
  lat =  "3"
  y = sr['Latitude']
  x = sr['Longitude']
  str_list = filter(None, y)
  str_list2 = filter(None, x)
   print str_list
   print str_list2



  dt = np.dtype([('LatitudeShape', '<f8'),
   ('LongitudeShape', '<f8'),
   ('Latitude', '<f8'),
   ('Longitude', '<f8'),
])

  items.append((str_list2,
  str_list,
  str_list2,
  str_list,
   ))

  sr = arcpy.SpatialReference(4326)

arr = np.array(items,dtype=dt)

NumPyArray = arcpy.da.NumPyArrayToFeatureClass(arr, fc, ['longitudeshape', 'latitudeshape'], sr)
print "success"

Re: Handling blank values not converting string to float with NumPy

JoshuaBixby — Wed, 13 May 2015 15:53:03 GMT

Can you paste the specific error message, the actual text returned? It would be helpful to know what function or method is actually raising the error.

Re: Handling blank values not converting string to float with NumPy

JoshuaBixby — Wed, 13 May 2015 18:14:49 GMT

After a quick check, I am guessing it is Line 53 of your code snippet that is throwing the error. I guess my first question is whether you want to rely on implicit type conversion? Python doesn't natively support implicit type conversion, and the Zen of Python states "explicit is better than implicit." Even though NumPy's array constructor supports implicit type conversion, I would argue relying on it is not very Pythonic.

What are you trying to achieve with Lines 32 & 33? If you explicitly convert strings to numbers and empty strings to None, I think the code will stop throwing errors at Line 53.

Re: Handling blank values not converting string to float with NumPy

DanPatterson_Retired — Sat, 11 Dec 2021 21:24:50 GMT

In order to use numpy arrays with null values, you have to implement masked arrays and not typical ndarrays. The documentation on masked arrays is quite good but there are some tricks particularly if your input data contains a mix of numbers and text, in which case, your dtype should be 'object'. A couple of illustrations follow:

>>> import numpy as np
>>> x = np.array([1, 2, 3, np.nan, 5, np.nan, 7,8, np.nan])
>>> y = np.ma.masked_array(x, np.isnan(x))
>>> y
masked_array(data = [1.0 2.0 3.0 -- 5.0 -- 7.0 8.0 --],
            mask = [False False False  True False  True False False  True],
      fill_value = 1e+20)
>>> u = np.array(['a', 2, 'c', "", 'e', "", 7, 'g', ""],dtype=object)
>>> u = np.ma.masked_values(u,"")
>>> u
masked_array(data = ['a' 2 'c' -- 'e' -- 7 'g' --],
            mask = [False False False  True False  True False False  True],
      fill_value = ?)
>>>

may be of help depending upon your data structure.

In the x,y example, the array 'null' values were represented as np.nan's this could be other numbers such as -9999 etc.

In the u,v example, the array is a mix of text and numbers and an empty string is used to represent null values.