NumPy 'ValueError: size of tuple must match number of fields.'

[GeoffreyWest]

I have a script which parses an json web-service and uses parses the fields to add to a file geodatabase, towards the end of my script I receive the error. I receive the error

File "C:/Python27/ArcGIS10.2/websocket-client-0.23.0/websocket-client-0.23.0/examples/JSONSerializer.py", line 43, in     narr = numpy.array([vehicles],  ndtype) ValueError: size of tuple must match number of fields.

import json
import jsonpickle
import requests
import arcpy
import numpy

fc = "C:\MYLATesting.gdb\MYLA311"
if arcpy.Exists(fc):
  arcpy.Delete_management(fc)

f = open('C:\Users\Administrator\Desktop\myla311.json', 'r')

data = jsonpickle.encode( jsonpickle.decode(f.read()) )

url = "https://myla311test.lacity.org/myla311router/mylasrbe/1/QuerySR"
headers = {'Content-type': 'text/plain', 'Accept': '/'}

r = requests.post(url, data=data, headers=headers)
sr = arcpy.SpatialReference(4326)

decoded = json.loads(r.text)

SRAddress = decoded['Response']['ListOfServiceRequest']['ServiceRequest'][0]['SRAddress']
latitude = decoded['Response']['ListOfServiceRequest']['ServiceRequest'][0]['Latitude']
longitude = decoded['Response']['ListOfServiceRequest']['ServiceRequest'][0]['Longitude']

a = {"items":[{SRAddress, latitude, longitude}]}


ndtype = numpy.dtype([
  ('SRAddress', 'S16'),
   ('latitude', 'F48'),
   ('longitude', 'F48')

])

vehicles = []
for item in a['items']:
  vehicles.append(tuple(unicode(item for k in ndtype.names)))
#
narr = numpy.array([vehicles], ndtype)
arcpy.da.NumPyArrayToFeatureClass(narr, fc, ['longitude', 'latitude'], sr)


print a.keys()

# parsed_json = json.dumps(decoded, sort_keys=True, indent=4)

# print json.dumps(decoded, sort_keys=True, indent=4)

[DanPatterson_Retired]

Your dtype specifies 3 fields as being expected...the error indicates that the number of values being returned back doesn't match the dtype , I would insert a line and get the length the vehicles list...or perhaps remove the [ ] from around [vehicles] since you may be making a list of lists which would mean that it would get the first element which is a list and that list may contain 3 elements

[GeoffreyWest]

When I

print vehicles 

I receive

[(u'<', u'g', u'e', u'n', u'e', u'r', u'a', u't', u'o', u'r', u' ', u'o', u'b', u'j', u'e', u'c', u't', u' ', u'<', u'g', u'e', u'n', u'e', u'x', u'p', u'r', u'>', u' ', u'a', u't', u' ', u'0', u'x', u'0', u'D', u'5', u'A', u'3', u'D', u'0', u'0', u'>')] 

What does this indicate??

[DanPatterson_Retired]

If you carefully examine what it is printing out, you code has produced a python generator object and it is parsing that phrase letter by letter.  Let me show you be example with simple data structures:

>>> a = ['hello']

>>> b = []

>>> c = []

>>> for i in a[0]: c.append(i)

>>> c

['h', 'e', 'l', 'l', 'o']

Obviously not what expected, which I am sure you already know...  so somewhere in your code you aren't getting what you want and I am not sure where you are printing 'vehicles'

EDIT  as confirmed here  python - NumPy 'ValueError: size of tuple must match number of fields.' - Stack Overflow

I will let them finish it off

[GeoffreyWest]

vehicles = []

for item in a['items']:

vehicles.append(tuple(unicode(item for k in ndtype.names)))

print vehicles

[DanPatterson_Retired]

some examples using the StackExchange thread ideas

item={'1':b'one','2':b'two','3':b'three'}
print('\nDictionary output\n item 1: {} 2: {} 3: {}'.format(item['1'],item['2'],item['2']))
a =  [unicode(item) for k in item]  # list comprehension with unicode
print('\nList comprehension output\n{}'.format(a))
b = tuple(item.decode() for k in item)
print'\nWith the decoder ring\n{}'.format(b )
c =  unicode(item for k in item)
print'\nWithout the decoder ring\n{}'.format(c )
print'\nGenerator output then by output\n'
for i in c: print i

[GeoffreyWest]

Thanks Dan,

I have tested the block above. How would this be implemented in my code to

solve my error? I receive multiple keys from my items dictionary.

Dictionary output

item SRAddress: 1200 W TEMPLE ST, 90026 longitude: -118.252968 latitude:

34.064937

List comprehension output

With the decoder ring

(u'34.064937', u'1200 W TEMPLE ST, 90026', u'-118.252968')

items ={'SRAddress': SRAddress, 'Longitude': longitude, 'Latitude': latitude}

[DanPatterson_Retired · Accepted solution]

More experimentation...

import numpy as np     #NOTE THIS
keys = ['SRAddress','longitude','latitude']
k1,k2,k3 = keys
data_line = {'SRAddress': '1200 W TEMPLE ST 90026', 'longitude': -118.252968, 'latitude':34.064937}
frmt = '\nStraight dictionary output\n Address: {} Long: {} Lat: {}'
print(frmt.format(item[k1],item[k2],item[k3]))  
print '\noption 1:  List comprehension with unicode'
a =  tuple([unicode(item[key]) for key in keys])  # list comprehension with unicode  
print('{}'.format(a))  
dt = np.dtype([('SRAddress','U40'), ('longitude','<f8'),('latitude','<f8')])  
arr = np.array(a,dtype=dt)
print'\narray unicode\n',arr
print'dtype',arr.dtype
print '\noption 2:List comprehension without unicode'
b = tuple([item[key] for key in keys])
print('{}'.format(b))
dt = np.dtype([('SRAddress','S40'), ('longitude','<f8'),('latitude','<f8')])  
arr = np.array(b,dtype=dt)
print'\narray without unicode\n',arr
print'dtype',arr.dtype

yields the following...your choice...I don't have need for the Unicode stuff

>>> 
Straight dictionary output
 Address: 1200 W TEMPLE ST 90026 Long: -118.252968 Lat: 34.064937
option 1:  List comprehension with unicode
(u'1200 W TEMPLE ST 90026', u'-118.252968', u'34.064937')
array unicode
(u'1200 W TEMPLE ST 90026', -118.252968, 34.064937)
dtype [('SRAddress', '<U40'), ('longitude', '<f8'), ('latitude', '<f8')]
option 2:List comprehension without unicode
('1200 W TEMPLE ST 90026', -118.252968, 34.064937)
array without unicode
('1200 W TEMPLE ST 90026', -118.252968, 34.064937)
dtype [('SRAddress', 'S40'), ('longitude', '<f8'), ('latitude', '<f8')]

[GeoffreyWest]

Hi Dan, The above solution works, thank you so much for your assistance.  Finally, in line 15 above, what does [key] refer to?  How are each of the individuals keys referenced to this item?

[DanPatterson_Retired]

b = tuple([item[key] for key in keys])

key is just a reference to the list of keys created in line 2, ergo, what you have is a list comprehension so that it will take each key in the keys list ... go to the item dictionary and retrieve its value(s). when that is done, the list is converted to a tuple to facilitate converting to a numpy recarray.

Geoffrey, if I have answered your question, could you please mark the post that answered your question or mark with helpful, so that others know that a solution was found and the thread can be closed.

Thanks and good luck with your coding