Community
Select to view content in your preferred language

Trouble accessing data in multi-dimensional JSON for NumPyArray to Feature

3916
3
05-08-2015 11:22 PM
GeoffreyWest
by
Frequent Contributor
‎05-08-2015 11:22 PM

This is more of a general Python question which will set me up to send this data from a NumPyArray to Feature Class.  I am trying to parse through a web-service and retrieve certain records, however I am consistently receiving a KeyError for responses with more than one object. These records are returned in intervals, so sometimes I might receive one record and others I might receive 300. If I receive one record, the logic of my code works, if there are multiple items then the code doesn't work. Here is an example of an output with one object. I have posted to StackOverFlow and no luck, so this is cross-posted.

{
   "status": {
   "code": 311,
   "message": "Service Request Successfully Queried.",
   "cause": ""
   },
   "Response": {
   "NumOutputObjects": "1",
   "ListOfServiceRequest": {
   "ServiceRequest": [
   {
   "AddressVerified": "Y",
   "SRNumber": "1-13967451",
   "SRType": "Service Not Complete",
   "CreatedDate": "05/08/2015 10:00:38",
   "UpdatedDate": "05/08/2015 10:00:49",
   "IntegrationId": "05082015100148678",
   "Status": "Open",
   "CreatedByUserLogin": "PROXYE",
   "UpdatedByUserLogin": "PROXYE",
   "Anonymous": "N",
   "Zipcode": "90032",
   "Latitude": "34.0843242531",
   "Longitude": "-118.171015007",
   "CustomerAccessNumber": "",
   "LADWPAccountNo": "",
   "NewContactFirstName": "jj",
   "NewContactLastName": "rambo",
   "NewContactPhone": "",
   "NewContactEmail": "",
   "ParentSRNumber": "1-10552271",
   "Priority": "Normal",
   "Language": "",
   "ReasonCode": "",
   "ServiceDate": "",
   "Source": "",
   "ClosedDate": "",
   "Email": "",
   "FirstName": "",
   "HomePhone": "",
   "LastName": "",
   "LoginUser": "",
   "ResolutionCode": "",
   "SRUnitNumber": "",
   "MobilOS": "",
   "SRAddress": "5163 E TEMPLETON ST, 90032",
   "SRAddressName": "",
   "SRAreaPlanningCommission": "East Los Angeles APC",
   "SRCommunityPoliceStation": "CENTRAL BUREAU",
   "SRCouncilDistrictMember": "Jose Huizar",
   "SRCouncilDistrictNo": "14",
   "SRDirection": "E",
   "SRNeighborhoodCouncilId": "48",
   "SRNeighborhoodCouncilName": "LA-32 NC",
   "SRStreetName": "TEMPLETON",
   "SRSuffix": "ST",
   "SRTBColumn": "F",
   "SRTBMapGridPage": "595",
   "SRTBRow": "6",
   "SRXCoordinate": "6509897",
   "SRYCoordinate": "1853117",
   "AssignTo": "NC",
   "Assignee": "NC eWaste Supervisor 01",
   "Owner": "BOS",
   "ParentSRStatus": "Open",
   "ParentSRType": "Electronic Waste",
   "ParentSRLinkDate": "05/08/2015 10:00:39",
   "ParentSRLinkUser": "PROXYE",
   "SRAreaPlanningCommissionId": "5",
   "SRCommunityPoliceStationAPREC": "HOLLENBECK",
   "SRCommunityPoliceStationPREC": "4",
   "SRCrossStreet": "",
   "ActionTaken": "",
   "SRCity": "",
   "RescheduleCounter": "",
   "SRHouseNumber": "5163",
   "SourceofRequestCouncil": "",
   "CCBPremiseType": "",
   "ContainerBlackCount": "",
   "ContainerBrownCount": "",
   "SRIntersectionDirection": "",
   "SRApproximateAddress": "N",
   "ContainerGreenCount": "",
   "OtherBureauName": "",
   "AssigneeName": "",
   "AssigneeOrganization": "E-Waste, NC",
   "AnotherBureauEmailId": "",
   "ListOfLa311BarricadeRemoval": {},
   "ListOfLa311BulkyItem": {},
   "ListOfLa311DeadAnimalRemoval": {},
   "ListOfLa311GraffitiRemoval": {},
   "ListOfLa311InformationOnly": {},
   "ListOfLa311MultipleStreetlightIssue": {},
   "ListOfLa311SingleStreetlightIssue": {},
   "ListOfLa311SrPhotoId": {
   "La311SrPhotoId": []
   },
   "ListOfLa311BusPadLanding": {},
   "ListOfLa311CurbRepair": {},
   "ListOfLa311Flooding": {},
   "ListOfLa311GeneralStreetInspection": {},
   "ListOfLa311GuardWarningRailMaintenance": {},
   "ListOfLa311GutterRepair": {},
   "ListOfLa311LandMudSlide": {},
   "ListOfLa311Pothole": {},
   "ListOfLa311Resurfacing": {},
   "ListOfLa311SidewalkRepair": {},
   "ListOfLa311StreetSweeping": {},
   "ListOfLa311BeesOrBeehive": {},
   "ListOfLa311MedianIslandMaintenance": {},
   "ListOfLa311OvergrownVegetationPlants": {},
   "ListOfLa311PalmFrondsDown": {},
   "ListOfLa311StreetTreeInspection": {},
   "ListOfLa311StreetTreeViolations": {},
   "ListOfLa311TreeEmergency": {},
   "ListOfLa311TreeObstruction": {},
   "ListOfLa311TreePermits": {},
   "ListOfLa311BrushItemsPickup": {},
   "ListOfLa311Containers": {},
   "ListOfLa311ElectronicWaste": {},
   "ListOfLa311IllegalDumpingPickup": {},
   "ListOfLa311ManualPickup": {},
   "ListOfLa311MetalHouseholdAppliancesPickup": {},
   "ListOfLa311MoveInMoveOut": {},
   "ListOfLa311HomelessEncampment": {},
   "ListOfLa311IllegalAutoRepair": {},
   "ListOfLa311IllegalConstruction": {},
   "ListOfLa311IllegalConstructionFence": {},
   "ListOfLa311IllegalDischargeOfWater": {},
   "ListOfLa311IllegalDumpingInProgress": {},
   "ListOfLa311IllegalExcavation": {},
   "ListOfLa311IllegalSignRemoval": {},
   "ListOfLa311IllegalVending": {},
   "ListOfLa311LeafBlowerViolation": {},
   "ListOfLa311NewsRackViolation": {},
   "ListOfLa311Obstructions": {},
   "ListOfLa311TablesAndChairsObstructing": {},
   "ListOfLa311GisLayer": {
   "La311GisLayer": [
   {
   "A_Call_No": "",
   "Area": "0",
   "Day": "MONDAY",
   "DirectionSuffix": "",
   "DistrictAbbr": "",
   "DistrictName": "NC",
   "DistrictNumber": "",
   "DistrictOffice": "",
   "Fraction": "",
   "R_Call_No": "",
   "SectionId": "",
   "ShortDay": "",
   "StreetFrom": "",
   "StreetTo": "",
   "StreetLightId": "",
   "StreetLightStatus": "",
   "Type": "GIS",
   "Y_Call_No": "",
   "Name": "05082015100148678100",
   "CommunityPlanningArea": "",
   "LastUpdatedBy": "",
   "BOSRadioHolderName": ""
   }
   ]
   },
   "ListOfLa311ServiceRequestNotes": {
   "La311ServiceRequestNotes": [
   {
   "CreatedDate": "05/08/2015 10:00:39",
   "Comment": "Materials have been out in a normal collection area, unsure why driver missed the e-waste items.",
   "CreatedByUser": "PROXYE",
   "IsSrNoAvailable": "",
   "CommentType": "External",
   "Notification": "N",
   "FeedbackSRType": "",
   "IntegrationId": "050820151001486782",
   "Date1": "",
   "Date2": "",
   "Date3": "",
   "Text1": "",
   "AnotherBureau": "",
   "EmailAddress": "",
   "ListOfLa311SrNotesAuditTrail": {}
   },
   {
   "CreatedDate": "05/08/2015 10:00:39",
   "Comment": "",
   "CreatedByUser": "PROXYE",
   "IsSrNoAvailable": "",
   "CommentType": "Address Comments",
   "Notification": "N",
   "FeedbackSRType": "",
   "IntegrationId": "050820151001486781",
   "Date1": "",
   "Date2": "",
   "Date3": "",
   "Text1": "",
   "AnotherBureau": "",
   "EmailAddress": "",
   "ListOfLa311SrNotesAuditTrail": {}
   }
   ]
   },
   "ListOfLa311SubscribeDuplicateSr": {},
   "ListOfChildServiceRequest": {},
   "ListOfLa311BillingCsscAdjustment": {},
   "ListOfLa311BillingEccAdjustment": {},
   "ListOfLa311BillingRsscAdjustment": {},
   "ListOfLa311BillingRsscExemption": {},
   "ListOfLa311SanitationBillingBif": {},
   "ListOfLa311SanitationBillingCssc": {},
   "ListOfLa311SanitationBillingEcc": {},
   "ListOfLa311SanitationBillingLifeline": {},
   "ListOfLa311SanitationBillingRssc": {},
   "ListOfLa311SanitationBillingSrf": {},
   "ListOfLa311DocumentLog": {},
   "ListOfAuditTrailItem2": {
   "AuditTrailItem2": [
   {
   "Date": "05/08/2015 10:00:49",
   "EmployeeLogin": "SADMIN",
   "Field": "Assignee",
   "NewValue": "NC eWaste Supervisor 01",
   "OldValue": ""
   }
   ]
   },
   "ListOfLa311GenericBc": {
   "La311GenericBc": [
   {
   "ATTRIB_08": "",
   "NAME": "05082015100148678100",
   "PAR_ROW_ID": "1-8BDCR",
   "ROW_ID": "1-8BOCG",
   "TYPE": "GIS",
   "ATTRIB_16": "",
   "ListOfLa311GenericbcAuditTrail": {}
   },
   {
   "ATTRIB_08": "",
   "NAME": "05082015100148678",
   "PAR_ROW_ID": "1-8BDCR",
   "ROW_ID": "1-8BOCJ",
   "TYPE": "Service Not Complete",
   "ATTRIB_16": "",
   "ListOfLa311GenericbcAuditTrail": {}
   }
   ]
   },
   "ListOfLa311ServiceNotComplete": {
   "La311ServiceNotComplete": [
   {
   "ContainerLocation": "",
   "ContainerType": "",
   "DriverFirstName": "",
   "DriverLastName": "",
   "MissedCollectionService": "Electronic Waste",
   "OtherServiceMissedReason": "",
   "ServiceDateRendered": "",
   "ServiceMissedReason": "I'm not sure",
   "TruckNo": "",
   "Type": "Service Not Complete",
   "WireBasketLocation": "",
   "LastUpdatedBy": "",
   "Name": "05082015100148678"
   }
   ]
   },
   "ListOfLa311Other": {},
   "ListOfLa311WeedAbatementForPrivateParcels": {},
   "ListOfLa311SanitationBillingInquiry": {}
   }
   ]
   }
   }
}

The code is below;


data2 = jsonpickle.decode((f2.read()))
Start = datetime.datetime.now()
data2 = jsonpickle.encode(data2)
url2 = "myURL"
headers2 = {'Content-type': 'text/plain', 'Accept': '/'}

r2 = requests.post(url2, data=data2, headers=headers2)
decoded2 = json.loads(r2.text)
try:
  r2
except requests.exceptions.ConnectTimeout as e:
   print "Too slow Mojo!"


items = []
for sr in decoded2['Response']['ListOfServiceRequest']['ServiceRequest']:
   SRAddress = sr['SRAddress']
   Latitude = sr['Latitude']
   Longitude = sr['Longitude']
   ReasonCode = sr['ReasonCode']
   SRNumber = sr['SRNumber']
   FirstName = sr['FirstName']
   LastName = sr['LastName']
   ResolutionCode = sr['ResolutionCode']
   HomePhone = sr['HomePhone']
   CreatedDate = sr['CreatedDate']
   UpdatedDate = sr['UpdatedDate']
   CreatedDate = datetime.datetime.strptime(CreatedDate, "%m/%d/%Y %H:%M:%S")
   UpdatedDate = datetime.datetime.strptime(UpdatedDate, "%m/%d/%Y %H:%M:%S")
   print SRAddress
   print SRNumber


ItemInfo = " "
for ew in sr["ListOfLa311ServiceRequestNotes"][u"La311ServiceRequestNotes"]:
Comment = ew['Comment']
print Comment

Output Materials have been out in a normal collection area, unsure why driver missed the e-waste items.

If I use the logic above for a response with more than one object returned I receive a KeyError value and am unable to access the array that I want to parse.

Example of code with multiple objects returned;

Output when I use; I receive a key error if I attempt to do something along the lines of CommodityType = sr['ListOfLa311ElectronicWaste']['ElectronicWasteType'] below

for sr in decoded2['Response']['ListOfServiceRequest']['ServiceRequest']:
   CommodityType = sr['ListOfLa311ElectronicWaste']
   # ItemType = sr['ElectronicWestType']
   # DriverFirstName = sr ['DriverFirstName']
   # DriverLastName = sr ['DriverLastName']
   # ItemCount = sr['ItemCount']
   # ItemInfo += '{0}, {1}, '.format(ItemType, ItemCount)
   # ParentNumber = sr['Name']
   #
   print CommodityType




{u'La311ElectronicWaste': [{u'IllegallyDumped': u'N', u'OtherElectronicWestType': u'hash', u'ItemCount': u'5', u'Name': u'6a31f058-ece1-4e7d-b682-7d9052a512f4', u'MobileHomeSpace': u'', u'DriverLastName': u'', u'ActiveStatus': u'Y', u'DriverFirstName': u'', u'LastUpdatedBy': u'', u'GatedCommunityMultifamilyDwelling': u'Outside the main gate', u'IllegalDumpCollectionLoc': u'', u'ElectronicWestType': u'Other', u'CollectionLocation': u'Gated Community', u'Type': u'Electronic Waste', u'ServiceDateRendered': u'', u'TruckNo': u''}]}
{u'La311ElectronicWaste': [{u'IllegallyDumped': u'Y', u'OtherElectronicWestType': u'', u'ItemCount': u'5', u'Name': u'3f4d9d20-a712-4be3-822f-e6a45219c1cf', u'MobileHomeSpace': u'', u'DriverLastName': u'', u'ActiveStatus': u'Y', u'DriverFirstName': u'', u'LastUpdatedBy': u'', u'GatedCommunityMultifamilyDwelling': u'', u'IllegalDumpCollectionLoc': u'Cul De Sac', u'ElectronicWestType': u'Electronic Equipment', u'CollectionLocation': u'Alley', u'Type': u'Electronic Waste', u'ServiceDateRendered': u'', u'TruckNo': u''}]}
0 Kudos
3 Replies
CarlSunderman
by
Frequent Contributor
‎05-11-2015 04:55 AM

You could try using a getcount and a if result >1, that should handle if it returns more than one

0 Kudos
GeoffreyWest
by
Frequent Contributor
‎05-11-2015 07:01 AM

I was able to solve this with an inner loop of those items. Now I have another issue where the inner loop is not writing to the db.  The inner loops read as;

for sr in Manual:
   for manualcount in Manual["La311ManualPickup":]:
   countmanual = manualcount['ItemCount']
print countmanual
0 Kudos
JeffWard
by
Honored Contributor
‎05-11-2015 07:49 AM

Your dictionary doesn't need the : in your key reference:

for manualcount in Manual["La311ManualPickup"]:
Jeff Ward
Summit County, Utah
0 Kudos