nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Removing duplicates from a dictionary

JamesCrandall

Hi all!

I'm in need of assistance in removing duplicate values within a single dictionary as I'm not having any success with the ideas/examples I've found.

I need to identify and remove any duplicate "asset" within this dicitionary:

{
"Features": [
  [
   {
    "asset": {
     "isCoastal": 0,
     "name": "G344E",
     "WMRegion": "SOUTHERN_REGION",
     "wcuname": "WCA3A",
     "type": "structure",
     "isActive": 1
    }
   },
   {
    "asset": {
     "isCoastal": 0,
     "name": "G344E",
     "WMRegion": "SOUTHERN_REGION",
     "wcuname": "WCA3A",
     "type": "structure",
     "isActive": 1
    }
   },
   {
    "asset": {
     "isCoastal": 0,
     "name": "G344E",
     "WMRegion": "SOUTHERN_REGION",
     "wcuname": "WCA3A",
     "type": "structure",
     "isActive": 1
    }
   },
   {
    "asset": {
     "isCoastal": 0,
     "name": "G344E",
     "WMRegion": "SOUTHERN_REGION",
     "wcuname": "WCA3A",
     "type": "structure",
     "isActive": 1
    }
   },
   {
    "asset": {
     "isCoastal": 0,
     "name": "G344F",
     "WMRegion": "SOUTHERN_REGION",
     "wcuname": "WCA3A",
     "type": "structure",
     "isActive": 1
    }
   },
   {
    "asset": {
     "isCoastal": 0,
     "name": "G344F",
     "WMRegion": "SOUTHERN_REGION",
     "wcuname": "WCA3A",
     "type": "structure",
     "isActive": 1
    }
   },
   {
    "asset": {
     "isCoastal": 0,
     "name": "S145",
     "WMRegion": "SOUTHERN_REGION",
     "wcuname": "WCA2B",
     "type": "structure",
     "isActive": 1
    }
   }
  ]
]
}

タグ付けされた投稿をもっと見る

Accepted answers

JamesCrandall

I think this does it. Would like to get validation though!

Thanks for looking.

    unique_data = []
    for d in JSONlist2:
        data_exists = False
        for ud in unique_data:
            if ud['asset'] == d['asset']:
              data_exists = True
              break
        if not data_exists:
            unique_data.append(d)

    data2['Features'] = unique_data

All comments

JamesCrandall

Nice!

ClintonDow1

Personally I found the listcomp solutions to be a bit hard to grok with the indices (no offense intended though, they are clever!)

This should get you a unique list of 'Asset' namedtuples - doesn't preserve the initial dictionary but since its a list of dictionaries each with a single kvp with a consistent key, its essentially a list anyways

from collections import namedtuple
asset = namedtuple('Asset', 'isCoastal name WMRegion wcuname type isActive')
unique_assets = []
for (k, v) in d.items():
    for l in v:
        for a in l:
            current = asset(**a['asset'])
            if current not in unique_assets:
                unique_assets.append(current)#  Asset(isCoastal=0, name='G344E', WMRegion='SOUTHERN_REGION', wcuname='WCA3A', type='structure', isActive=1)
#  Asset(isCoastal=0, name='G344F', WMRegion='SOUTHERN_REGION', wcuname='WCA3A', type='structure', isActive=1)
#  Asset(isCoastal=0, name='S145', WMRegion='SOUTHERN_REGION', wcuname='WCA2B', type='structure', isActive=1)

(Python 3.5 - d.items() would be d.iteritems() in 2.7)

JoshuaBixby

Xander Bakker‌, thanks. My initial thoughts were to use set, somehow, but that sent me down the rabbit hole of Python not having frozen dictionaries as a built-in data type. Implementing a frozen dict is quite simple, but the reasons PEP 416 -- Add a frozendict builtin type were rejected kept coming up, mainly performance. The idea of trying MappingProxyType was intriguing, but that would only work in ArcGIS Pro since it requires Python 3.3+. After running timeit on several different solutions, it became clear that the general approach James Crandall‌ settled on was going to perform the best.

JamesCrandall

Thanks Joshua -- I'm pretty sure I tried to implement that but didn't fully see (as usual) and overlooked what key to use and moved on to the next example. I will learn one of these days.

I still struggle with dictionaries, but they are really well suited for our GP services output because many of the org's apps are written by JavaScript developers and makes for a slick way to integrate them into our GIS world. In this instance, the dictionary output is really just the attribute result of spatial selections that run far more efficiently on the ArcGIS Server for obvious reasons, then just compiling the results into a dict to return to the calling application as JSON that they are accustomed to working with.

XanderBakker

That sounds a lot better! Thanks for sharing, bixb0012 !

JoshuaBixby

Adapting from Remove duplicate dict in list in Python, the same general logic/approach of your code can be implemented through a list comprehension:

dct<SPAN class="punctuation token">[</SPAN><SPAN class="string token">'Features'</SPAN><SPAN class="punctuation token">]</SPAN><SPAN class="punctuation token">[</SPAN><SPAN class="number token">0</SPAN><SPAN class="punctuation token">]</SPAN> <SPAN class="operator token">=</SPAN> <SPAN class="punctuation token">[</SPAN>d <SPAN class="keyword token">for</SPAN> n<SPAN class="punctuation token">,</SPAN> d
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <SPAN class="keyword token">in</SPAN> enumerate<SPAN class="punctuation token">(</SPAN>dct<SPAN class="punctuation token">[</SPAN><SPAN class="string token">'Features'</SPAN><SPAN class="punctuation token">]</SPAN><SPAN class="punctuation token">[</SPAN><SPAN class="number token">0</SPAN><SPAN class="punctuation token">]</SPAN><SPAN class="punctuation token">)</SPAN>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <SPAN class="keyword token">if</SPAN> d <SPAN class="operator token">not</SPAN> <SPAN class="keyword token">in</SPAN> dct<SPAN class="punctuation token">[</SPAN><SPAN class="string token">'Features'</SPAN><SPAN class="punctuation token">]</SPAN><SPAN class="punctuation token">[</SPAN><SPAN class="number token">0</SPAN><SPAN class="punctuation token">]</SPAN><SPAN class="punctuation token">[</SPAN>n<SPAN class="operator token">+</SPAN><SPAN class="number token">1</SPAN><SPAN class="punctuation token">:</SPAN><SPAN class="punctuation token">]</SPAN><SPAN class="punctuation token">]</SPAN><SPAN class="line-numbers-rows"><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN></SPAN>

Between switching to a list comprehension and using in with a slice of the original list, the adapted code runs right around twice as fast.

XanderBakker

Perhaps a one-liner would do (although readability will gone...)

dct<SPAN class="punctuation token">[</SPAN><SPAN class="string token">'Features'</SPAN><SPAN class="punctuation token">]</SPAN><SPAN class="punctuation token">[</SPAN><SPAN class="number token">0</SPAN><SPAN class="punctuation token">]</SPAN> <SPAN class="operator token">=</SPAN> list<SPAN class="punctuation token">(</SPAN><SPAN class="punctuation token">[</SPAN>eval<SPAN class="punctuation token">(</SPAN>s<SPAN class="punctuation token">)</SPAN> <SPAN class="keyword token">for</SPAN> s <SPAN class="keyword token">in</SPAN> set<SPAN class="punctuation token">(</SPAN><SPAN class="punctuation token">[</SPAN>str<SPAN class="punctuation token">(</SPAN>d<SPAN class="punctuation token">)</SPAN> <SPAN class="keyword token">for</SPAN> d <SPAN class="keyword token">in</SPAN> dct<SPAN class="punctuation token">[</SPAN><SPAN class="string token">'Features'</SPAN><SPAN class="punctuation token">]</SPAN><SPAN class="punctuation token">[</SPAN><SPAN class="number token">0</SPAN><SPAN class="punctuation token">]</SPAN><SPAN class="punctuation token">]</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">]</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="line-numbers-rows"><SPAN>‍</SPAN></SPAN>

The idea is to use the "set" method to get a unique list, but since you can't hash dictionaries, they are converted to string. Afterwards the "eval" (which one should never use) is used to create dictionaries again.

JamesCrandall

I was down that road but found the implementation I posted above. I'm pretty sure it's what I need, just need to validate when I get a chance.

Thanks!

DarrenWiens2

That's prettier than what I came up with (basically, muscle dictionary into a set, then back to dictionary):

<SPAN class="punctuation token">.</SPAN><SPAN class="punctuation token">.</SPAN><SPAN class="punctuation token">.</SPAN> my_set <SPAN class="operator token">=</SPAN> set<SPAN class="punctuation token">(</SPAN><SPAN class="punctuation token">[</SPAN><SPAN class="punctuation token">]</SPAN><SPAN class="punctuation token">)</SPAN>
<SPAN class="punctuation token">.</SPAN><SPAN class="punctuation token">.</SPAN><SPAN class="punctuation token">.</SPAN> <SPAN class="keyword token">for</SPAN> asset <SPAN class="keyword token">in</SPAN> dict<SPAN class="punctuation token">[</SPAN><SPAN class="string token">'Features'</SPAN><SPAN class="punctuation token">]</SPAN><SPAN class="punctuation token">[</SPAN><SPAN class="number token">0</SPAN><SPAN class="punctuation token">]</SPAN><SPAN class="punctuation token">:</SPAN>
<SPAN class="punctuation token">.</SPAN><SPAN class="punctuation token">.</SPAN><SPAN class="punctuation token">.</SPAN>&nbsp;&nbsp;&nbsp;&nbsp; <SPAN class="keyword token">for</SPAN> k1<SPAN class="punctuation token">,</SPAN>v1 <SPAN class="keyword token">in</SPAN> asset<SPAN class="punctuation token">.</SPAN>iteritems<SPAN class="punctuation token">(</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">:</SPAN>
<SPAN class="punctuation token">.</SPAN><SPAN class="punctuation token">.</SPAN><SPAN class="punctuation token">.</SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; new_list <SPAN class="operator token">=</SPAN> <SPAN class="punctuation token">[</SPAN><SPAN class="punctuation token">]</SPAN>
<SPAN class="punctuation token">.</SPAN><SPAN class="punctuation token">.</SPAN><SPAN class="punctuation token">.</SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <SPAN class="keyword token">for</SPAN> k2<SPAN class="punctuation token">,</SPAN>v2 <SPAN class="keyword token">in</SPAN> v1<SPAN class="punctuation token">.</SPAN>iteritems<SPAN class="punctuation token">(</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">:</SPAN>
<SPAN class="punctuation token">.</SPAN><SPAN class="punctuation token">.</SPAN><SPAN class="punctuation token">.</SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; new_list<SPAN class="punctuation token">.</SPAN>append<SPAN class="punctuation token">(</SPAN>str<SPAN class="punctuation token">(</SPAN>k2<SPAN class="punctuation token">)</SPAN> <SPAN class="operator token">+</SPAN> <SPAN class="string token">':'</SPAN> <SPAN class="operator token">+</SPAN> str<SPAN class="punctuation token">(</SPAN>v2<SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">)</SPAN> <SPAN class="comment token"># add key/value to a list</SPAN>
<SPAN class="punctuation token">.</SPAN><SPAN class="punctuation token">.</SPAN><SPAN class="punctuation token">.</SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; new_string <SPAN class="operator token">=</SPAN> <SPAN class="string token">','</SPAN><SPAN class="punctuation token">.</SPAN>join<SPAN class="punctuation token">(</SPAN>new_list<SPAN class="punctuation token">)</SPAN> <SPAN class="comment token"># convert list to string</SPAN>
<SPAN class="punctuation token">.</SPAN><SPAN class="punctuation token">.</SPAN><SPAN class="punctuation token">.</SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; my_set<SPAN class="punctuation token">.</SPAN>add<SPAN class="punctuation token">(</SPAN>new_string<SPAN class="punctuation token">)</SPAN> <SPAN class="comment token"># add string to set</SPAN>
<SPAN class="punctuation token">.</SPAN><SPAN class="punctuation token">.</SPAN><SPAN class="punctuation token">.</SPAN> new_dict <SPAN class="operator token">=</SPAN> dict
<SPAN class="punctuation token">.</SPAN><SPAN class="punctuation token">.</SPAN><SPAN class="punctuation token">.</SPAN> new_dict<SPAN class="punctuation token">[</SPAN><SPAN class="string token">'Features'</SPAN><SPAN class="punctuation token">]</SPAN><SPAN class="punctuation token">[</SPAN><SPAN class="number token">0</SPAN><SPAN class="punctuation token">]</SPAN> <SPAN class="operator token">=</SPAN> <SPAN class="punctuation token">[</SPAN><SPAN class="punctuation token">]</SPAN>
<SPAN class="punctuation token">.</SPAN><SPAN class="punctuation token">.</SPAN><SPAN class="punctuation token">.</SPAN> <SPAN class="keyword token">for</SPAN> asset <SPAN class="keyword token">in</SPAN> my_set<SPAN class="punctuation token">:</SPAN> <SPAN class="comment token"># convert set back to dictionary</SPAN>
<SPAN class="punctuation token">.</SPAN><SPAN class="punctuation token">.</SPAN><SPAN class="punctuation token">.</SPAN>&nbsp;&nbsp;&nbsp;&nbsp; new_dict<SPAN class="punctuation token">[</SPAN><SPAN class="string token">'Features'</SPAN><SPAN class="punctuation token">]</SPAN><SPAN class="punctuation token">[</SPAN><SPAN class="number token">0</SPAN><SPAN class="punctuation token">]</SPAN><SPAN class="punctuation token">.</SPAN>append<SPAN class="punctuation token">(</SPAN><SPAN class="punctuation token">{</SPAN><SPAN class="string token">'asset'</SPAN><SPAN class="punctuation token">:</SPAN><SPAN class="punctuation token">{</SPAN>i<SPAN class="punctuation token">.</SPAN>split<SPAN class="punctuation token">(</SPAN><SPAN class="string token">':'</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">[</SPAN><SPAN class="number token">0</SPAN><SPAN class="punctuation token">]</SPAN><SPAN class="punctuation token">:</SPAN>i<SPAN class="punctuation token">.</SPAN>split<SPAN class="punctuation token">(</SPAN><SPAN class="string token">':'</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">[</SPAN><SPAN class="number token">1</SPAN><SPAN class="punctuation token">]</SPAN> <SPAN class="keyword token">for</SPAN> i <SPAN class="keyword token">in</SPAN> asset<SPAN class="punctuation token">.</SPAN>split<SPAN class="punctuation token">(</SPAN><SPAN class="string token">','</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">}</SPAN><SPAN class="punctuation token">}</SPAN><SPAN class="punctuation token">)</SPAN>
<SPAN class="punctuation token">.</SPAN><SPAN class="punctuation token">.</SPAN><SPAN class="punctuation token">.</SPAN> <SPAN class="keyword token">print</SPAN> new_dict
<SPAN class="punctuation token">.</SPAN><SPAN class="punctuation token">.</SPAN><SPAN class="punctuation token">.</SPAN> 
<SPAN class="punctuation token">{</SPAN><SPAN class="string token">'Features'</SPAN><SPAN class="punctuation token">:</SPAN> <SPAN class="punctuation token">[</SPAN><SPAN class="punctuation token">[</SPAN><SPAN class="punctuation token">{</SPAN><SPAN class="string token">'asset'</SPAN><SPAN class="punctuation token">:</SPAN> <SPAN class="punctuation token">{</SPAN><SPAN class="string token">'isCoastal'</SPAN><SPAN class="punctuation token">:</SPAN> <SPAN class="string token">'0'</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'name'</SPAN><SPAN class="punctuation token">:</SPAN> <SPAN class="string token">'G344E'</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'WMRegion'</SPAN><SPAN class="punctuation token">:</SPAN> <SPAN class="string token">'SOUTHERN_REGION'</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'wcuname'</SPAN><SPAN class="punctuation token">:</SPAN> <SPAN class="string token">'WCA3A'</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'type'</SPAN><SPAN class="punctuation token">:</SPAN> <SPAN class="string token">'structure'</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'isActive'</SPAN><SPAN class="punctuation token">:</SPAN> <SPAN class="string token">'1'</SPAN><SPAN class="punctuation token">}</SPAN><SPAN class="punctuation token">}</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="punctuation token">{</SPAN><SPAN class="string token">'asset'</SPAN><SPAN class="punctuation token">:</SPAN> <SPAN class="punctuation token">{</SPAN><SPAN class="string token">'isCoastal'</SPAN><SPAN class="punctuation token">:</SPAN> <SPAN class="string token">'0'</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'name'</SPAN><SPAN class="punctuation token">:</SPAN> <SPAN class="string token">'G344F'</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'WMRegion'</SPAN><SPAN class="punctuation token">:</SPAN> <SPAN class="string token">'SOUTHERN_REGION'</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'wcuname'</SPAN><SPAN class="punctuation token">:</SPAN> <SPAN class="string token">'WCA3A'</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'type'</SPAN><SPAN class="punctuation token">:</SPAN> <SPAN class="string token">'structure'</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'isActive'</SPAN><SPAN class="punctuation token">:</SPAN> <SPAN class="string token">'1'</SPAN><SPAN class="punctuation token">}</SPAN><SPAN class="punctuation token">}</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="punctuation token">{</SPAN><SPAN class="string token">'asset'</SPAN><SPAN class="punctuation token">:</SPAN> <SPAN class="punctuation token">{</SPAN><SPAN class="string token">'isCoastal'</SPAN><SPAN class="punctuation token">:</SPAN> <SPAN class="string token">'0'</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'name'</SPAN><SPAN class="punctuation token">:</SPAN> <SPAN class="string token">'S145'</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'WMRegion'</SPAN><SPAN class="punctuation token">:</SPAN> <SPAN class="string token">'SOUTHERN_REGION'</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'wcuname'</SPAN><SPAN class="punctuation token">:</SPAN> <SPAN class="string token">'WCA2B'</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'type'</SPAN><SPAN class="punctuation token">:</SPAN> <SPAN class="string token">'structure'</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">'isActive'</SPAN><SPAN class="punctuation token">:</SPAN> <SPAN class="string token">'1'</SPAN><SPAN class="punctuation token">}</SPAN><SPAN class="punctuation token">}</SPAN><SPAN class="punctuation token">]</SPAN><SPAN class="punctuation token">]</SPAN><SPAN class="punctuation token">}</SPAN>‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍<SPAN class="line-numbers-rows"><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN></SPAN>

JamesCrandall

I think this does it. Would like to get validation though!

Thanks for looking.

    unique_data = []
    for d in JSONlist2:
        data_exists = False
        for ud in unique_data:
            if ud['asset'] == d['asset']:
              data_exists = True
              break
        if not data_exists:
            unique_data.append(d)

    data2['Features'] = unique_data

会話に参加する

サインインしたメンバーは投稿、更新のフォローなどができます。初めてですか？無料アカウントを登録してください。

クイックリンク

Community Help
Find useful guides, FAQs, and documents to help you navigate and make the most of Esri Community.