Python list comprehension, with unique items

Question

Is there a way to make a list comprehension in Python that only contains unique items?

My original idea was to use something like this : new_items = [unicode(item) for item in items]

However, I later realized that I needed to omit duplicate items. So I ended up with this ugly monstrosity :

unique_items = []
for item in items :
    unicode_item = unicode(item)
    if unicode_item not in unique_items :
        unique_items.append(unicode_item)

Now this is far less pretty (and readable) than a simple list comprehension. So, is there a way to make a list comprehension equivalent to the above code?

Also order does matter, so I can't just use a set comprehension.

Interesting, my initial thought was that you can do filtering in list comprehensions, but then I realized you'd need access to the new list that you're creating in the filter condition. — Davy8, Oct 01 '12 at 22:05
If order matters, how do I know what instance of a repeated item to use? The first, the last, or one in the middle? — lvella, Oct 01 '12 at 22:09
possible duplicate of [How do you remove duplicates from a list in Python whilst preserving order?](http://stackoverflow.com/questions/480214/how-do-you-remove-duplicates-from-a-list-in-python-whilst-preserving-order) — mgilson, Oct 01 '12 at 22:12
@mgilson After a quick check it seems the accepted answer to this question is more efficient than the accepted answer of the linked question. — Dunes, Oct 01 '12 at 22:43

Michael · Accepted Answer · 2012-10-01T22:22:42.240

18

Well, there is no ordered set, but we can misuse OrderedDict:

from collections import OrderedDict
t = "never gonna give you up"
OrderedDict.fromkeys(t).keys()

Gives:

['n', 'e', 'v', 'r', ' ', 'g', 'o', 'a', 'i', 'y', 'u', 'p']

edited Oct 01 '12 at 22:22

answered Oct 01 '12 at 22:14

Michael

6,725
1
36
62

Oh, very nice. On a side note, try using `repeat(None)` from `itertools` rather than `[None]*len(t)` – Dunes Oct 01 '12 at 22:20
3

How about `OrderedDict.fromkeys(t).keys()` instead? [Note that this approach, as well as the set approach, limits us to hashable elements.] – DSM Oct 01 '12 at 22:20
Yep, better. Thanks for suggestion! – Michael Oct 01 '12 at 22:22

score 10 · Answer 2 · answered Sep 25 '14 at 07:17

10

Your original idea works with a set comprehension:

new_items = {unicode(item) for item in items}

answered Sep 25 '14 at 07:17

schryer

103
1
3

score 6 · Answer 3 · answered Oct 01 '12 at 22:12

6

I short one liner might be:

s = "some string"
unique_items = [unicode(ch) for ch in sorted(set(s), key=s.index)]

answered Oct 01 '12 at 22:12

Dunes

34,599
7
78
91

1

Works, but isn't very efficient for larger lists. – l4mpi Oct 01 '12 at 22:13
1

Nope, but it's short and easy to understand its intent. – Dunes Oct 01 '12 at 22:16

score 4 · Answer 4 · answered Oct 01 '12 at 22:04

Make it a helper function, like so.

def unique_iter(iterable):
  seen = set()
  for item in iterable:
    if item in seen:
      continue
    seen.add(item)
    yield item

for ch in unique_iter("never gonna give you up"):
  print ch,

outputs

n e v r g o a i y u p

Python list comprehension, with unique items

4 Answers4

Linked