2

I was curious if you could remove duplicates from list of lists and return uniques as a list. I was trying this:

def do_list( lists ):
    res = [ [ one for one in temp if one not in res ] for temp in lists ]
    return res

So for example, if:

lists = [ [ "a","b","c" ],[ "d","a" ],[ "c","a","f" ] ]

the result should be:

[ "a","b,"c","d","f" ]

But it gives me error that I refference variable res before assigment.

TheTask1337
  • 349
  • 1
  • 2
  • 9

3 Answers3

6

You could do this:

set(itertools.chain.from_iterable(lists))

set will remove all duplicates, inside of the set is just flattening your list(s) to a single list.

Pythonista
  • 11,152
  • 2
  • 29
  • 49
2

You get an error because you're referencing res inside the comprehension. This doesn't work, since res is only available after the expression is finished.

As I'm a curious sort, and because the title asks "Removing duplicates from list of lists by using list comprehension", I wanted to see if you can do this using only a list comprehension, and not by cheating such as using itertools :p

And here's how:

>>> lists = [ [ "a","b","c" ],[ "d","a" ],[ "c","a","f" ] ]
>>> lists2 = sorted(sum(lists, []))
>>> [ item for i, item in enumerate(lists2) if i == 0 or i == len(lists2) or lists2[i - 1] != item ]
['a', 'b', 'c', 'd', 'f']

For more insanity, you can combine them on a single line, but you'd have to repeat the sum() and sorted() calls. I couldn't move myself to write such ugly code ;-)

  • sum(lists, []) will flatten the list; it returns the sum (+ operator) of all the items in lists, with [] as the initial list.
  • sorted() will sort it. This is needed since we only check against the last item
  • the if statement checks if the previous item is the same as the current item.

But it's ugly and un-Pythonic. For the love of Guido, use Pythonista's answer (or some variation thereof)!

Pythonista
  • 11,152
  • 2
  • 29
  • 49
Martin Tournoij
  • 24,971
  • 24
  • 101
  • 136
  • 2
    @TheTask1337 Not sure if "beautiful" is the right word to describe this sort of code ;-) It looks like the sort of thing a Perl programmer would do! :p Like I said, it's merely an educational exercise. Please don't actually use it ;-) – Martin Tournoij Mar 13 '16 at 02:07
1

res isn't created until the entire list comprehension has been evaluated. You can use a set to remove duplicates:

res = list(set(sum(lists, [])))

If you want it sorted:

res = sorted(set(sum(lists, [])))

If you want it ordered exactly how it comes, a list comprehension is probably not the best way. Instead, do this:

res = []
for temp in lists:
    res.append([])
    for one in temp:
        if one not in res[-1]:
            res[-1].append(one)
zondo
  • 19,040
  • 7
  • 42
  • 82