Home / Questions / 87813
Accepted
Jorge Ponti
Asked: 2020-07-19 10:27:38 +0800 CST 2020-07-19 10:27:38 +0800 CST

Remove duplicate items in a list

  • 772

I need to find the most efficient way to remove duplicates from a list in Python.

I am doing it this way:

for i in mj:
    if i not in mj2:
        mj2.append(i)

where kjis a list like [2, 4, 4, 4, 4, 4, 9, 9]and the output mj2is of the form:

   [2, 4, 9]

There is a more efficient way that doesn't include loops, since I have to parse large lists.

python

4 Answers

  1. Best Answer

    The simplest is to use set():

    >>> mj = [2, 4, 4, 4, 4, 4, 9, 9]
>>> mj2 = set(mj1)
>>> mj2
set([9, 2, 4])
>>> list(mj2)
[9, 2, 4]

    If you want to keep the order (since the setsare an unordered list of elements), you can pass a sortat the end:

    >>> sorted(list(mj2))
[2, 4, 9]

    Another option, if your list is originally ordered and you want to maintain the order, you can use the class OrderedDictand leverage it to maintain this order:

    >>> from collections import OrderedDict
>>> OrderedDict.fromkeys(mj)
OrderedDict([(2, None), (4, None), (9, None)])
>>> OrderedDict.fromkeys(mj).keys()
[2, 4, 9]

    OrderedDictis an implementation of dictionaries that allows you to "remember" the order in which its elements have been inserted. Therefore, you can use the fromkeysdictionary method to use the elements of mjas the keys of the dictionary, since the elements of mjare pre-ordered so the order is preserved.

    • 15

  2. You can test how the performance is with the following line of code: 

    mj2 = sorted(set(mj))

    although using sorted able to consume some resource. If you don't have problems with the order you can use as follows:

    mj2 = set(mj)
    • 4

  3. If the original list is very large and ordered, it's much more efficient to use itetools.groupbywhich creates an iterator without creating new lists:

    from itetools import groupby

mj2 = (k for (k,_) in groupby(m2))

    It is possible to get the first elements without processing the whole list:

    first = next(mj2)
second = next(mj2)
...
    • 3

  4. For 'groups' you can do it like this too.

    grupos= [['calculo', 1], ['calculo', 1], ['calculo', 1], ['Algebra', 5], ['Algebra', 5], ['Algebra', 5]]

grupos_sin_repetir = []
for i in grupos:
    if i not in grupos_sin_repetir:        
      grupos_sin_repetir.append(i)
    
print(grupos_sin_repetir)
    • 1