Home / Questions / 289577
Accepted
ortiga
Asked: 2020-09-01 02:16:46 +0800 CST 2020-09-01 02:16:46 +0800 CST

remove duplicates in python

  • 772

I have a python file with the following content:

file.txt

http://www.alhondigalaunion.es
http://www.alhondigalaunion.es
http://www.unicagroup.es
http://www.jcarrion.es
http://www.jcarrion.es
http://www.aceiteslapedriza.com
http://www.vicasol.com
http://www.medgaz.com
http://www.medgaz.com
http://www.medgaz.com

How could I remove the duplicates in python working directly on the file without deleting it? I'm trying with set but nothing I leave you the code:

 with open(file, 'r') as f2:
        for i in f2:
            array = set(f2)
            print(array)                   
            with open(file2, 'a+') as e:
                for a in array:
                    e.writelines(array[a]) + '\n')
python

1 Answers

  1. Best Answer

    Just read all the lines of the file with readlines()and save them in a set(). This type of data does not save duplicates, but only once each different data. Then you dump the contents of the set to another file (or to the same file, if you want to make the change "in-place"). The trick is to convert the set to a string so you can dump it, which is easy via str.join().

    Namely:

    with open("file.txt", "r") as f_in:
   unicos = set(f_in.readlines())
with open("output.txt", "w") as f_out:
   f_out.write("".join(unicos))
    • 4