Python去重方法

1、利用集合set的特性

>>> lst = [1, 2 , 2, 3, 3, 5, 6, 7, 2, 1]
>>> list(set(lst))
[1, 2, 3, 5, 6, 7]
>>>

可以通过列表中索引（index）的方法保证去重后的顺序不变

>>> ls = [1, 2, 3, 3, 5, 2, 1, 3, 6, 1]
>>> lst = list(set(ls))
>>> lst.sort(key=ls.index)
>>> lst
[1, 2, 3, 5, 6]
>>>

如果序列的值都是 hashable 类型，可以利用生成器解决去重后的顺序不变的问题。（Python Cookbook）

>>> def dedupe(items):
...     seen = set()
...     for item in items:
...         if item not in seen:
...             yield item
...             seen.add(item)
...
>>>
>>> lst = [1, 2, 3, 3, 5, 2, 1, 3, 6, 1]
>>> list(dedupe(lst))
[1, 2, 3, 5, 6]
>>>

2、利用字典键不能重复的特性

>>> lst
[1, 2, 2, 3, 3, 5, 6, 7, 2, 1]
>>> list(dict.fromkeys(lst))
[1, 2, 3, 5, 6, 7]
>>>

3、利用for循环

>>> new_list = []
>>> for i in lst:
	if i not in new_list:
		new_list.append(i)

		
>>> new_list
[1, 2, 3, 5, 6, 7]
>>>

4、利用while循环

>>> def dup_rem(lst):
	for i in lst:
		while lst.count(i) > 1:
			lst.remove(i)
	return lst

>>> 
>>> ls = [1, 2, 3, 3, 5, 2, 1, 3, 6, 1]
>>> dup_rem(ls)
[5, 2, 3, 6, 1]

5、使用itertools模块的grouby方法

>>> import itertools
>>> ls = [1, 2, 3, 3, 5, 2, 1, 3, 6, 1]
>>> ls.sort()
>>> lt = itertools.groupby(ls)
>>> lt       # lt 是 itertools模块下的groupby对象，类似生成器
<itertools.groupby object at 0x000002543BED2C78>
>>> list(lt)
[(1, <itertools._grouper object at 0x000002543BE82278>), (2, <itertools._grouper object at 0x000002543BED54A8>),
 (3, <itertools._grouper object at 0x000002543BED55F8>), 
(5, <itertools._grouper object at 0x000002543BED5630>), (6, <itertools._grouper object at 0x000002543BED5668>)]
>>> for k, g in lt:
	print(k, end=' ')

	
>>>

二、移除重复数据