别人没那么重要,我也没那么重要,好好活着,把能做的小事做好,够不到的东西就放弃,承认就好。做一个心情好能睡着的人,你所有事情都会在正轨上。

Python数据分析-数据合并

  Pandas模块的merge()concat()方法常用于数据合并。

1. 方法:merge()

语法如下:

DataFrame.merge(right, how='inner', on=None, left_on=None, right_on=None, left_index=False, right_index=False, sort=False, suffixes=('_x', '_y'), copy=None, indicator=False, validate=None)

  Merge DataFrame or named Series objects with a database-style join.

  A named Series object is treated as a DataFrame with a single named column.

  The join is done on columns or indexes. If joining columns on columns, the DataFrame indexes will be ignored. Otherwise if joining indexes on indexes or indexes on a column or columns, the index will be passed on. When performing a cross merge, no column specifications to merge on are allowed.

参数说明:

  • right:DataFrame or named Series

  Object to merge with.

  • how:{‘left’, ‘right’, ‘outer’, ‘inner’, ‘cross’}, default ‘inner’

  Type of merge to be performed.

    • left: use only keys from left frame, similar to a SQL left outer join; preserve key order.
    • right: use only keys from right frame, similar to a SQL right outer join; preserve key order.
    • outer: use union of keys from both frames, similar to a SQL full outer join; sort keys lexicographically.
    • inner: use intersection of keys from both frames, similar to a SQL inner join; preserve the order of the left keys.
    • cross: creates the cartesian product from both frames, preserves the order of the left keys.
  • on:label or list

  Column or index level names to join on. These must be found in both DataFrames. If on is None and not merging on indexes then this defaults to the intersection of the columns in both DataFrames.

  • left_on:label or list, or array-like

  Column or index level names to join on in the left DataFrame. Can also be an array or list of arrays of the length of the left DataFrame. These arrays are treated as if they are columns.

  • right_on:label or list, or array-like

  Column or index level names to join on in the right DataFrame. Can also be an array or list of arrays of the length of the right DataFrame. These arrays are treated as if they are columns.

  • left_index:bool, default False

  Use the index from the left DataFrame as the join key(s). If it is a MultiIndex, the number of keys in the other DataFrame (either the index or a number of columns) must match the number of levels.

  • right_index:bool, default False

  Use the index from the right DataFrame as the join key. Same caveats as left_index.

  • sort:bool, default False

  Sort the join keys lexicographically in the result DataFrame. If False, the order of the join keys depends on the join type (how keyword).

  • suffixes:list-like, default is (“_x”, “_y”)

  A length-2 sequence where each element is optionally a string indicating the suffix to add to overlapping column names in left and right respectively. Pass a value of None instead of a string to indicate that the column name from left or right should be left as-is, with no suffix. At least one of the values must not be None.

  • copy:bool, default True

  If False, avoid copy if possible.

  • indicator:bool or str, default False

  If True, adds a column to the output DataFrame called “_merge” with information on the source of each row. The column can be given a different name by providing a string argument. The column will have a Categorical type with the value of “left_only” for observations whose merge key only appears in the left DataFrame, “right_only” for observations whose merge key only appears in the right DataFrame, and “both” if the observation’s merge key is found in both DataFrames.

  • validate:str, optional

If specified, checks if merge is of specified type.

    • “one_to_one” or “1:1”: check if merge keys are unique in both left and right datasets.
    • “one_to_many” or “1:m”: check if merge keys are unique in left dataset.
    • “many_to_one” or “m:1”: check if merge keys are unique in right dataset.
    • “many_to_many” or “m:m”: allowed, but does not result in checks.

代码示例1:

 1 df1 = pd.DataFrame({'lkey': ['foo', 'bar', 'baz', 'foo'],
 2                     'value': [1, 2, 3, 5]})
 3 df2 = pd.DataFrame({'rkey': ['foo', 'bar', 'baz', 'foo'],
 4                     'value': [5, 6, 7, 8]})
 5 
 6 print(df1)
 7 print(df2)
 8 
 9 ### 结果
10 #   lkey  value
11 # 0  foo      1
12 # 1  bar      2
13 # 2  baz      3
14 # 3  foo      5
15 
16 #   rkey  value
17 # 0  foo      5
18 # 1  bar      6
19 # 2  baz      7
20 # 3  foo      8
 1 # 合并lkey和rkey列上的df1和df2。,值列附加了默认后缀_x和_y
 2 df3 = df1.merge(df2, left_on='lkey', right_on='rkey')
 3 print(df3)
 4 
 5 ### 结果
 6 #   lkey  value_x rkey  value_y
 7 # 0  foo        1  foo        5
 8 # 1  foo        1  foo        8
 9 # 2  foo        5  foo        5
10 # 3  foo        5  foo        8
11 # 4  bar        2  bar        6
12 # 5  baz        3  baz        7
 1 # 合并DataFrames df1和df2,并将指定的左后缀和右后缀附加到任何重叠的列上
 2 df3 = df1.merge(df2, left_on='lkey', right_on='rkey', suffixes=('_left', '_right'))
 3 print(df3)
 4 
 5 ### 结果
 6 #   lkey  value_left rkey  value_right
 7 # 0  foo           1  foo            5
 8 # 1  foo           1  foo            8
 9 # 2  foo           5  foo            5
10 # 3  foo           5  foo            8
11 # 4  bar           2  bar            6
12 # 5  baz           3  baz            7
1 # 合并dataframe df1和df2,但如果dataframe有任何重叠列,则引发异常
2 df3 = df1.merge(df2, left_on='lkey', right_on='rkey', suffixes=(False, False))
3 print(df3)
4 
5 ### 结果
6 # Traceback (most recent call last):
7 # ...
8 # ValueError: columns overlap but no suffix specified:
9 #     Index(['value'], dtype='object')

代码示例2:

 1 df1 = pd.DataFrame({'a': ['foo', 'bar'], 'b': [1, 2]})
 2 df2 = pd.DataFrame({'a': ['foo', 'baz'], 'c': [3, 4]})
 3 print(df1)
 4 print(df2)
 5 
 6 ### 结果
 7 #      a  b
 8 # 0  foo  1
 9 # 1  bar  2
10 #      a  c
11 # 0  foo  3
12 # 1  baz  4
1 df3 = df1.merge(df2, how='inner', on='a')
2 print(df3)
3 
4 ### 结果
5 #      a  b  c
6 # 0  foo  1  3
1 df3 = df1.merge(df2, how='left', on='a')
2 print(df3)
3 
4 ### 结果
5 #       a  b  c
6 # 0   foo  1  3.0
7 # 1   bar  2  NaN
 1 df1 = pd.DataFrame({'left': ['foo', 'bar']})
 2 df2 = pd.DataFrame({'right': [7, 8]})
 3 
 4 df3 = df1.merge(df2, how='cross')
 5 print(df3)
 6 
 7 ### 结果
 8 #   left  right
 9 # 0  foo      7
10 # 1  foo      8
11 # 2  bar      7
12 # 3  bar      8

2. 方法:concat()

语法如下:

pandas.concat(objs, *, axis=0, join='outer', ignore_index=False, keys=None, levels=None, names=None, verify_integrity=False, sort=False, copy=None)

  Concatenate pandas objects along a particular axis.

  Allows optional set logic along the other axes.

  Can also add a layer of hierarchical indexing on the concatenation axis, which may be useful if the labels are the same (or overlapping) on the passed axis number.

参数说明:

  • objs:a sequence or mapping of Series or DataFrame objects

  If a mapping is passed, the sorted keys will be used as the keys argument, unless it is passed, in which case the values will be selected (see below). Any None objects will be dropped silently unless they are all None in which case a ValueError will be raised.

  • axis:{0/’index’, 1/’columns’}, default 0

  The axis to concatenate along.

  • join:{‘inner’, ‘outer’}, default ‘outer’

  How to handle indexes on other axis (or axes).

  • ignore_index:bool, default False

  If True, do not use the index values along the concatenation axis. The resulting axis will be labeled 0, …, n - 1. This is useful if you are concatenating objects where the concatenation axis does not have meaningful indexing information. Note the index values on the other axes are still respected in the join.

  • keys:sequence, default None

  If multiple levels passed, should contain tuples. Construct hierarchical index using the passed keys as the outermost level.

  • levels:list of sequences, default None

  Specific levels (unique values) to use for constructing a MultiIndex. Otherwise they will be inferred from the keys.

  • names:list, default None

  Names for the levels in the resulting hierarchical index.

  • verify_integrity:bool, default False

  Check whether the new concatenated axis contains duplicates. This can be very expensive relative to the actual data concatenation.

  • sort:bool, default False

  Sort non-concatenation axis if it is not already aligned.

  • copy:bool, default True

  If False, do not copy data unnecessarily.

代码示例1:

 1 s1 = pd.Series(['a', 'b'])
 2 s2 = pd.Series(['c', 'd'])
 3 df1 = pd.concat([s1, s2])
 4 print(df1)
 5 
 6 ### 结果
 7 # 0    a
 8 # 1    b
 9 # 0    c
10 # 1    d
11 # dtype: object
 1 # 通过将ignore_index选项设置为True,清除现有索引并在结果中重置它
 2 df1 = pd.concat([s1, s2], ignore_index=True)
 3 print(df1)
 4 
 5 ### 结果
 6 # 0    a
 7 # 1    b
 8 # 2    c
 9 # 3    d
10 # dtype: object
 1 # 使用keys选项在数据的最外层添加层次索引
 2 df1 = pd.concat([s1, s2], keys=['s1', 's2'])
 3 print(df1)
 4 
 5 ### 结果
 6 # s1  0    a
 7 #     1    b
 8 # s2  0    c
 9 #     1    d
10 # dtype: object
 1 # 用names选项标记创建的索引
 2 df1 = pd.concat([s1, s2], keys=['s1', 's2'], names=['Series name', 'Row ID'])
 3 print(df1)
 4 
 5 ### 结果
 6 # Series name  Row ID
 7 # s1           0         a
 8 #              1         b
 9 # s2           0         c
10 #              1         d
11 # dtype: object

代码示例2:

 1 # 组合具有相同列的两个DataFrame对象
 2 df1 = pd.DataFrame([['a', 1], ['b', 2]], columns=['letter', 'number'])
 3 df2 = pd.DataFrame([['c', 3], ['d', 4]], columns=['letter', 'number'])
 4 df3 = pd.concat([df1, df2])
 5 print(df3)
 6 
 7 ### 结果
 8 #   letter  number
 9 # 0      a       1
10 # 1      b       2
11 # 0      c       3
12 # 1      d       4
 1 # 组合具有重叠列的DataFrame对象并返回所有内容。交集之外的列将用NaN值填充
 2 df3 = pd.DataFrame([['c', 3, 'cat'], ['d', 4, 'dog']], columns=['letter', 'number', 'animal'])
 3 df4 = pd.concat([df1, df3], sort=False)
 4 print(df4)
 5 
 6 ### 结果
 7 #   letter  number animal
 8 # 0      a       1    NaN
 9 # 1      b       2    NaN
10 # 0      c       3    cat
11 # 1      d       4    dog
 1 # 组合具有重叠列的DataFrame对象,并仅返回那些通过向join关键字参数传递inner共享的列
 2 df4 = pd.concat([df1, df3], join="inner")
 3 print(df4)
 4 
 5 ### 结果
 6 #   letter  number
 7 # 0      a       1
 8 # 1      b       2
 9 # 0      c       3
10 # 1      d       4
1 # 通过传入axis=1,沿x轴水平组合DataFrame对象
2 df4 = pd.DataFrame([['bird', 'polly'], ['monkey', 'george']], columns=['animal', 'name'])
3 df5 = pd.concat([df1, df4], axis=1)
4 print(df5)
5 
6 ### 结果
7 #   letter  number  animal    name
8 # 0      a       1    bird   polly
9 # 1      b       2  monkey  george
 1 # 使用verify_integrity选项防止结果包含重复的索引值
 2 df5 = pd.DataFrame([1], index=['a'])
 3 df6 = pd.DataFrame([2], index=['a'])
 4 
 5 df7 = pd.concat([df5, df6], verify_integrity=True)
 6 print(df7)
 7 
 8 ### 结果
 9 # Traceback (most recent call last):
10 #     ...
11 # ValueError: Indexes have overlapping values: ['a']
 1 # 在DataFrame对象的末尾追加一行
 2 df7 = pd.DataFrame({'a': 1, 'b': 2}, index=[0])
 3 new_row = pd.Series({'a': 3, 'b': 4})
 4 df8 = pd.concat([df7, new_row.to_frame().T], ignore_index=True)
 5 print(df8)
 6 
 7 ### 结果
 8 #    a  b
 9 # 0  1  2
10 # 1  3  4

 

时间:2024年2月8日

 

posted @ 2024-02-08 14:26  一路狂奔的乌龟  阅读(48)  评论(0)    收藏  举报
返回顶部