如何通过 pandas 或火花数据框删除所有行中具有相同值的列?
问题描述
假设我有类似以下的数据:
Suppose I've data similar to following:
index id name value value2 value3 data1 val5
0 345 name1 1 99 23 3 66
1 12 name2 1 99 23 2 66
5 2 name6 1 99 23 7 66
我们如何在一个命令中删除所有行具有相同值的所有列，例如 (value, value2, value3)还是使用 python 的几个命令?
How can we drop all those columns like (value, value2, value3) where all rows have the same values, in one command or couple of commands using python?
假设我们有许多列类似于 value、value2、value3...value200.
Consider we have many columns similar to value, value2, value3...value200.
输出:
index id name data1
0 345 name1 3
1 12 name2 2
5 2 name6 7
解决方案
我们可以做的是使用 nunique 计算数据框每一列中唯一值的个数，并丢弃只有一个唯一值:
What we can do is use nunique to calculate the number of unique values in each column of the dataframe, and drop the columns which only have a single unique value:
In [285]:
nunique = df.nunique()
cols_to_drop = nunique[nunique == 1].index
df.drop(cols_to_drop, axis=1)
Out[285]:
index id name data1
0 0 345 name1 3
1 1 12 name2 2
2 5 2 name6 7
另一种方法是只 diff 数字列，获取 abs 值和 sums 它们:
Another way is to just diff the numeric columns, take abs values and sums them:
In [298]:
cols = df.select_dtypes([np.number]).columns
diff = df[cols].diff().abs().sum()
df.drop(diff[diff== 0].index, axis=1)
Out[298]:
index id name data1
0 0 345 name1 3
1 1 12 name2 2
2 5 2 name6 7
另一种方法是使用具有相同值的列的标准差为零的属性:
Another approach is to use the property that the standard deviation will be zero for a column with the same value:
In [300]:
cols = df.select_dtypes([np.number]).columns
std = df[cols].std()
cols_to_drop = std[std==0].index
df.drop(cols_to_drop, axis=1)
Out[300]:
index id name data1
0 0 345 name1 3
1 1 12 name2 2
2 5 2 name6 7
其实以上都可以单行完成:
Actually the above can be done in a one-liner:
In [306]:
df.drop(df.std()[(df.std() == 0)].index, axis=1)
Out[306]:
index id name data1
0 0 345 name1 3
1 1 12 name2 2
2 5 2 name6 7
相关文章