逐行比较文件是否相同,如果相同则输出它们
问题描述
我该怎么做,我有文件,我已经在其中对信息进行了排序,我想将该文件中的某个索引与另一个文件中的索引进行比较,一个问题是文件非常大,数百万行.我想逐行比较我拥有的文件,如果它们匹配,我想使用索引方法输入这两个值以及其他值.
How would I go about this, I have files which I have sorted the information in, I want to compare a certain index in that file with an index in another, one problem is that the files are enormously large, millions of lines. I want to compare line by line the files I have, if they match I want to input both those values along with other values using an index method.
=========================
=======================
让我澄清一下,我想说 line[x] x 将保持不变,因为它是统一格式的,我想在另一个文件中针对 line[y] 运行 line[x],我想这样做到整个文件并将每个匹配对输出到另一个文件.在那个其他文件中,我还希望能够包含第一个文件中的其他部分,就像添加更多索引一样;line[a],line[b],line[c],line[d],最后是 line[y] 作为该信息的匹配项.
Let me clarify, I want to take say line[x] the x will remain the same as it is formatted uniformly, I want to run line[x] against line[y] in another file, I want to do this to the whole file and output every matching pair to another file. In that other file I also want to be able to include other pieces from the first file which would be like just adding more indexes such as; line[a],line[b],line[c],line[d], and finally line[y] as the match to that information.
尝试 3:
我有一个包含这种格式信息的文件:
I have a file with information in this format:
#x 是一行
x= data,data,data,data,data,data
有数百万行.
我有另一个文件,格式相同:
I have another file, same format:
xis a line
x= data,data,data,data
我想使用第一个文件中的 x[#] 和第二个文件中的 x[#],我想看看这两个值是否匹配,如果它们匹配,我想输出它们以及其他几个 x[#]第二个文件中的值,位于同一行.
I want to use x[#] from first file and x[#] from second file, I want to see if those two values match, if they do I want to output those, along with several other x[#] values from the second file, which are on the same line.
这对理解有帮助吗?文件的格式就像我说的那样:(但是有数百万,我想在两个文件中找到对,因为它们都应该匹配)
Did that help at all to understand? The format the files are in are like i said:(but there is millions, and I want to find the pairs in the two files because they all should match up)
line 1 data,data,data,data
line 2 data,data,data,data
来自文件 1 的数据:
data from file 1:
(N'068D556A1A665123A6DD2073A36C1CAF', N'A76EEAF6D310D4FD2F0BD610FAC02C04DFE6EB67',
N'D7C970DFE09687F1732C568AE1CFF9235B2CBB3673EA98DAA8E4507CC8B9A881');
来自文件 2 的数据:
data from file 2:
00000040f2213a27ff74019b8bf3cfd1|index.docbook|Redhat 7.3 (32bit)|Linux
00000040f69413a27ff7401b8bf3cfd1|index.docbook|Redhat 8.0 (32bit)|Linux
00000965b3f00c92a18b2b31e75d702c|Localizable.strings|Mac OS X 10.4|OSX
0000162d57845b6512e87db4473c58ea|SYSTEM|Windows 7 Home Premium (32bit)|Windows
000011b20f3cefd491dbc4eff949cf45|totem.devhelp|Linux Ubuntu Desktop 9.10 (32bit)|Linux
它的排序顺序是字母数字,我想使用滑块方法.我的意思是如果 file1[x] 是 <file2[x] 根据一个值是否大于另一个值向下或向上移动滑块,直到找到匹配项,如果找到匹配项,则打印输出以及可以识别该哈希的其他值.
The order it is sorted in is alphanumeric, and I want to use a slider method. By that I mean if file1[x] is < file2[x] move the slider down or up depending on whether one value is greater than the other, until a match is found, when and if so, print the output along with other values that will identify that hash.
我想要的结果是:
file1[x] 及其在 file2[x] 上的对应匹配输出到一个文件,以及其他 file1[x],其中 x 可以是该行中的任何索引.
file1[x] and its corresponding match on file2[x] outputted to a file, as well as other file1[x] where x can be any index from the line.
解决方案
我从澄清中得到的:
file1 和 file2 格式相同,每一行的样子
file1 and file2 are in the same format, where each line looks like
{32 char hex key}|{text1}|{text2}|{text3}
文件按key升序排列
the files are sorted in ascending order by key
对于同时出现在 file1 和 file2 中的每个键,您需要合并输出,所以每一行看起来像
for each key that appears in both file1 and file2, you want merged output, so each line looks like
{32 char hex key}|{text11}|{text12}|{text13}|{text21}|{text22}|{text23}
您基本上想要来自合并排序的冲突:
You basically want the collisions from a merge sort:
import csv
def getnext(csvfile, key=lambda row: int(row[0], 16)):
row = csvfile.next()
return key(row),row
with open('file1.dat','rb') as inf1, open('file2.dat','rb') as inf2, open('merged.dat','wb') as outf:
a = csv.reader(inf1, delimiter='|')
b = csv.reader(inf2, delimiter='|')
res = csv.writer(outf, delimiter='|')
a_key, b_key = -1, 0
try:
while True:
while a_key < b_key:
a_key, a_row = getnext(a)
while b_key < a_key:
b_key, b_row = getnext(b)
if a_key==b_key:
res.writerow(a_row + b_row[1:])
except StopIteration:
# reached the end of an input file
pass
我仍然不知道你试图通过以及其他 file1[x] 来传达什么,其中 x 可以是行中的任何索引".
I still have no idea what you are trying to communicate by 'as well as other file1[x] where x can be any index from the line'.
相关文章