使用 Xpath 处理较大的 XML 文件的最佳方法是什么?

2022-01-10 00:00:00 python xml xml-parsing

问题描述

我有一个要求，我必须使用大型 XML(4 GB 文件)来查找其中的值.基本上我必须编写大约 30 个不同的 Xpath 并将值存储在一个列表中.当我尝试解析 XML 时，它会引发内存错误.我已经尝试使用 lxml 和 ElementTree 来处理开始和结束事件，但处理时间太长还是不行，而且我的 Pycharm/Jupyter 笔记本会引发内存错误.

I have a requirement where I have to use an large XML (4 GB file) for finding values in it. Basically I have to write around 30 different Xpath and store the values in a list. When I try to parse an XML, it throws memory error. I have tried using lxml and ElementTree with start and end events, still no luck the processing time is too high and my Pycharm/Jupyter notebook throws me memory error.

有没有更好的方法呢?尽管此实现不限于任何编程语言，但我更喜欢 Python，因为它是我的右手.提前致谢.

Is there a better way to do it? Even though this implementation is not restricted to any programming language, I prefer Python because its my right hand. Thanks in advance.

例如搜索:如果我想要类别正在烹饪的年份的值.然后我使用 ./bookstore[@category=cooking]/book/year.所以值为 2005

Eg of a search: If I want the value of year where category is cooking. Then I use ./bookstore[@category=cooking]/book/year. So the value is 2005

同样，我必须根据我的业务需求找到我的标签的值.在我的要求中，XML 并不像下面的示例那样简单.

Similarly I have to find the values of my tags based on my business requirements. In my requirement the XML is not simple as the below example.

<?xml version="1.0" encoding="UTF-8"?>

<bookstore>

<book category="cooking">
  <title lang="en">Everyday Italian</title>
  <author>Giada De Laurentiis</author>
  <year>2005</year>
  <price>30.00</price>
</book>

<book category="children">
  <title lang="en">Harry Potter</title>
  <author>J K. Rowling</author>
  <year>2005</year>
  <price>29.99</price>
</book>

<book category="web">
  <title lang="en">XQuery Kick Start</title>
  <author>James McGovern</author>
  <author>Per Bothner</author>
  <author>Kurt Cagle</author>
  <author>James Linn</author>
  <author>Vaidyanathan Nagarajan</author>
  <year>2003</year>
  <price>49.99</price>
</book>

</bookstore>

解决方案

不知道这样能不能满足你的需求.先试试看.如果有问题，我们可以继续沟通.

I don't know if this will meet your needs. Try it first. If there is a problem, we can continue to communicate.

from simplified_scrapy import SimplifiedDoc,req,utils
def getBook(lines,category,year):
  html = "".join(lines)
  book = SimplifiedDoc(html).book
  if book.category==category and book.year.text==year:
    return book.category,book.title.text,book.authors.text,book.year.text,book.price.text
lst = []
with open('bookstore.xml', 'r') as file: # bookstore.xml is your xml file path
  lines = []
  flag = False
  for line in file:
    if flag or line.find('<book ')>=0:
      flag = True
      lines.append(line)
    if line.find('</book>')>=0:
      b = getBook(lines,'web','2003')
      if b: 
        lst.append(b)
        break # If you only want the first one, add a break
      flag = False
      lines = []
print (lst)

结果:

[('web', 'XQuery Kick Start', ['James McGovern', 'Per Bothner', 'Kurt Cagle', 'James Linn', 'Vaidyanathan Nagarajan'], '2003', '49.99')]

相关文章