是否可以使用Rvest刮除html节点内的子类以外的数据?

2022-09-02 00:00:00 r rvest web-scraping html css

我有一个URL(https://forums.vwvortex.com/showthread.php?8829402-Atlas-V6-Oil-Change-Routine)可以从中抓取帖子。其中一些帖子是回复,其初始文本为"原始发帖人……"。我想刮除最初通过文本发布的帖子内的所有数据。例如,

User  df_text
 A    Hi, how are you ?
 B    This is beautiful!
 C    Heuwi
 D    Originally posted by C Heuwi 
      Hellou
 E    Hello guys
 F    Originally posted by A Hi, how are you ?
      I am doing good
 G    Whats going on ?

对于用户D,"发帖人.."位于div.QUOTE_CONTAINER类(子类)下,而"I Am Do Good"位于父类BLOCQUTE.postcontent.Restore下。

预期结果:

User  df_text
 A    Hi, how are you ?
 B    This is beautiful!
 C    Heuwi
 D    Hellou
 E    Hello guys
 F    I am doing good
 G    Whats going on ?

我尝试了以下代码:

url<-"https://forums.vwvortex.com/showthread.php?8829402-Atlas-V6-Oil-Change-Routine"
review <- read_html(url)
threads<- cbind(review %>% html_nodes("blockquote.postcontent.restore:not(.quote_container)") %>% html_text())

也尝试了其他几个:

threads <- cbind(review %>% html_nodes(xpath = '//div[@class="blockquote.postcontent.restore"]/node()[not(self::div)]') %>% html_text())

threads <- review %>% html_nodes(".content")
close_nodes <- threads %>% html_nodes(".quote_container")
chk <- xml_remove(close_nodes)

这些都不管用。请帮我找到一种方法来刮掉所有的帖子数据,不包括儿童类。提前感谢!!


解决方案

使用xml_remove函数是相对容易的解决方案,该函数是xml2库的一部分(使用rvest自动加载)

library(rvest)
#read page
url<-"https://forums.vwvortex.com/showthread.php?8829402-Atlas-V6-Oil-Change-Routine"
review <- read_html(url)

#find parent nodes
threads<- review %>% html_nodes("blockquote.postcontent.restore:not(.quote_container)")
#find children nodes to exclude
toremove<-threads %>% html_node("div.bbcode_container")
#remove nodes
xml_remove(toremove)

#convert the parent nodes to text
threads %>% html_text(trim=TRUE)

来自xml_remove的文档:"使用XML_Remove()时需要小心"。请经常审阅、谨慎使用并保存。

相关文章