使用BeautifulSoup对网页进行文本分析和数据挖掘

pip install beautifulsoup4

import requests
from bs4 import BeautifulSoup

# 发送请求获取网页内容
url = "https://pidancode.com"
response = requests.get(url)
html_content = response.content

# 使用BeautifulSoup解析html内容
soup = BeautifulSoup(html_content, 'html.parser')

# 找到所有a标签并输出链接和文本内容
a_tags = soup.find_all('a')
for a in a_tags:
    print(a.get('href'), a.text)

# 找到所有h2标签并输出文本内容
h2_tags = soup.find_all('h2')
for h2 in h2_tags:
    print(h2.text)

# 找到class为post-meta的元素并输出文本内容
post_meta = soup.find_all(class_='post-meta')
for meta in post_meta:
    print(meta.text)

https://pidancode.com 网站首页
https://pidancode.com/about/ 关于
https://pidancode.com/archive/ 文章归档
https://pidancode.com/category/python/ Python
https://pidancode.com/category/nodejs/ Node.js
https://pidancode.com/category/ai/ 人工智能
https://pidancode.com/category/javascript/ JavaScript
https://pidancode.com/post/104/ 浏览器渲染流程详解
https://pidancode.com/post/103/ Python如何快速读写Excel文件
https://pidancode.com/post/100/ Python面试宝典：必备Python技能汇总
https://pidancode.com/post/98/ Python与MongoDB的最佳实践
...
Python如何快速读写Excel文件
Python面试宝典：必备Python技能汇总
Python与MongoDB的最佳实践
JavaScript防抖和节流原理详解
AI中的Batch Normalization
Python基础教程：Python常用数据类型详解
Python与Redis的最佳实践
post on December 14, 2021 by admin
post on December 11, 2021 by admin
post on December 7, 2021 by admin
post on December 4, 2021 by admin
post on December 1, 2021 by admin
post on November 27, 2021 by admin
post on November 24, 2021 by admin
post on November 20, 2021 by admin