爬虫笔记14——爬取网页数据写入MongoDB数据库，以爱奇艺为例

admin管理员组

文章数量:1636810

下载MongoDB数据库

首先，需要下载MongoDB数据库，下载的话比较简单，直接去官网找到想要的版本下载即可，具体安装过程可以看这里。

pycharm下载pymongo库

pip install pymongo

然后在在python程序中我们可以这样连接MongoDB数据库：

import pymongo

#指定数据库与表
# client = pymongo.MongoClient(host='127.0.0.1', port=27017)
# connect = client['table']

client = pymongo.MongoClient(host='127.0.0.1', port=27017)
connect = client['table']['table_info']
# 插入一条数据
info = {'name': 'python', 'age': 18}
result = connect.insert_one(info)
print(result)
# 查询数据
res = connect.find()
print(res)

# 插入多条数据
info_1 = {'name': 'python', 'age': 18}
info_2 = {'name': 'spider', 'age': 18}
result = connect.insert_many([info_1, info_2])
print(result)
res = connect.find()
print(list(res))

了解pymongo的常用语法后，我们来练习爬取爱奇艺的视频数据信息：标题、播放地址、简介并存入MongoDB数据库。
目标地址：https://list.iqiyi/www/2/15-------------11-1-1-iqiyi–.html?s_source=PCW_SC
可以先试试，再来看下面的代码：

# -*- coding: utf-8 -*-
# @Time:      2024/06/22 0:05
# @Author:     马再炜
# @File:       爬取爱奇艺存入MongoDB.py

import requests
import pymongo
import time

# 爬取爱奇艺的视频数据信息：标题、播放地址、简介并存入MongoDB数据库。
class AiQiYi:
    url = "https://pcw-api.iqiyi/search/recommend/list"

    def __init__(self):
        self.headers = {
            "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"
        }
        self.params = {
            "channel_id": "2",
            "data_type": "1",
            "mode": "11",
            "page_id": "2",
            "ret_num": "48",
            "session": "31dd983cf8e6ca3c75b4faaa17d88eac",
            "three_category_id": "15;must"
        }

    def require_info(self):
        response = requests.get(AiQiYi.url, headers=self.headers, params=self.params).json()
        # print(response["data"]["list"])
        return response["data"]["list"]

    def insert_in_mongo(self):
        insertLists = list()
        client = pymongo.MongoClient(host='127.0.0.1', port=27017)
        connect = client['py_spider']['movie_data']
        movieLists = self.require_info()
        # print(movieLists)
        for movie in movieLists:
            insertTemp = dict()
            insertTemp["movie_name"] = movie["name"]
            insertTemp["description"] = movie["description"]
            insertTemp["playUrl"] = movie["payMarkUrl"]
            # insertLists.append({
            #     "movie_name": movie["name"], "description": movie["description"], "playUrl": movie["payMarkUrl"]
            # })
            insertLists.append(insertTemp)
        # print(insertLists)
        connect.insert_many(insertLists)
        # time.sleep(1)
        print('插入完成！')

    def main(self):
        self.insert_in_mongo()


if __name__ == '__main__':
    aiqiyi = AiQiYi()
    aiqiyi.main()

最终结果如图：

本文标签： 爬虫为例数据库网页笔记