推荐学习书目
› Learn Python the Hard Way
Python Sites
› PyPI - Python Package Index
› http://diveintopython.org/toc/index.html
› Pocoo
值得关注的项目
› PyPy
› Celery
› Jinja2
› Read the Docs
› gevent
› pyenv
› virtualenv
› Stackless Python
› Beautiful Soup
› 结巴中文分词
› Green Unicorn
› Sentry
› Shovel
› Pyflakes
› pytest
Python 编程
› pep8 Checker
Styles
› PEP 8
› Google Python Style Guide
› Code Style from The Hitchhiker's Guide
Colorful
V2EX  ›  Python

请教个爬 虫存 mysql 的问题

  •  
  •   Colorful · May 7, 2020 · 3586 views
    This topic created in 2378 days ago, the information mentioned may be changed or developed.

    import pymongo

    from pymongo import MongoClient

    import pymysql as pq

    class Pipeline(object): def init(self): self.conn = pq.connect(host='127.0.0.1', user='root',port='3306', passwd='123456', db='root', charset='utf8') self.cur = self.conn.cursor() # def init(self): # host = '127.0.0.1' # port = 27017 # client = pymongo.MongoClient(host=host,port=port) # collection = client.local.letsgojpt # self.post = collection

    def process_item(self, item, spider):
        for i in range(0,len(item["title_all"])):
            # sql = "insert into letsgojp(title_all, center_all) VALUES (%s, %s)"
            self.cur.execute("insert into letsgojp(title_all,center_all) values(%s,%s)",(item['title_all'][i],item['center_all'][i]))
            self.conn.commit()
            # self.post.insert(data)
        return item
    
    def close_spider(self, spider):
        self.cur.close()
        self.conn.close()
    

    主要是这行报错

    self.cur.execute("insert into letsgojp(title_all,center_all) values(%s,%s)",(item['title_all'][i],item['center_all'][i]))

    用 mongo 的时候是正常存数据的,但是用 mysql 就报错,网上找一些,不太会用,请教各位大神了

    14 replies  •  2020-05-08 16:19:26 +08:00
    ClericPy
    ClericPy
    May 7, 2020
    1
    1. 找个贴代码的地方把缩进留下...

    2. 报错信息解释 90% 问题, 大部分时间报错信息最后一行一搜 stackoverflow 上都有同行遇到过

    3. 纯看的话, 实在猜不到啊... 就语法来说, pymysql 会自动转义, 问题不大, 看你的列名也没有敏感关键字所以不加反引号也没事
    jugelizi
    jugelizi
    May 7, 2020
    2
    [item['title_all'][i],item['center_all'][i]] 是不是?
    也有可能没处理好特殊字符 错误提示贴下啊
    Colorful
    Colorful
    OP
    May 7, 2020
    3
    @jugelizi 应该不会的吧,因为存 mongo 的时候是没总是的
    chaneyccy
    chaneyccy
    May 7, 2020
    4
    不贴报错信息,这也看不出来哪里有问题吧

    没用过 SQL 好奇搜了一圈,没有发现模块导入是 import pymysql as pq,问题在这?
    Colorful
    Colorful
    OP
    May 7, 2020
    5
    @chaneyccy 不是那里,能爬到一条数据
    self.cur.execute("insert into letsgojp(title_all,center_all) values(%s,%s)",(item['title_all'][i],item['center_all'][i]))

    主要就是这行
    aydd2004
    aydd2004
    May 7, 2020
    6
    execute 的内容 print 出来看一下

    实在不行粘到 mysql 里面执行一下
    dorothyREN
    dorothyREN
    May 8, 2020
    7
    占位符那里的问题,插入字符串 需要 加上 \'%s\'
    dorothyREN
    dorothyREN
    May 8, 2020
    8
    @dorothyREN #7 ``` def save_data(self, title, content):
    sql = "INSERT INTO test(title,content) VALUES (\'%s\',\'%s\')" % (title, content)
    # print(sql)
    self.cur.execute(sql)
    self.conn.commit()
    ```
    crella
    crella
    May 8, 2020 via Android
    9
    mysql 新版的 mb-utf8 编码设置有点麻烦。

    建议直接 postgresql
    triangle111
    triangle111
    May 8, 2020
    10
    print 一下 sql 语句到 mysql 执行就知道错哪了, 应该是字符串没有引号导致的吧
    annielong
    annielong
    May 8, 2020
    11
    一般都是编码和非法字符的问题,打印 sql 语句就可以看出来
    zhangysh1995
    zhangysh1995
    May 8, 2020
    12
    添加 try except,抓 mysql.connector.errors.DatabaseError 错误
    leapV3
    leapV3
    May 8, 2020
    13
    你试试将表名与列名加上反引号
    AmberJiang
    AmberJiang
    May 8, 2020
    14
    感觉像是 insert 语句那里存在语法错误。。。贴报错图出来看看吧
    About   ·   Help   ·   Advertise   ·   Blog   ·   API   ·   FAQ   ·   Privacy   ·   Solana   ·   946 Online   Highest 6679   ·     Select Language
    创意工作者们的社区
    World is powered by solitude
    VERSION: 3.9.8.5 · 44ms · UTC 20:16 · PVG 04:16 · LAX 12:16 · JFK 15:16
    ♥ Do have faith in what you're doing.