meta The initial values for the Request.meta attribute. If given, the dict passed in this parameter will be shallow copied.该参数也是字典形式 , 是为了在spider类的多个parse函数之间传递信息 , 见 知乎 。 注意Response对象也有一个它对应的Request对象 :The Request object that generated this response. This attribute is assigned in the Scrapy engine, after the response and the request have passed through all Downloader Middlewares In particular, this means that:HTTP redirections will cause the original request (to the URL before redirection) to be assigned to the redirected response (with the final URL after redirection).Response.request.url doesn’t always equal Response.urlThis attribute is only available in the spider code, and in the Spider Middlewares, but not in Downloader Middlewares (although you have the Request available there by other means) and handlers of the response_downloaded signal.But Unlike the Response.request attribute, the Response.meta attribute is propagated along redirects and retries, so you will get the original Request.meta sent from your spider.Response obejct这里仅介绍一些reponse对象的属性:
- url 即该response的来源url
- status 即该response的状态码
- headers response的响应头 , 形式为dict
- body response的相应数据体 , 形式为bytes
- request response对应的Request对象 , 对于它上文已经介绍 , 即Response.url可能不等于Reponse.request.url , 因为redirection的原因
SettingsSettings can be populated using different mechanisms, each of which having a different precedence. Here is the list of them in decreasing order of precedence:
- Command line options (most precedence)
- Settings per-spider
- Project settings module(settings.py)
- Default settings per-command
- Default global settings (less precedence)
一般我们直接在settings.py文件中对其进行修改 , 常见需要增改的有:user-agent指定 , ITEM_PIPELINES解除注释以开启pipeline功能 , LOG_LEVEL和LOG_FILE指定 , ROBOTSTXT_OBEY设为False等等 。
【Scrapy框架初探】
推荐阅读
-
-
-
-
爆米花沏茶泡汤了是什么生肖 爆米花沏茶泡汤了是什么生肖歇后语
-
吉雅志佩历史|还将春节定为法定节日,中国最后的藩属国不仅将中国人印在钱上
-
『运势』5月4日星座运势:巨蟹拓展眼界,天秤拼尽全力,水瓶贵人相助
-
-
-
一朝天子一朝臣,领导换届期间如何与新领导建立关系?
-
太史慈|三国志战略版: 谁说白板没活路?输出减伤两开花的吴肉弓前来报到
-
|?大叔收藏了数不清的藏品,唯独对其中的一件有疑问,忐忑鉴宝
-
游戏|LOL:PDD表示SN运营问题蛮大,想当冠军还差一些品质!
-
人民网|广西:建立企业信用修复机制 以“稳企”促“活市”
-
成都市锦江区教育局|成都一幼稚园多名儿童呕吐腹泻 官方通报:系诺如病毒感染
-
菁妈育儿|脏一点更健康,邋遢家长养出健康娃!孩子身上几个部位别乱洗
-
-
龙背坊|他会是你的福星,能助你钱财如雨下!,属羊人:请珍惜这个人
-
美女|美女的修身连衣裙,既显瘦又有魅力,“照搬”就行!
-
铿锵一族|天天兄弟前主持沦为网红,呆萌型男恋上争议女明星,小五怎么了?
-
从中国独立出快100年的外蒙古,现在究竟如何?驴友道出实情!