Scrapy Core API cleanup & per-spider settings

by Julia Medina for Python Software Foundation

The proposed project aims to add a new mechanism to override Scrapy settings with spiders (classes where users define crawling behaviour). In order to do that, and pursuing a simpler and easier API usage, a major refactoring in core code of crawl process and settings population has to be made. This proposal attempts to describe all required changes and estimate how much time they would take.