现象

源代码如下

class HrSpider4Spider(CrawlSpider):
    """CrawlSpider类"""
    name = 'hr_spider4'
    allowed_domains = ['https://hr.tencent.com']  # 留意此处是一个完整的URL地址
    start_urls = ["https://hr.tencent.com/position.php?&start=0"]

    rules = (
        Rule(LinkExtractor(allow=r'position.php\?&start=\d+')), # 默认follow参数为True,表示继续提取
        Rule(LinkExtractor(allow=r'position_detail\.php\?id=\d+'), callback="parse_item", follow=False)
    )

    def parse_item(self, response):
        item = PositionItem()
        item['position_duty'] = response.xpath("//ul[@class='squareli']")[0].xpath(".//li/text()").extract()
        item['position_duty'] = response.xpath("//ul[@class='squareli']")[1].xpath(".//li/text()").extract()
        yield item

在运行该爬虫的时候会报错:
URLWarning: allowed_domains accepts only domains, not URLs.
原因显而易见: 允许范围接收的是范围, 而非URL地址.

解决方法

将第4行代码修改为

allowed_domains = ['hr.tencent.com'] 

也就是仅保留后缀.