Scrapy only outputs '[' '

Question

私はフロントページのすべてのredditリンクを取るだけで、scrapyを使用してWebクローラーを構築しています。私がjsonフォルダに入れようとすると、私が得るのは「[」です。

ここは私のクモです。

from scrapy import Spider 
from scrapy.selector import Selector 
from redditScrape.items import RedditscrapeItem 


class RedditSpider(Spider): 
    name = "redditScrape" 
    allowed_domains = ["reddit.com"] 
    start_urls = [ 
     "https://www.reddit.com/r/all" 
    ] 

    def parse(self, response): 
     titles = Selector(response).xpath('//div[@class="entry unvoted lcTagged"]/p[@class="title"]') 

     for title in titles: 
      item = RedditscrapeItem() 
      item['title'] = title.xpath('/a[@class="title may-blank loggedin srTagged imgScanned"]/text()').extract() 
      yield item 

私はGoogle Chromeのコンソールでxpathクエリを実行するたびに、結果を探しています。なぜ私のスクレーパー文句を言わないの出力を正しく

任意のアイデア？

scrapy crawl redditScrape -o items.json -t json 

Answer 1

を私は問題がある内容を正確に把握していないが、私は、私はあなたのコードで間違って何を参照してくださいに行く必要があります。

これは、私が実行するために使用していますコマンドです。

• まず第一に、私は-t引数があるかわからないが、私はあなたが、出力がJSONファイルであったことを安心させたかった疑い。あなたはする必要はありません。 -o items.jsonで十分です。 scrapy crawl redditScrape -o items.json

• Selectorを宣言する必要はありません。同様にtitles = response.xpath('//div[@class="entry unvoted lcTagged"]/p[@class="title"]')でもかまいません。これは、生活の質の向上である限り、エラーではありません。アイテムが正常に得られているときはいつでも

• 二xpathがitem['title'] = title.xpath('a[@class="title may-blank loggedin srTagged imgScanned"]/text()').extract_first()

控えめに言っても日陰で、scrapyは、ランタイムで、出力ファイルに追加します。

編集。

このxpath //p[@class="title"]/a/text()を使用すると、すべてのタイトルをフロントページから取得できます。あなたのコードでは、それはこのCSSセレクタは、すべてのタイトルを取得します。この

for title in response.xpath('//p[@class="title"]/a'): 
     item = RedditscrapeItem() 
     item['title'] = title.xpath('text()').extract_first() 
     yield item 

Answer 2

のようになります。

In [13]: response.css("a.title.may-blank::text").extract() 
Out[13]: 
[u'TIL of a millionaire who announced he would bury his Bentley for his afterlife. After lots of negative reaction, he revealed the publicity stunt about organ donations. "People bury things that are much more valuable then cars and nobody seems to care".', 
u'Dog thinks he has a bunch of friends', 
u'Sewage leak at a movie theater. Looks like black tile.', 
u'3:48 am "Hydraulic Press"', 
u'I told her it was for their protection...', 
u'Long visits to nature linked to improved mental health, study finds', 
u"Vladimir Putin Says Brexit Caused by British Politicians 'Arrogance'", 
u"World's smallest man dancing with his pet cat. 26th October 1956.", 
u'I am Sue Sullivan and Reddit saved my sauce and rub company, Hot Squeeze. Tomorrow, I\u2019m heading to Wal-Mart for my last, big pitch for distribution. Whatever happens, I wanted to say thank you for all your support and AMA! Helping me out with this AMA will be Lawrence Wu, the founder WUJU hot sauce!', 
u"Cartoons made me think dog catchers were super common, but now I'm pretty sure they don't even exist", 
u'Zarya ultimate chain kill', 
u'Shaqiri scores vs Poland to make it 1-1', 
u'Mythbusters, during their later seasons', 
u"'Why not Texit?': Texas nationalists look to the Brexit vote for inspiration", 
u'Ken M on Hitler', 
u'Skill, pure skill', 
u'My girlfriend paints things. This is a pair of Vans she is currently working on.', 
u'I made a magnet wall to display my PS4 steelbook game collection!', 
u'HuffPo in 2008: "Muslims appear to be far more concerned about perceived slights to their religion than about atrocities committed daily in its name"', 
u"It's been almost 3 years since the removal of the Rose block. Never forget.", 
u"Xherdan Shaqiri's insane bicycle kick goal vs. Poland", 
u"US Customs wants to collect social media account names at the border: 'Please enter information associated with your online presence'", 
u'How was the cameraman for Finding Dory able to hold his breath for the entire filming?', 
u'Star Guardian Urgot', 
u'I made some doorstops! (Not as lame as it sounds)'] 

アイテムを追加するには、あなたのコードは、単に必要があります。

In [9]: for text in response.css("a.title.may-blank::text").extract(): 
    ...:  item['title'] = text 
    ...:  yield item