python - very simple scrapy crawler not following links -


this simple scrapy spider crawls yelp.com , fetches data

i've set rule(linkextractor(allow=('.*')),follow=true,callback="parsebusiness")

to follow links , callback parsebusiness

however, scrapy here, not follow links

this specific output (full output here http://pastebin.com/bkuervmq)

2015-07-14 01:06:22 [scrapy] debug: telnet console listening on 127.0.0.1:6023 2015-07-14 01:06:25 [scrapy] debug: crawled (200) <get http://www.yelp.com/search?find_desc=hotels&find_loc=san+francisco%2c+ca&ns=1> (referer: none) 2015-07-14 01:06:26 [scrapy] debug: crawled (200) <get http://www.yelp.com/biz/ucsf-medical-center-at-mount-zion-san-francisco> (referer: none) 2015-07-14 01:06:26 [scrapy] info: closing spider (finished) 2015-07-14 01:06:26 [scrapy] info: dumping scrapy stats:


code below

import sys import scrapy  scrapy.spiders import crawlspider, rule scrapy.linkextractors import linkextractor   class business(scrapy.item):     name = scrapy.field()     contactnumber = scrapy.field()     address = scrapy.field()  class yelpspider(crawlspider):     name = "yelp"     allowed_domains = ["www.yelp.com"]     start_urls = [     "http://www.yelp.com/search?find_desc=hotels&find_loc=san+francisco%2c+ca&ns=1",     "http://www.yelp.com/biz/ucsf-medical-center-at-mount-zion-san-francisco" ]  rule(linkextractor(allow=()),follow=true,callback="parsebusiness")  def parsebusiness(self, response):         business = business()         business['name'] = stripchars(response.xpath('//h1[@itemprop="name"]//text()').extract())         business['contactnumber'] = stripchars(response.xpath('//span[@itemprop="telephone"]//text()').extract())         business['address'] = stripchars(response.xpath('//li[@class="address"]//text()').extract())         yield business

what missing here? scrapy follow links

you not setting rules attribute of spider:

class yelpspider(crawlspider):     name = "yelp"     allowed_domains = ["www.yelp.com"]     start_urls = [         "http://www.yelp.com/search?find_desc=hotels&find_loc=san+francisco%2c+ca&ns=1",         "http://www.yelp.com/biz/ucsf-medical-center-at-mount-zion-san-francisco"     ]      rules = [         rule(linkextractor(allow=('.*')),follow=true,callback="parsebusiness")     ]

Comments

Post a Comment

Popular posts from this blog

javascript - Using jquery append to add option values into a select element not working -

i'm looking dynamically populate select element names of items stored array. for reason it's not working , i'm not sure why. html: <div class="col-md-4 col-md-offset-4"> <select class="select1"> </select> , <select class="select2"> </select> </div> note i'm using bootstrap layout. here's jquery i'm using try , populate ".select1" jquery: select: function() { $.each(state.greens, function(index, value) { $(".select1").append("<option value =' " + index + " '>" + state.greens[index].name + "</option>"); }); } note function 'select' stored within object called 'display'. state.greens name of array i'm trying access. so @ end of html page call display.select(); call function. no error messages showing in console. also, saw question: appending
Read more

Android soft keyboard reverts to default keyboard on orientation change -

i'm building android soft keyboard , can't seem fix bug - have arabic , qwerty keyboard , when rotate device on qwerty keyboard (or arabic shift), it's if program has "restarted" , becomes arabic keyboard without shift. the onsaveinstancestate(bundle savedinstancestate) not work because application not extend activity inputmethodservice . i put following in android manifest android:configchanges="keyboard|keyboardhidden|orientation" android:windowsoftinputmode="stateunchanged|adjustresize"> i tried using @override public void onconfigurationchanged(configuration newconfig) { super.onconfigurationchanged(newconfig); log.i(mydebug, "config changed " + currentkeyboard.equals(qwerty)); } however, currentkeyboard.equals(qwerty)) results false , made sure true before orientation change. any appreciated. i think applications fault. if application restarting on orientation change
Read more

jquery - javascript onscroll fade same class but with different div -

i new @ javascript , want learn new. there wrong code : why div fades @ same time. i want ask how fade different div s same class name.can me , explain me. thanks in advance. javascript: $(window).scroll(function(){ if ($( "div.fade" ).offset().top - $(window).scrolltop() <= 100){ $('.fade').addclass( "fade-in"); } else { $('.fade').removeclass("fade-in"); } }); here jsfiddle onscroll = function () { var scrolltop = $(this).scrolltop() ; $("div.fade").each(function(){ if(($(this).offset().top - scrolltop) <= 100){ $(this).toggleclass("fade-in"); } }) }
Read more