Scraping external website within main project with python scrapy framework -


i have been looking better way scrape external website main source website. better explain let me use example yelp.com explain trying (though target not yelp).

  1. i scrape title , address
  2. visit link titles leads to company website
  3. i extract emails source code of main website. (i know difficult, not crawling pages assuming site have contact in url e.g site.com/contact.php)
  4. the point while scraping yelp , storing data in field, want external data companies main website.

below code can't figure out how using scrapy.

# -*- coding: utf-8 -*- import scrapy scrapy.contrib.spiders import crawlspider, rule scrapy.linkextractors import linkextractor comb.items import combitem, siteitem  class comberspider(crawlspider):     name = "comber"     allowed_domains = ["example.com"]     query = 'shoe'     page = 'http://www.example.com/corp/' + query + '/1.html'     start_urls = (         page,     )     rules = (rule(linkextractor(allow=(r'corp/.+/\d+\.html'), restrict_xpaths=("//a[@class='next']")),                   callback="parse_items", follow=true),              )       def parse_items(self, response):          sel in response.xpath("//div[@class='item-main']"):             item = combitem()             item['company_name'] = sel.xpath("h2[@class='title']/a/text()").extract()             item['contact_url'] = sel.xpath("div[@class='company']/a/@href").extract()[0]             item['gold_supplier'] = sel.xpath("div[@class='item-title']/a/@title").extract()[0]             company_details = sel.xpath("div[@class='attrs']/div[@class='attr']/span['name']/text()").extract()              item = self.parse_meta(sel, item, company_details)             request = scrapy.request(item['contact_url'], callback=self.parse_site)             request.meta['item'] = item              yield request      def parse_meta(self, sel, item, company_details):          if (company_details):             if "products:" in company_details:                 item['products'] = sel.xpath("./div[@class='value']//text()").extract()             if "country/region:" in company_details:                  item['country'] = sel.xpath("./div[@class='right']"                                         + "/span[@data-coun]/text()").extract()             if "revenue:" in company_details:                 item['revenue'] = sel.xpath("./div[@class='right']/"                                         + "span[@data-reve]/text()").extract()             if "markets:" in company_details:                 item['markets'] = sel.xpath("./div[@class='value']/span[@data-mark]/text()").extract()         return item      def parse_site(self, response):         item = response.meta['item']         # value of item['websites'] http://target-company.com, http://any-other-website.com         # aim jump http://company.com , scrap data it's contact page ,         # store item item['emails'] = [info@company.com, sales@company.com]          # please how can done in same project         # thing can think of store item['websites'] , other values of item , make project         # still not work because of allowed_domains , start_urls           item['websites'] = response.xpath("//div[@class='company-contact-information']/table/tr/td/a/@href").extract()         print(item)         print('*'* 50)         yield item    """  scrapy.item import item, field   class combitem(item):     company_name = field()     main_products = field()     contact_url = field()     revenue = field()     gold_supplier = field()     country = field()     markets= field()     product_home = field()     websites = field()     """     #emails = field() not implemented because emails need extracted websites different start_url

when issue request, passing dont_filter=true turn off offsitemiddleware , url not filtered allowed_domains:

if request has dont_filter attribute set, offsite middleware allow request if domain not listed in allowed domains.


Comments

Post a Comment

Popular posts from this blog

javascript - Using jquery append to add option values into a select element not working -

i'm looking dynamically populate select element names of items stored array. for reason it's not working , i'm not sure why. html: <div class="col-md-4 col-md-offset-4"> <select class="select1"> </select> , <select class="select2"> </select> </div> note i'm using bootstrap layout. here's jquery i'm using try , populate ".select1" jquery: select: function() { $.each(state.greens, function(index, value) { $(".select1").append("<option value =' " + index + " '>" + state.greens[index].name + "</option>"); }); } note function 'select' stored within object called 'display'. state.greens name of array i'm trying access. so @ end of html page call display.select(); call function. no error messages showing in console. also, saw question: appending
Read more

Android soft keyboard reverts to default keyboard on orientation change -

i'm building android soft keyboard , can't seem fix bug - have arabic , qwerty keyboard , when rotate device on qwerty keyboard (or arabic shift), it's if program has "restarted" , becomes arabic keyboard without shift. the onsaveinstancestate(bundle savedinstancestate) not work because application not extend activity inputmethodservice . i put following in android manifest android:configchanges="keyboard|keyboardhidden|orientation" android:windowsoftinputmode="stateunchanged|adjustresize"> i tried using @override public void onconfigurationchanged(configuration newconfig) { super.onconfigurationchanged(newconfig); log.i(mydebug, "config changed " + currentkeyboard.equals(qwerty)); } however, currentkeyboard.equals(qwerty)) results false , made sure true before orientation change. any appreciated. i think applications fault. if application restarting on orientation change
Read more

jquery - javascript onscroll fade same class but with different div -

i new @ javascript , want learn new. there wrong code : why div fades @ same time. i want ask how fade different div s same class name.can me , explain me. thanks in advance. javascript: $(window).scroll(function(){ if ($( "div.fade" ).offset().top - $(window).scrolltop() <= 100){ $('.fade').addclass( "fade-in"); } else { $('.fade').removeclass("fade-in"); } }); here jsfiddle onscroll = function () { var scrolltop = $(this).scrolltop() ; $("div.fade").each(function(){ if(($(this).offset().top - scrolltop) <= 100){ $(this).toggleclass("fade-in"); } }) }
Read more