apache spark - How to Transpose an rdd in pyspark (list is not a matrix) -


i have rdd list of strings ['abc', 'ccd', 'xyz'...'axd']

when "print rdd.take(2), expecting return me ['abc', 'ccd'], instead gives me everything. new spark or python please forgive me if dumb question. there way transpose list rows?

eventually need convert dataframe , insert hive table.

here pice of code

domainsrdd = zonerdd.reducebykey(lambda x,y: x + ' ' + y).map(lambda a:     (a[0], a[1].split(' ')))  print domainsrdd.take(2)  [(u'cool', [u'shirtmaker.cool', u'videocandy.cool', u'the-happy-factory.cool', u'vic.cool', u'atl.cool',...... u'booze.cool'])]  def sampler(l, tldvar):     tld = l[0]     domain_data = l[1]     domains = []     ct = tldvar.value[tld]     item in domain_data:         domains.extend([item])         if len(domains) == ct:             break     return domains  domainslist = domainsrdd.map(lambda l: sampler(l, tldvar))  print domainslist.take(2) # still returns  [[u'shirtmaker.cool', u'videocandy.cool', u'the-happy-factory.cool',...])]

long story short, trying loop thru set of domains grouped tld's , producing sample of domain names, tldvar dictionary has set of domains need return specific tld. tld = com, net, org etc!

domainslist of type rdd[array[string]], when take, you're going array[array[string]]. which, in case filled arrays never limited down based on saying (len(domains) == ct never true)


Comments

Post a Comment

Popular posts from this blog

searchKeyword not working in AngularJS filter -

i writing app using angularjs on front end. want search through table word/field; 1 search box everything. according article here: http://hello-angularjs.appspot.com/searchtable can use filter: searchkeyword this. this code, , not working expected. <label>search: <input ng-model="searchkeyword"></label> <table ng-repeat="post in posts | orderby: sort | filter: searchkeyword"> <tr> <td> {{index(post)}} </td> <td> {{post.title}} </td> <td> {{post.author}} </td> <td> {{post.content}} </td> <td> {{post.date | date: "d/m/yyyy"}} </td> </tr> </table> what should do? thank you try this: controller $scope.posts = [ {title:'title one', author:'author one', content: 'content one'}, {title:'title two', author:'author two', content: 'cont...
Read more

sequelize.js - Sequelize: sort by enum cases -

i have model ( article ) field (the name type ) of type enum('source', 'translated') , return 1 article ordered type field. source articles should returned before translated articles. like: article.findone(order: [ { type: [ 'source', 'translated' ] } ]); unfortunately there appears no built-in solution sorting enums. instead have sort by case , couldn't find documentation on how sequelize. what conventional approach here? sorting case hard in sequelize, sort field (the second answer in linked post) should possible: order: sequelize.fn('field', sequelize.col('type'), 'source', 'translated') it's supported mysql far know though
Read more

user interface - how to replace an ongoing process of image capture from another process call over the same ImageLabel in python's GUI TKinter -

i trying use following code create interactive gui changes color-space(hsv, rbg, grayscale etc) on event of button click. display opencv video in tkinter using multiprocessing being new python having problems multiprocessing , attempts on making gui change it's color space on button clicks hangs entire system. on it's implementation highly appreciated. thankyou. below code: import cv2 pil import image,imagetk import tkinter tk import numpy np multiprocessing import process , queue def quit_it(root,process): root.destroy() process.terminate() def black_andwhite(root,process): process.terminate p=process(target=capture_image, args=(5,queue, )) p.start() root.after(0, func=lambda: update_all(root, image_label, queue)) def update_image(image_label, queue): frame = queue.get() = image.fromarray(frame) b = imagetk.photoimage(image=a) image_label.configure(image=b) image_label._image_cache = b root.update() def update_all(root...
Read more