python - Flatten a RDD in PySpark -


i trying process data using pyspark. following sample code:

rdd = sc.parallelize([[u'9', u'9', u'hf', u'63300001', u'in hf', u'03/09/2004', u'9', u'hf'], [u'10', u'10', u'hf', u'63300001', u'in hf', u'03/09/2004', u'9', u'hf']])   out = rdd.map(lambda l : (l[0:3],str(l[3]).zfill(8)[:4],l[4:]))  out.take(2)  [([u'9', u'9', u'hf'], '6330', [u'in hf', u'03/09/2004', u'9', u'hf']), ([u'10', u'10', u'hf'], '6330', [u'in hf', u'03/09/2004', u'9', u'hf'])]  expected output: [[u'9', u'9', u'hf', '6330', u'in hf', u'03/09/2004', u'9', u'hf'], [u'10', u'10', u'hf', '6330', u'in hf', u'03/09/2004', u'9', u'hf']]

is there method flatten rdd in spark?

you don't need spark specific here. should more enough:

out = rdd.map(lambda l : (l[0:3] + [str(l[3]).zfill(8)[:4]] + l[4:])

destructuring inside lambda more readable though. mean this:

rdd = sc.parallelize([(1, 2, 3), (4, 5, 6)]) rdd.map(lambda (x, y, z): (x, str(y).zfill(8), z))

Comments

Post a Comment

Popular posts from this blog

javascript - Using jquery append to add option values into a select element not working -

i'm looking dynamically populate select element names of items stored array. for reason it's not working , i'm not sure why. html: <div class="col-md-4 col-md-offset-4"> <select class="select1"> </select> , <select class="select2"> </select> </div> note i'm using bootstrap layout. here's jquery i'm using try , populate ".select1" jquery: select: function() { $.each(state.greens, function(index, value) { $(".select1").append("<option value =' " + index + " '>" + state.greens[index].name + "</option>"); }); } note function 'select' stored within object called 'display'. state.greens name of array i'm trying access. so @ end of html page call display.select(); call function. no error messages showing in console. also, saw question: appending
Read more

Android soft keyboard reverts to default keyboard on orientation change -

i'm building android soft keyboard , can't seem fix bug - have arabic , qwerty keyboard , when rotate device on qwerty keyboard (or arabic shift), it's if program has "restarted" , becomes arabic keyboard without shift. the onsaveinstancestate(bundle savedinstancestate) not work because application not extend activity inputmethodservice . i put following in android manifest android:configchanges="keyboard|keyboardhidden|orientation" android:windowsoftinputmode="stateunchanged|adjustresize"> i tried using @override public void onconfigurationchanged(configuration newconfig) { super.onconfigurationchanged(newconfig); log.i(mydebug, "config changed " + currentkeyboard.equals(qwerty)); } however, currentkeyboard.equals(qwerty)) results false , made sure true before orientation change. any appreciated. i think applications fault. if application restarting on orientation change
Read more

Rendering JButton to get the JCheckBox behavior in a JTable by using images does not update my table -

i rendering jbutton image on simulate jcheckbox behavior. why don't let jtable render jcheckbox? because need bigger checkboxes , there's no way can resize them (as far know). so, have own table cell renderer , editor. both renders jbutton , checkbox images on them. trying change image checked image unchecked image (or vice versa) , update cell value (boolean true or false). however, reason, getcelleditorvalue() not update table. so, release mouse, value goes original one. any appreciated! lot! public class tableexample extends jframe { jscrollpane pane; jpanel panel; jtable table; object[][] data; mytablemodel tm; renderer buttoncolumn; public tableexample(object[][] data) throws ioexception { string[] columns = {"column#1", "column#2", "column#3", "column#4", "column#5", "column#6"}; this.data = data; this.setpreferredsize(new dimension(700, 500));
Read more