How do I use Dataflow to calculate the top unique value results per key? -


i'm relatively new dataflow , programming model , struggling problem requires calculating top 10 weeks customer has highest spend. apologise if seems silly question.

the data have consists of customer ids use key , few million records containing timestamp , spend value.

i've created extract method looks (excluding logging , date formatter). receives bigquery table row extract customer id, spend , timestamp week number:

static class extractspend extends dofn<tablerow, kv<string, spendbyweek>> {     private static final long serialversionuid = 0;      @override     public void processelement(processcontext c) {         string custid = (string) row.get("customerid");         localdatetime date = localdatetime.parse((string) row.get("timestamp"), datetimeformatter);          weekfields weekfields = weekfields.of(locale.getdefault());         int weeknumber = date.get(weekfields.weekofweekbasedyear());          double spend = (double) row.get("spend");          spendbyweek spendbyweek = new spendbyweek(weeknumber, spend.doublevalue());         c.output(kv.of(custid, spendbyweek));     } }

but can't figure out how take output , group in such way can add spend values per customer id , week, sort them , output pcollection<string, list<double>> of each customer , top 10 weekly spend values.

would able me out this, please?

if want accomplish using grouping, you'd need first group customer id , week compute sum move week value , regroup customer id computing top. can using windowing rather putting week in key. see end details on doing that.

once you've done that, have pcollection<kv<string, spendbyweek>> each week occurs once given key. can determine top spendbyweek each given user-id defining comparator<spendbyweek> implements serializable , using top.perkey().

computing spend-per-week-per-user windows

as mentioned @ top can use windowing computing spend-per-week.

  1. write dofn similar extractspend takes tablerow , outputs kv individual rows of spend keyed customer id , output outputs using outputwithtimestamp.
  2. then apply windowing transformation, such fixedwindows take divide events windows of specified size. in case, want fixedwindows.of(duration.standardweeks(1)) or calendarwindows.weeks(...).
  3. then apply transform such sum.doublesperkey().

at point, you'll have pcollection contains per-week-window kv<string, double> each entry total spend key in week.

  1. then can run dofn takes hourly windows , moves information value (so have kv<string, spendandweek>)
  2. apply window.into switch globalwindows
  3. and apply top.perkey operation described above.

Comments

Post a Comment

Popular posts from this blog

javascript - Using jquery append to add option values into a select element not working -

i'm looking dynamically populate select element names of items stored array. for reason it's not working , i'm not sure why. html: <div class="col-md-4 col-md-offset-4"> <select class="select1"> </select> , <select class="select2"> </select> </div> note i'm using bootstrap layout. here's jquery i'm using try , populate ".select1" jquery: select: function() { $.each(state.greens, function(index, value) { $(".select1").append("<option value =' " + index + " '>" + state.greens[index].name + "</option>"); }); } note function 'select' stored within object called 'display'. state.greens name of array i'm trying access. so @ end of html page call display.select(); call function. no error messages showing in console. also, saw question: appending
Read more

Android soft keyboard reverts to default keyboard on orientation change -

i'm building android soft keyboard , can't seem fix bug - have arabic , qwerty keyboard , when rotate device on qwerty keyboard (or arabic shift), it's if program has "restarted" , becomes arabic keyboard without shift. the onsaveinstancestate(bundle savedinstancestate) not work because application not extend activity inputmethodservice . i put following in android manifest android:configchanges="keyboard|keyboardhidden|orientation" android:windowsoftinputmode="stateunchanged|adjustresize"> i tried using @override public void onconfigurationchanged(configuration newconfig) { super.onconfigurationchanged(newconfig); log.i(mydebug, "config changed " + currentkeyboard.equals(qwerty)); } however, currentkeyboard.equals(qwerty)) results false , made sure true before orientation change. any appreciated. i think applications fault. if application restarting on orientation change
Read more

Rendering JButton to get the JCheckBox behavior in a JTable by using images does not update my table -

i rendering jbutton image on simulate jcheckbox behavior. why don't let jtable render jcheckbox? because need bigger checkboxes , there's no way can resize them (as far know). so, have own table cell renderer , editor. both renders jbutton , checkbox images on them. trying change image checked image unchecked image (or vice versa) , update cell value (boolean true or false). however, reason, getcelleditorvalue() not update table. so, release mouse, value goes original one. any appreciated! lot! public class tableexample extends jframe { jscrollpane pane; jpanel panel; jtable table; object[][] data; mytablemodel tm; renderer buttoncolumn; public tableexample(object[][] data) throws ioexception { string[] columns = {"column#1", "column#2", "column#3", "column#4", "column#5", "column#6"}; this.data = data; this.setpreferredsize(new dimension(700, 500));
Read more