How to connect to Amazon Redshift or other DB's in Apache Spark? -


i'm trying connect amazon redshift via spark, can join data have on s3 data on our rs cluster. found spartan documentation here capability of connecting jdbc:

https://spark.apache.org/docs/1.3.1/sql-programming-guide.html#jdbc-to-other-databases

the load command seems straightforward (although don't know how enter aws credentials here, maybe in options?).

df = sqlcontext.load(source="jdbc", url="jdbc:postgresql:dbserver", dbtable="schema.tablename")

and i'm not entirely sure how deal spark_classpath variable. i'm running spark locally through ipython notebook (as part of spark distribution). define spark loads it?

anyway, now, when try running these commands, bunch of undecipherable errors, i'm kind of stuck now. or pointers detailed tutorials appreciated.

it turns out need username/pwd access redshift in spark, , done follows (using python api):

from pyspark.sql import sqlcontext sqlcontext = sqlcontext(sc) df = sqlcontext.load(source="jdbc",                       url="jdbc:postgresql://host:port/dbserver?user=yourusername&password=secret",                       dbtable="schema.table")

hope helps someone!


Comments

Post a Comment

Popular posts from this blog

javascript - Using jquery append to add option values into a select element not working -

i'm looking dynamically populate select element names of items stored array. for reason it's not working , i'm not sure why. html: <div class="col-md-4 col-md-offset-4"> <select class="select1"> </select> , <select class="select2"> </select> </div> note i'm using bootstrap layout. here's jquery i'm using try , populate ".select1" jquery: select: function() { $.each(state.greens, function(index, value) { $(".select1").append("<option value =' " + index + " '>" + state.greens[index].name + "</option>"); }); } note function 'select' stored within object called 'display'. state.greens name of array i'm trying access. so @ end of html page call display.select(); call function. no error messages showing in console. also, saw question: appending
Read more

Android soft keyboard reverts to default keyboard on orientation change -

i'm building android soft keyboard , can't seem fix bug - have arabic , qwerty keyboard , when rotate device on qwerty keyboard (or arabic shift), it's if program has "restarted" , becomes arabic keyboard without shift. the onsaveinstancestate(bundle savedinstancestate) not work because application not extend activity inputmethodservice . i put following in android manifest android:configchanges="keyboard|keyboardhidden|orientation" android:windowsoftinputmode="stateunchanged|adjustresize"> i tried using @override public void onconfigurationchanged(configuration newconfig) { super.onconfigurationchanged(newconfig); log.i(mydebug, "config changed " + currentkeyboard.equals(qwerty)); } however, currentkeyboard.equals(qwerty)) results false , made sure true before orientation change. any appreciated. i think applications fault. if application restarting on orientation change
Read more

jquery - javascript onscroll fade same class but with different div -

i new @ javascript , want learn new. there wrong code : why div fades @ same time. i want ask how fade different div s same class name.can me , explain me. thanks in advance. javascript: $(window).scroll(function(){ if ($( "div.fade" ).offset().top - $(window).scrolltop() <= 100){ $('.fade').addclass( "fade-in"); } else { $('.fade').removeclass("fade-in"); } }); here jsfiddle onscroll = function () { var scrolltop = $(this).scrolltop() ; $("div.fade").each(function(){ if(($(this).offset().top - scrolltop) <= 100){ $(this).toggleclass("fade-in"); } }) }
Read more