Spark off heap memory leak on Yarn with Kafka direct stream -


i running spark streaming 1.4.0 on yarn (apache distribution 2.6.0) java 1.8.0_45 , kafka direct stream. using spark scala 2.11 support.

the issue seeing both driver , executor containers gradually increasing physical memory usage till point yarn container kill it. have configured upto 192m heap , 384 off heap space in driver runs out of it

the heap memory appears fine regular gc cycles. there no outoffmemory encountered ever in such runs

infact not generating traffic on kafka queues still happens. here code using

object simplesparkstreaming extends app {  val conf = new sparkconf() val ssc = new streamingcontext(conf,seconds(conf.getlong("spark.batch.window.size",1l))); ssc.checkpoint("checkpoint") val topics = set(conf.get("spark.kafka.topic.name"));      val kafkaparams = map[string, string]("metadata.broker.list" -> conf.get("spark.kafka.broker.list"))             val kafkastream = kafkautils.createdirectstream[string,string,stringdecoder,stringdecoder](ssc, kafkaparams, topics)             kafkastream.foreachrdd(rdd => {                 rdd.foreach(x => {                     println(x._2)                 })              })     kafkastream.print()             ssc.start()               ssc.awaittermination()  }

i running on centos 7. command used spark submit following

./bin/spark-submit --class com.rasa.cloud.prototype.spark.simplesparkstreaming \ --conf spark.yarn.executor.memoryoverhead=256 \ --conf spark.yarn.driver.memoryoverhead=384 \ --conf spark.kafka.topic.name=test \ --conf spark.kafka.broker.list=172.31.45.218:9092 \ --conf spark.batch.window.size=1 \ --conf spark.app.name="simple spark kafka application" \ --master yarn-cluster \ --num-executors 1 \ --driver-memory 192m \ --executor-memory 128m \ --executor-cores 1 \ /home/centos/spark-poc/target/lib/spark-streaming-prototype-0.0.1-snapshot.jar

any appreciated

regards,

apoorva

try increasing executor cores. in example core dedicated consuming streaming data, leaving no cores process in incoming data.


Comments

Post a Comment

Popular posts from this blog

searchKeyword not working in AngularJS filter -

i writing app using angularjs on front end. want search through table word/field; 1 search box everything. according article here: http://hello-angularjs.appspot.com/searchtable can use filter: searchkeyword this. this code, , not working expected. <label>search: <input ng-model="searchkeyword"></label> <table ng-repeat="post in posts | orderby: sort | filter: searchkeyword"> <tr> <td> {{index(post)}} </td> <td> {{post.title}} </td> <td> {{post.author}} </td> <td> {{post.content}} </td> <td> {{post.date | date: "d/m/yyyy"}} </td> </tr> </table> what should do? thank you try this: controller $scope.posts = [ {title:'title one', author:'author one', content: 'content one'}, {title:'title two', author:'author two', content: 'cont...
Read more

sequelize.js - Sequelize: sort by enum cases -

i have model ( article ) field (the name type ) of type enum('source', 'translated') , return 1 article ordered type field. source articles should returned before translated articles. like: article.findone(order: [ { type: [ 'source', 'translated' ] } ]); unfortunately there appears no built-in solution sorting enums. instead have sort by case , couldn't find documentation on how sequelize. what conventional approach here? sorting case hard in sequelize, sort field (the second answer in linked post) should possible: order: sequelize.fn('field', sequelize.col('type'), 'source', 'translated') it's supported mysql far know though
Read more

user interface - how to replace an ongoing process of image capture from another process call over the same ImageLabel in python's GUI TKinter -

i trying use following code create interactive gui changes color-space(hsv, rbg, grayscale etc) on event of button click. display opencv video in tkinter using multiprocessing being new python having problems multiprocessing , attempts on making gui change it's color space on button clicks hangs entire system. on it's implementation highly appreciated. thankyou. below code: import cv2 pil import image,imagetk import tkinter tk import numpy np multiprocessing import process , queue def quit_it(root,process): root.destroy() process.terminate() def black_andwhite(root,process): process.terminate p=process(target=capture_image, args=(5,queue, )) p.start() root.after(0, func=lambda: update_all(root, image_label, queue)) def update_image(image_label, queue): frame = queue.get() = image.fromarray(frame) b = imagetk.photoimage(image=a) image_label.configure(image=b) image_label._image_cache = b root.update() def update_all(root...
Read more