python - CPickle performance mystery -


i have python code have continuous loop pickle load inside. have 200 pickle files in loop each 80 mb each on ssd drive.

when ran code experienced performance of pickle load fluctuates continuously: @ 0,2s @ times "pauses" 4-6s debasing overall benchmark of process.

what problem?

def unpickle(filename):     fo = open(filename, 'r')     contents = cpickle.load(fo)     fo.close()     return contents  xd in self.x:     tt = time()                     xdf = unpickle(xd)     tt = time() - tt     print tt

out:

1.87527704239 4.30886101723 0.259668111801 0.234542131424 0.228765964508 0.214528799057 0.213661909103 0.215914011002 0.217473983765 0.225739002228

the way created pickle files: have pandas dataframe column: 'name','source','level','image','path','is_train'. main data regarding size 'image'. pickle with:

def pickle(filename, data):     open(filename, 'w') fo:         cpickle.dump(data, fo, protocol=cpickle.highest_protocol)

your question terribly unclear (in particular, should giving enough information reproduce test case ourselves), feels gc pauses or memory defragmentation.

pickle terribly inefficient format, , processing 16 gigabytes' worth of bound cause serious memory thrashing.


Comments

Post a Comment

Popular posts from this blog

searchKeyword not working in AngularJS filter -

i writing app using angularjs on front end. want search through table word/field; 1 search box everything. according article here: http://hello-angularjs.appspot.com/searchtable can use filter: searchkeyword this. this code, , not working expected. <label>search: <input ng-model="searchkeyword"></label> <table ng-repeat="post in posts | orderby: sort | filter: searchkeyword"> <tr> <td> {{index(post)}} </td> <td> {{post.title}} </td> <td> {{post.author}} </td> <td> {{post.content}} </td> <td> {{post.date | date: "d/m/yyyy"}} </td> </tr> </table> what should do? thank you try this: controller $scope.posts = [ {title:'title one', author:'author one', content: 'content one'}, {title:'title two', author:'author two', content: 'cont...
Read more

sequelize.js - Sequelize: sort by enum cases -

i have model ( article ) field (the name type ) of type enum('source', 'translated') , return 1 article ordered type field. source articles should returned before translated articles. like: article.findone(order: [ { type: [ 'source', 'translated' ] } ]); unfortunately there appears no built-in solution sorting enums. instead have sort by case , couldn't find documentation on how sequelize. what conventional approach here? sorting case hard in sequelize, sort field (the second answer in linked post) should possible: order: sequelize.fn('field', sequelize.col('type'), 'source', 'translated') it's supported mysql far know though
Read more

user interface - how to replace an ongoing process of image capture from another process call over the same ImageLabel in python's GUI TKinter -

i trying use following code create interactive gui changes color-space(hsv, rbg, grayscale etc) on event of button click. display opencv video in tkinter using multiprocessing being new python having problems multiprocessing , attempts on making gui change it's color space on button clicks hangs entire system. on it's implementation highly appreciated. thankyou. below code: import cv2 pil import image,imagetk import tkinter tk import numpy np multiprocessing import process , queue def quit_it(root,process): root.destroy() process.terminate() def black_andwhite(root,process): process.terminate p=process(target=capture_image, args=(5,queue, )) p.start() root.after(0, func=lambda: update_all(root, image_label, queue)) def update_image(image_label, queue): frame = queue.get() = image.fromarray(frame) b = imagetk.photoimage(image=a) image_label.configure(image=b) image_label._image_cache = b root.update() def update_all(root...
Read more