c++ - speed up a file scan crc algorithm -


i trying scan folder 1000 files of 1.4 gb in total takes 10 minutes on lot of computers, unacceptable. coded below , know how can speed up, if possible. note files never on 250mb, hence buffer. code run against each file in folder.

handle hfile = createfilea(szfilepath, generic_read, file_share_read, null, open_existing, 0, null); if (hfile != invalid_handle_value) {     dword dwsize = getfilesize(hfile, null);     if (dwsize != -1)     {         if (dwsize > 0)         {             dword dwchecksum = 0;             dword dwreads = 0;             dword dwreadfar = 0;             bool bfailed = false;              if (readfile(hfile, pfilebuffer, 250000000, &dwreads, null))             {                 dwchecksum = calculatechksum(pfilebuffer, dwreads);             }             else             {                 bfailed = true;             }         }     } }  dword calculatechksum(byte* pdata, int len, dword crc) {     dword  crc = crc;     const dword* current = (const dword*)pdata;      while (len >= 4)     {         crc = *current++ + crc;         len -= 4;     }      const byte* currentchar = (const byte*)current;     while (len-- > 0)         crc = *currentchar++ + crc;      return crc; }

your cpu idling when file partially read. speed up, read 1 mb chunks , decode them on fly. pass file_flag_sequential_scan createfile windows know prefetch next chunks.

also, make sure test optimized build. function simple enough single thread should keep disk i/o, might not case in debug builds.

btw, know that's not crc ? algorithm fails catch trivial modifications such endianness swapping.


Comments

Post a Comment

Popular posts from this blog

searchKeyword not working in AngularJS filter -

i writing app using angularjs on front end. want search through table word/field; 1 search box everything. according article here: http://hello-angularjs.appspot.com/searchtable can use filter: searchkeyword this. this code, , not working expected. <label>search: <input ng-model="searchkeyword"></label> <table ng-repeat="post in posts | orderby: sort | filter: searchkeyword"> <tr> <td> {{index(post)}} </td> <td> {{post.title}} </td> <td> {{post.author}} </td> <td> {{post.content}} </td> <td> {{post.date | date: "d/m/yyyy"}} </td> </tr> </table> what should do? thank you try this: controller $scope.posts = [ {title:'title one', author:'author one', content: 'content one'}, {title:'title two', author:'author two', content: 'cont...
Read more

sequelize.js - Sequelize: sort by enum cases -

i have model ( article ) field (the name type ) of type enum('source', 'translated') , return 1 article ordered type field. source articles should returned before translated articles. like: article.findone(order: [ { type: [ 'source', 'translated' ] } ]); unfortunately there appears no built-in solution sorting enums. instead have sort by case , couldn't find documentation on how sequelize. what conventional approach here? sorting case hard in sequelize, sort field (the second answer in linked post) should possible: order: sequelize.fn('field', sequelize.col('type'), 'source', 'translated') it's supported mysql far know though
Read more

user interface - how to replace an ongoing process of image capture from another process call over the same ImageLabel in python's GUI TKinter -

i trying use following code create interactive gui changes color-space(hsv, rbg, grayscale etc) on event of button click. display opencv video in tkinter using multiprocessing being new python having problems multiprocessing , attempts on making gui change it's color space on button clicks hangs entire system. on it's implementation highly appreciated. thankyou. below code: import cv2 pil import image,imagetk import tkinter tk import numpy np multiprocessing import process , queue def quit_it(root,process): root.destroy() process.terminate() def black_andwhite(root,process): process.terminate p=process(target=capture_image, args=(5,queue, )) p.start() root.after(0, func=lambda: update_all(root, image_label, queue)) def update_image(image_label, queue): frame = queue.get() = image.fromarray(frame) b = imagetk.photoimage(image=a) image_label.configure(image=b) image_label._image_cache = b root.update() def update_all(root...
Read more