High R memory usage with only small objects when web scraping -


i'm scraping website , calling scraping function for-loop. around iteration 4,000 of loop, computer warned me rstudio using memory. after breaking loop escape key don't see large objects in r environment.

i tried tips on these two posts don't reveal cause. when call mem_used() pryr package get:

2.3 gb

which aligns windows task manager said initially. said 2.3 gb, dropped 1.7 gb ten minutes after terminating loop , 1.2 gb twenty minutes after loop. mem_used() continues 2.3 gb.

but r objects small, according lsos() function in first post linked above:

> lsos()                        type     size  rows columns all_raw              tbl_df 17390736 89485      12 all_clean            tbl_df 14693336 89485      15 all_no_pavs          tbl_df 14180576 86050      15 all_no_dupe_names    tbl_df 13346256 79646      15 sample_in            tbl_df  1917128  9240      15 testdat              tbl_df  1188152  5402      15 username_res         tbl_df   792936  4091      14 getusername        function   151992    na      na dupe_names           tbl_df   132040  2802       3 time_per_iteration  numeric    65408  4073      na

that says largest object 17 mb, not close 2.3 gb. how can find culprit of memory use , fix it? there in loop gradually tying memory?

here reproducible test example, scraping imdb.com:

library(rvest) # rvest 0.2.0 needed produce error, fixed in 0.3.0 library(stringr) library(dplyr)  search_list <- make.names(names(precip))   scrape_top_titles <- function(search_string) {   urltoget <- paste0("http://www.imdb.com/find?q=", search_string)   print(urltoget)   page <- html(urltoget)   top_3_hits <- page %>%     html_nodes(xpath='//td[contains(@class, "result_text")]') %>%     html_text %>%     str_trim %>%     .[1:3]    result <- list(search_term = search_string, hit_1 = top_3_hits[1], hit_2 = top_3_hits[2], hit_3 = top_3_hits[3], page_length = nchar(page %>% html_text))  result }  # these 70 scrapes start filling memory scrapes <- bind_rows(   lapply(search_list, function(x) {data.frame(scrape_top_titles(x), stringsasfactors = false)}) )  # more dramatic memory filling, scrape 770 pages instead longer_list <- as.vector(outer(search_list, names(mtcars), paste, sep="_")) long_scrapes <- bind_rows(   lapply(longer_list, function(x) {data.frame(scrape_top_titles(x), stringsasfactors = false)}) )

update: looks memory leak xml package called rvest, similar described in this question , manifests in this other question. 0.3.0 release of rvest calls xml2 package instead , has solved memory leak, above code no longer generates error unless old version of rvest used.

i'm still looking answer describe going on here: can explain "memory leak"? problem fixed i'm curious happening.


Comments

Post a Comment

Popular posts from this blog

searchKeyword not working in AngularJS filter -

i writing app using angularjs on front end. want search through table word/field; 1 search box everything. according article here: http://hello-angularjs.appspot.com/searchtable can use filter: searchkeyword this. this code, , not working expected. <label>search: <input ng-model="searchkeyword"></label> <table ng-repeat="post in posts | orderby: sort | filter: searchkeyword"> <tr> <td> {{index(post)}} </td> <td> {{post.title}} </td> <td> {{post.author}} </td> <td> {{post.content}} </td> <td> {{post.date | date: "d/m/yyyy"}} </td> </tr> </table> what should do? thank you try this: controller $scope.posts = [ {title:'title one', author:'author one', content: 'content one'}, {title:'title two', author:'author two', content: 'cont...
Read more

sequelize.js - Sequelize: sort by enum cases -

i have model ( article ) field (the name type ) of type enum('source', 'translated') , return 1 article ordered type field. source articles should returned before translated articles. like: article.findone(order: [ { type: [ 'source', 'translated' ] } ]); unfortunately there appears no built-in solution sorting enums. instead have sort by case , couldn't find documentation on how sequelize. what conventional approach here? sorting case hard in sequelize, sort field (the second answer in linked post) should possible: order: sequelize.fn('field', sequelize.col('type'), 'source', 'translated') it's supported mysql far know though
Read more

user interface - how to replace an ongoing process of image capture from another process call over the same ImageLabel in python's GUI TKinter -

i trying use following code create interactive gui changes color-space(hsv, rbg, grayscale etc) on event of button click. display opencv video in tkinter using multiprocessing being new python having problems multiprocessing , attempts on making gui change it's color space on button clicks hangs entire system. on it's implementation highly appreciated. thankyou. below code: import cv2 pil import image,imagetk import tkinter tk import numpy np multiprocessing import process , queue def quit_it(root,process): root.destroy() process.terminate() def black_andwhite(root,process): process.terminate p=process(target=capture_image, args=(5,queue, )) p.start() root.after(0, func=lambda: update_all(root, image_label, queue)) def update_image(image_label, queue): frame = queue.get() = image.fromarray(frame) b = imagetk.photoimage(image=a) image_label.configure(image=b) image_label._image_cache = b root.update() def update_all(root...
Read more