MongoDB $or / $and command optimisations -


i'm trying figure out why below mongodb aggregate query takes long, when there's 18 $or conditions.

i'm able remove any one of conditions (making 17), , query completed in less 100ms, 18 jumps ~20 seconds.

i understand if there specific condition causing issue, can't head around why can remove of them , response time drops down ~100ms.

i've checked memory useage while commands running , takes 15% of boxes total memory. understanding there's no specified memory limit mongodb?

also, query brings expected result.

any help/insight appreciated!

thanks in advance!

db.getcollection('messages').aggregate([         {             $match: {                 $or: [                     {"$and":[{"channel_id":8},{"created_date":{"$gt":0}}]},                     {"$and":[{"channel_id":64},{"created_date":{"$gt":0}}]},                     {"$and":[{"channel_id":9},{"created_date":{"$gt":0}}]},                     {"$and":[{"channel_id":77},{"created_date":{"$gt":0}}]},                     {"$and":[{"channel_id":76},{"created_date":{"$gt":0}}]},                     {"$and":[{"channel_id":6},{"created_date":{"$gt":0}}]},                     {"$and":[{"channel_id":62},{"created_date":{"$gt":0}}]},                     {"$and":[{"channel_id":63},{"created_date":{"$gt":0}}]},                     {"$and":[{"channel_id":84},{"created_date":{"$gt":0}}]},                     {"$and":[{"channel_id":99},{"created_date":{"$gt":1436793531100}}]},                     {"$and":[{"channel_id":1},{"created_date":{"$gt":0}}]},                     {"$and":[{"channel_id":93},{"created_date":{"$gt":0}}]},                     {"$and":[{"channel_id":148},{"created_date":{"$gt":0}}]},                     {"$and":[{"channel_id":114},{"created_date":{"$gt":0}}]},                     {"$and":[{"channel_id":145},{"created_date":{"$gt":0}}]},                     {"$and":[{"channel_id":174},{"created_date":{"$gt":0}}]},                     {"$and":[{"channel_id":175},{"created_date":{"$gt":0}}]},                     {"$and":[{"channel_id":146},{"created_date":{"$gt":0}}]}                     ]                 }         },         {             $group: { _id : '$channel_id', count: { $sum: 1 }}         }     ]);

edit 1:

i've tried removing redundant $and statements sammaye pointed out, same results.

i've noticed can add 19th $or condition , returns in under 100ms.

also i'm on version 2.6.7 of mongodb.

i have indexes on both channel_id , created_date fields.

i tried add explain option aggregate command, ishamael suggested, doesn't seem have effect on output. adding .hint( { channel_id: 1 } ) or .hint( { created_date: 1 } ) onto end of query doesn't seem work , end typeerror.

here's updated query - explain turned on:

db.getcollection('messages').aggregate([     {         $match: {             $or: [                 {"channel_id":8, "created_date":{"$gt":0}},                 {"channel_id":64, "created_date":{"$gt":0}},                 {"channel_id":9, "created_date":{"$gt":0}},                 {"channel_id":77, "created_date":{"$gt":0}},                 {"channel_id":76, "created_date":{"$gt":0}},                 {"channel_id":6, "created_date":{"$gt":0}},                 {"channel_id":62, "created_date":{"$gt":0}},                 {"channel_id":63, "created_date":{"$gt":0}},                 {"channel_id":84, "created_date":{"$gt":0}},                 {"channel_id":99, "created_date":{"$gt":1436793531100}},                 {"channel_id":1, "created_date":{"$gt":0}},                 {"channel_id":93, "created_date":{"$gt":0}},                 {"channel_id":148, "created_date":{"$gt":0}},                 {"channel_id":114, "created_date":{"$gt":0}},                 {"channel_id":145, "created_date":{"$gt":0}},                 {"channel_id":174, "created_date":{"$gt":0}},                 {"channel_id":175, "created_date":{"$gt":0}},                 {"channel_id":146, "created_date":{"$gt":0}}                 ]             }     },     {         $group: { _id : '$channel_id', count: { $sum: 1 }}     } ], { explain: true });

and result:

{ "result" : [      {         "_id" : 175,         "count" : 4.0000000000000000     },      {         "_id" : 146,         "count" : 1.0000000000000000     },      {         "_id" : 145,         "count" : 3.0000000000000000     },      {         "_id" : 8,         "count" : 17.0000000000000000     },      {         "_id" : 148,         "count" : 1.0000000000000000     },      {         "_id" : 84,         "count" : 16.0000000000000000     },      {         "_id" : 1,         "count" : 5067.0000000000000000     },      {         "_id" : 76,         "count" : 15.0000000000000000     },      {         "_id" : 77,         "count" : 2.0000000000000000     },      {         "_id" : 174,         "count" : 8.0000000000000000     },      {         "_id" : 64,         "count" : 2.0000000000000000     },      {         "_id" : 93,         "count" : 7.0000000000000000     },      {         "_id" : 6,         "count" : 93.0000000000000000     },      {         "_id" : 114,         "count" : 6.0000000000000000     } ], "ok" : 1.0000000000000000 }

still not 100% sure if bug in 2.6.7 upgrading latest version (3.0.4 @ time of posting) seemed fix problems.


Comments

Post a Comment

Popular posts from this blog

searchKeyword not working in AngularJS filter -

i writing app using angularjs on front end. want search through table word/field; 1 search box everything. according article here: http://hello-angularjs.appspot.com/searchtable can use filter: searchkeyword this. this code, , not working expected. <label>search: <input ng-model="searchkeyword"></label> <table ng-repeat="post in posts | orderby: sort | filter: searchkeyword"> <tr> <td> {{index(post)}} </td> <td> {{post.title}} </td> <td> {{post.author}} </td> <td> {{post.content}} </td> <td> {{post.date | date: "d/m/yyyy"}} </td> </tr> </table> what should do? thank you try this: controller $scope.posts = [ {title:'title one', author:'author one', content: 'content one'}, {title:'title two', author:'author two', content: 'cont...
Read more

sequelize.js - Sequelize: sort by enum cases -

i have model ( article ) field (the name type ) of type enum('source', 'translated') , return 1 article ordered type field. source articles should returned before translated articles. like: article.findone(order: [ { type: [ 'source', 'translated' ] } ]); unfortunately there appears no built-in solution sorting enums. instead have sort by case , couldn't find documentation on how sequelize. what conventional approach here? sorting case hard in sequelize, sort field (the second answer in linked post) should possible: order: sequelize.fn('field', sequelize.col('type'), 'source', 'translated') it's supported mysql far know though
Read more

user interface - how to replace an ongoing process of image capture from another process call over the same ImageLabel in python's GUI TKinter -

i trying use following code create interactive gui changes color-space(hsv, rbg, grayscale etc) on event of button click. display opencv video in tkinter using multiprocessing being new python having problems multiprocessing , attempts on making gui change it's color space on button clicks hangs entire system. on it's implementation highly appreciated. thankyou. below code: import cv2 pil import image,imagetk import tkinter tk import numpy np multiprocessing import process , queue def quit_it(root,process): root.destroy() process.terminate() def black_andwhite(root,process): process.terminate p=process(target=capture_image, args=(5,queue, )) p.start() root.after(0, func=lambda: update_all(root, image_label, queue)) def update_image(image_label, queue): frame = queue.get() = image.fromarray(frame) b = imagetk.photoimage(image=a) image_label.configure(image=b) image_label._image_cache = b root.update() def update_all(root...
Read more