News for hackers. Stuff that matters.

Ask HN: What is the best MongoDB AWS configuration for maximum performance?

We are struggling with performance issues in MongoDB on AWS using Node.js and the Mongoose module. Has anyone struggled with similar issues and know how to improve this?

We constantly perform inserts to a collection with 13 indices and 16 basic types fields. On a Dell notebook (XPS 9343 i7 2015 8gb with SSD) running Microsoft Windows it is 10x faster than in AWS running Linux. I don't think the OS is the main issue.

We have tried to improve the performance of the AWS instance using SSDs (io1 volume type) and a m4.2xlarge (8 vcpu and 32 GiB of memory) but we are not seeing significant improvements when scaling the VM or changing the storage type.

( 9 comments | 4 points | on HN )
Re:Ask HN: What is the best MongoDB AWS configuration for ma... by zippy786 on Friday June 17, 2016 @09:20AM (#11922455)
> We constantly perform inserts to a collection with 13 indices and 16 basic types fields. On a Dell notebook (XPS 9343 i7 2015 8gb with SSD) running Microsoft Windows it is 10x faster than in AWS running Linux. I don't think the OS is the main issue.

Does your Dell machine has exactly the same data files as the aws instance ? As the size grows index grows as well. Also you have not mentioned how many mongo instances in the cluster. One easy way is, you could separate the index and data directories into separate volumes, think this is only possible in wired tiger though.

Re:Ask HN: What is the best MongoDB AWS configuration for ma... by pyabo on Friday June 17, 2016 @10:00AM (#11922647)
The date is exactly the same since it is retrieved from an internet db.
Re:Ask HN: What is the best MongoDB AWS configuration for ma... by zippy786 on Friday June 17, 2016 @11:50AM (#11923334)
What's the size of the data directory on AWS and Dell workstation ?
Re:Ask HN: What is the best MongoDB AWS configuration for ma... by avitzurel on Thursday June 16, 2016 @06:14PM (#11919295)
We don't have MongoDB in production anymore BUT when we did...

We had 27 instances (9 shards) 3 routers and 3 configs

Main instances ran on c*.4xl machines with 5,000-10,000iops disks. SSD wasn't nearly enough for us.

Re:Ask HN: What is the best MongoDB AWS configuration for ma... by davismwfl on Friday June 17, 2016 @12:35AM (#11920705)
Is it possible the data schema is poor? Honestly I may be misunderstanding your description, but 1 collection with 13 indices and 16 basic type fields seems out of balance to me. And misplaced, misused indices can be a huge issue for any database. You may not see the issue locally on a laptop because of the lack of contention based on the small number of concurrent requests.
Re:Ask HN: What is the best MongoDB AWS configuration for ma... by wslh on Friday June 17, 2016 @08:31AM (#11922213)
No, the laptop and AWS instance are inserting exactly the same data.
Re:Ask HN: What is the best MongoDB AWS configuration for ma... by davismwfl on Saturday June 18, 2016 @12:14PM (#11929197)
The data may be the same but is the contention? In other words, if you have 100 clients accessing and writing in AWS but only 1 on the laptop it isn't realistic and the schema may still be at issue.

How large is the dataset too?

Honestly, I know people like to complain about Mongo a lot of times, but frankly I have had really large and small datasets use it and when setup properly with a well designed schema it never was the bottle neck in terms of performance etc. But on a couple of occasions we chose to move away from Mongo because the trade off in schema and machine config to make it work was more of a problem for us.

Re:Ask HN: What is the best MongoDB AWS configuration for ma... by romanovcode on Friday June 17, 2016 @06:27AM (#11921766)
Maybe your local Mongo is running wiredTiger and the Amazon is running the old storage engine?
Re:Ask HN: What is the best MongoDB AWS configuration for ma... by codeonfire on Saturday June 18, 2016 @12:29AM (#11927114)
It seemed to me that there is some bottleneck for mongo on aws with Ubuntu. Hardware did not matter, mongo was slow for whatever reason. Why don't you run some perf tools?