shamsulbuddy
shamsulbuddy on Hacker News
I tried Tokudb as well and below are the stats comparison for complex SUM and COUNT query.
Please note that the number of rows and size is exactly the same and there is a partition on a Date column
Table size 10GB and rows 12 Million
Also note that I am using the default settings of Tokudb
With Tokudb - 1 min 41.38 sec
With Innodb - 58.17 sec
So almost double.
Do you think that 10GB data is easily to manage .. what all common practices do you suggest . any examples with use case
Recently i have got the task to work on a 10 GB and 12 Million rows table to make it work with complex aggregation queries like SUM or COUNT.
Server - Ubuntu 14.0.4
Mysql - Percona XtraDB Cluster 5.6
RAM - 16 GB
CPU - 8 Cores - 2.95 Ghz
SSD - 100 GB
I tried many options --
1) Increased the innodb_buffer_pool_size to 8GB on a 16GB RAM machine -- It helped but nothing magical here
2) Add few more Keys on Date based columns and force the users in front end to select at least one date range column -- Seen some performance gain here
3) Tried MyISAM engine -- I would say in present days MyISAM is history since Innodb itself is pretty much comparable to MyISAM -- so didn't see amy much performance gain .. one disadvantage was that loading of this 12 Million rows took ages .. and also "SELECT table_rows FROM information_schema.tables" had hanged , so i was not able to to figure out that how many rows have been loaded in my Table.
4) Finally I tried Partitioning the Table based on a DATE column -- Massive Performance gain .. If the user has selected a date range which falls under one or two partitions , then you can get the results very fast , even if it falls in many partitions still the performance is acceptable .. Useful ref - http://www.slideshare.net/datacharmer/mysql-partitions-tutor... -- remember that its Mandatory that you add that date column as part of Primary Key if you are Partitioning on that date column .
--- For Ref -- my Innodb settings
# INNODB #
innodb-flush-method = O_DIRECT
innodb-log-files-in-group = 2
innodb-log-file-size = 512M
innodb-flush-log-at-trx-commit = 2
innodb-file-per-table = 1
innodb-buffer-pool-size = 6G
bulk-insert-buffer-size = 256M
innodb_thread_concurrency = 16
You have hit the pain point ..Very useful app ..great job
Posted by
shamsulbuddy on Saturday August 02, 2014 @02:16PM
from the beatsbydre.com dept.
yeah we can wait for later stages for the implementation in demo but in the meantime any documentation or pointers on how to achieve the direct download of results..
In Demo it says to get see the data in browsers console.. thats cool .. but how do we Download the data , like the results should directly prompt a Save dialog window..
I was wondering , if we can do Home Automation and use the latest iOS 8 features to control most of the stuff in house
Posted by
shamsulbuddy on Wednesday June 11, 2014 @09:36AM
from the blog.derivatived.com dept.
Posted by
shamsulbuddy on Thursday June 05, 2014 @09:53PM
from the venoo.ie dept.
Posted by
shamsulbuddy on Sunday June 01, 2014 @03:28PM
from the battlemesh.org dept.
Posted by
shamsulbuddy on Thursday May 29, 2014 @10:29AM
from the github.com dept.
can't see d3 for the funnel/cohort charts .. then what are you using for drawing the charts?
Cool , this looks very good. In the page
https://www.empatica.com/info.php there are few projects been mentioned something like "Parkinson's disease early detection" ..are there any corresponding links for this to get more info as in how does it detect the Parkinsons disease in early stages.
Posted by
shamsulbuddy on Wednesday May 21, 2014 @06:49PM
from the medium.com dept.
what technology you have used here for charts as it doesnt seems to be d3. Also will you be able to opensource the data behind it ?
Mixture of Webaudio API with D3 is awesome , liked the themes as well .. good work :)
Check for heapanalytics.com ..awesome eventing framework
This is rocksolid basic script..one quick question..if you post on every event then dont you think so that it will generate huge traffic..how did you handle that
Posted by
shamsulbuddy on Thursday March 27, 2014 @12:29PM
from the ask-hn dept.
Today when I saw btappjs.com a thought came to my mind as in why we dont have a .js domain even though we have so many sites dedicated to some kind of Javascript library . I am aware of jsdb.io which is very helpful for getting all the Javascript libraries at one place .
Few basic things which can be done is like .. install Fail2ban , change the default SSH port to something else, and use PermitRootlogin as No in sshd_config file
looks like Webscraping of major news sites ..
Posted by
shamsulbuddy on Thursday January 23, 2014 @08:43PM
from the self-post dept.
I just have reinstalled Debian 7 on my VPS. Logged in for the first time with "root" and on port 22..then I didn't locked down anything and within an hour I can see the below root password breaking attempt in /var/log/auth.log file .. WHOIS shows its an Chinese IP.
God knows when these people will get rid of Script kiddies.
Now I have locked down my VPS... does anybody else have similar story...
Does such Webscraping is allowed legally. Since it is not done directly from our servers and if any legal action will be taken by the scraped website , will it be on kimonolabs..or the user..
http://hnapp.com/ -- This is the best HN Scraped site.. returns data in JSON / RSS format.
Will u mind sharing some more info on this real time analytics using d3 as i am also planning to do something like tht..