If you're comfortable using R and the tooling and performance it offers, you should probably keep using it. Julia isn't just about big data, but it does tend to appeal to people who are struggling with problems that are sufficiently hard that their old tools left them in a world of pain. That pain may comes from size, complexity, CPU-intensiveness, or need for more language expressiveness (a hard thing to define). In particular, Julia offers a unique combination of productivity and speed for numerical work that can't be found anywhere else.
I guess I am seeking the one "pet project" that would get me to jump in and give Julia a test drive.
Thanks for all your work even though I don't directly benefit from your work.
I generally find Julia to be a more expressive and fun language to program in than R or Python. It may be my background -- I've done a fair bit of work in Scheme, and in many ways Julia has lots of what I liked about Lisp in it. I don't need all the numerics and stats libraries that R or Python have; what little I need is easy to cobble together very quickly in Julia or has already been implemented. And the FFI is easy to use; calling C is pretty straightforward.
Fortunately the DataFrames ecosystem and related packages (like Query.jl and StructuredQueries.jl) seem to be headed in the right direction.
I have used Julia with a 50GB dataset for feature extraction while I was waiting for R to process the same dataset (10 minutes to an hour depending on the function). I actually learned some Julia while waiting for R. For what I was doing, Julia felt roughly 100 times faster (most delays under a minute).
However, I did not know about dplyr at the time and I will certainly try it in the next project. Although, I am skeptical that using fast libraries is a silver bullet since, at some point, the data will pass through some slow custom R code that I wrote myself.
I prefer to use Julia - the API feels cleaner and more manageable, I don't suffer from subtle type errors, and there is also the possibility of running things in parallel from the same repl. I also had no memory leaks -- I suffered a lot from the bloat caused by my use of R closures.
That said, the power and output quality of ggplot means that I will probably stay with R for plotting. There will also be times when CRAN has the only library that does something we need.
data.table is fast for your project I would imagine.
Could you elaborate? Which new tools?
RStudio has also been a God send for me. RMarkdown and now R Notebooks just amaze me. Then I work in a very MS Office environment that ReportRs (http://davidgohel.github.io/ReporteRs/) is the most under valued library right now.
I didn't have these things 5 years ago. (RStudio was initially released Feb 2011 but didn't start with it till 2013.
I'm interested in people's everyday use of Julia and how it has impacted your workflow. I don't work with "Big Data" most of my data sets are bellow 100k in size. Anyone using Julia for medium and small data sets?