babahoyo

For me the appeal is 3-fold

1) I can contribute to widely used packages like DataFrames and HypothesisTests. I had never made a git commit before this and my only "real" programming was CS 101 in Java. The fact that I could get up and running so easily contributing is a testament to the language's ease of use

2) I think its tough to predict your computational needs at the start of a project. Sure everything can be done in `lme` in R at the outset, but if you need some new bootstrapping procedure that a reviewer wants you might be left connecting some high performance code to an existing, large, R-based codebase. That's tough. I think Julia makes that "refactoring" (if you can call it that) easy.

3) Hopefully Julia will open a lot of doors for me in the future in my research career. I will be able to write interesting simulation procedures that are otherwise too unweildy for the comparison R or Stata economist.

If you are running a script over and over (or using small modules to do organize things) it will be very fast and easy to work with. The compilation is just for the first time a script is run, not every time.

It really isn't a barrier any more than, say, waiting for `library` commands in R.

Yeah i guess the UI isnt that bad. Performance > UI in many cases for me though. Hopefully responsiveness times are something people consider more in designing an interface.

Weird question, but why is there so much bad UI out there? Do you feel like you've failed as a field when commonly used applications like Gmail and adobe reader have such bad user experiences?

It's crazy that microsoft got sued for anti-trust because explorer dominated so much of the market, and now they are ceding space so google can do the same.

ah thanks! Yeah these are super interesting

wow that is pretty dystopian

I've been looking for blog posts and content about macOS UX design and testing. Because it really is stellar. I'm curious what kind of stuff the elementaryOS team is reading that informs their design choices. Any ideas where to look?

But the nice interactivity of it is only in RStudio iirc, which is a major selling point.

I think there is a lot of reason for moralistic criticism of excel.

> That would make sharing and reproducibility just as easy as easy as getting them to switch out their complete toolchain

Reproducibility is about reading as much as it is getting the right number. Excel hides the operations from the reader, so that there could be a cell X182that has the key operation that you never area able to trace back.

https://www.bloomberg.com/news/articles/2013-04-18/faq-reinh...

But I totally agree that an excel spreadsheet is the most popular and easiest to learn programming language on the planet. That's why I mentioned ipython notebooks above. Excel is great because you can enter something and immediately see your results in front of you, which is something that jupyter does as well. But we definitely need to draw more from excel's UI and workflow if we want people to switch to those tools.

> But I have doubts that unwillingness to share code & data is actually a technical problems, instead of a lack of incentives/embarrassment of sharing ones' awful code/dreams of patents future/privacy issues.

That's true. But at the very least internal use of plain text and git would be immediately helpful for collaboration and project management (branching, code reviews etc.). I think you underestimate how unwieldy a project like this can get very quickly.

> Last time I tried Julia, about every second package had a specific version that it required, and very few of them shared their preferences.

The landscape is still settling down after the 1.0 release. I share the concern about particular version numbers, but notice that basically all packages are on 0.X in SemVer. When the ecosystem makes it to 1.0 as well things will improve dramatically.

I'm worried about seeing a script that says

    df = read_csv(file)
    df <- df %>% mutate(log_income = log(income))
    # do manipulations in shiny
    Shiny(df)
    # click "transform" in the top right hand corner...

I think this is missing the forest for the trees. The audience for these tools is people who have never written in a text editor in their life! They literally might not even recognize that some fonts are fixed-width and not know what textedit is.

That is to say, emacs might be over-kill. But literate programming is a pretty powerful concept in general, and rmarkdown and Rstudio (along with simliar tools in Python, Julia, etc.) make this super easy, and I think its a great way to introduce people to programming.

Getting people to use Git is the fight worth having imo, but its still hard. I was talking to someone the other day who didn't like git because tough to have comments about the work: they didn't use code reviews on github to get consensus, just comments in the code.

Plus there is the whole data-management problem. Quilt, SQL connections, and DataDeps for Julia are good solutions, but there isn't a single one answer that we've coalesced behind.

These are all really tough problems. I'm actually really interested in MS's acquisition of github for this reason. Maybe they can put some money towards the obstacles that have prevented mass adoption of scripting and git.

Getting people to switch to R (or Julia) from excel is to balance two competing goals.

First, you need to establish that scripting is far superior to point-and-click interfaces via reproducibility and legibility. This is the fight worth having, as it gets to the core of what it means to do science and share results.

Second, you need to convince them that they don't have to sacrifice the tractability and ease of use of excel. Jupyter notebooks, for all their faults, really sell this idea well. But the first goal should dominate the second. Nteract and shiny are great, but I they are big tools that are difficult to teach beginners to code up. We shouldn't say "use R because you can still use point-and-click interfaces", we should argue against point-and-click interfaces all together in this context.

I also agree with the poster below that running a script again with a new parameter changed is super super easy, and you dont need a GUI to explore data like that.

I weirdly love reading about other people's workflow. This is really cool thanks for sharing.

One issue I have with literate programming is that its difficult enough to write code, now you have to worry about writing code in the middle of writing prose. For that reason I prefer to just have all my code output a bunch of latex files so I can separate the code-writing from the paper-writing. Which I guess is against the philosophy of literate programming as a whole...

How did you get around this? Do you share the same frustration that you are constantly being distracted by one or the other?

This reminds me of how incredibly good TurboTax is at UI, and if US and state governments were to invest in UI we wouldn't have to pay $100 a year just to do our taxes.

iirc pandoc's markdown provides the set of functionality that one is capable of transforming back and forth. So as long as you stay within those formatting confines, you are set.

This works for everything except table notes a la ```threeparttable```

you were somehow satisfied with the way stata handles missing values???

    gen x = y if z > 4 // headaches abound

Julia's missing value support is great now and is only going to get better. You have to be more careful with how you use them, but you won't get anything like the output above in julia.

* For reference, stata uses +Inf as missing value, so any operation with "greater then" is going to assign missing values to something. And yes, there have research papers retracted due to this behavior.

Check out DataFramesMeta, which unfortunately isn't working on 1.0 yet. They have basically a 1-1 matching of `dplyr` verbs to julia versions.

I don't think a standardized and idiomatic data-cleaning process has been established yet, which is for the best right now. There is `JuliaDBMeta` for metaprogramming with JuliaDB tables, and the `Queryverse` for working with a wide array of objects.

One way that Julia's metaprogramming shines is with the ability to go into the AST and replace symbols, enabling local scopes that are more readable than other scopes. One workflow I'm excited to experiment with is something like this

    @as my_long_dataset d begin # make d = my_long_dataset in this scope
    @with d begin 
    t = :x1 + :x2 + x3 # these symbols are arrays inside this @with scope
    d.new_var = t # assign the variable
    end
    end

Of course, with the `@as` macro you probably don't save that many keystrokes if you are just doing `d.x` or `d[:x1, :x2]`... The ecosystem is still evolving but the point is that I like how you can replicate something like `attach` scoping in R without all the headaches. I think it makes a cleaning script feel more like you are only working with the data you care about.

Julia is increasingly becoming better than R and Stata for data cleaning. Many of its metaprogramming tools beat `dplyr` in syntax and features. So if the data-cleaning to regression stack (which i would guess is different than scientific computing) is your thing, then i would recommend trying Julia out.

Many scientists might have mathematical ideas about how an operation should be done, but dont want to learn C++ to implement them. We create a division between scientists and programmers that hurts productivity.

If you are coming from R I would say that DataFrames and DataFramesMeta are just as good as dplyr. But thats just a super small, less scientific, corner of the entire excellent ecosystem