> best graphing calculator attached to an OK scheme.

I discovered "How To Design Programs" somewhere late in my first year of using R. Like most beginning R coders with nominal experience in other languages, I wrote a lot of monolithic scripts in a very imperative style. HtDP gave me a mental framework for decomposing larger problems into bite-sized chunks. The lispy roots of R lent itself particularly well to the model of thinking presented in that book.

Ever since then, I've pined for the graphing calculator parts in a more modern Scheme. When ggplot and then the tidyverse (neé hadleyverse) came on the scene, I was even more convinced that Scheme, especially Racket, was the ideal future for data science. If R could support a large ecosystem like tidyverse, just imagine what the metaprogramming facilities of Racket could do!

But I think those graphing calculator parts are hard to reproduce. Attempts to clone ggplot2 fall short year after year, because most other languages don't have grid graphics to build on top of. R is a deep ecosystem on "an OK scheme," which is damned hard to beat.

Aside: my first year with R, was in an urban planning masters program and I was terrified of my first big kid statistics course (taught in SPSS). I decided I'd give myself bonus work by learning R. While it was absurd to be doing my stats homework in SPSS, then R, then reviewing HtDP on top of the rest of my course load, I did ace that stats course. :-)

There is an better alternate universe where xlisp-stat doesn't fall behind and S doesn't happen.

Interestingly, there appears to be an attempted reboot: https://lisp-stat.dev/

My first reaction, is "why not on a modern Scheme as opposed to Common Lisp" and in so thinking, I have demonstrated exactly why no lisp / scheme has ever achieved critical mass :-)

I would think that the reasons for "why not Scheme" would include the fact that CL has more existing libraries, better built-in support for multidimensional arrays, efficient low-level code (now even with SIMD on SBCL using SB-SIMD), application-controlled JIT compilation of specialized code, almost guaranteed support for native multithreading, etc.

Shocking. I still using the xlisp-stat based social simulation book first edition (20+ years ago). The move to ... does not fit my mind. I do not know there is a re-start!!!

Obvious answer: 3rd party library support. Same thing that makes R ubiquitous.

I really don’t think this is that different from how a lot of us learned to code though. I learned by writing html and PHP specifically to solve a problem of having a website. The only difference I see is that the average CS student has to write reliable and working code as their career, while quants and statisticians tend to think that code is just a means to the end. Both are right I’d say depending on the problem.

> R is a highly optimized, well-oiled machine if you're using it for its highly-optimized, well-oiled purposes.

This hits home for me. We are just starting to use R for risk modeling where I work. R, more than any language I've ever used, makes me appreciate "worse is better". From a theoretical "aesthetic" perspective R is a mess. Yet for data processing all those theoretical concerns don't matter. It just works.

It's honestly kind of humbling that something so theoretically messy can be so practically coherent. It makes me question my assumptions about simplicity.

R "just works" now because a huge amount of effort has gone into improving the language over the last 10 or so years, in part spurred by the tidyverse movement, although not restricted in scope to tidyverse. When I was starting grad school around 2010, if someone sent you some R code, the chances that you would be able to "just run" it were basically zero: there would be weird version mismatches in how functions worked, file paths would be specified in inconsistent ways in different parts of the script, all kinds of crazy impenetrable errors were the norm. Now there are several R code snippets posted in these HN comments that will run without trouble. If I could have gone back in time and told myself that this is how R would develop, I would have been shocked (and happy).

Part of it has to do with strict testing in CRAN as well. Packages have to pass tests and confirmed to compile. This adds reliability to package management across platforms.

That said I still run into trouble with package deprecations. I was trying to install the optmatch package (deprecated but still used by causal inference packages) and had a really tough time getting it to compile on macOS.

I dunno man, python has always seemed a little bit worse on this stuff to me. At least with R if you had a consistent version, everything off CRAN worked together.

I think R 3.0 introduced namespaces which fixed a lot of the really crazy stuff.

Also, I was writing Sweave in 2010 for my thesis, and I definitely wasn't alone.

Namespaces showed up around 2004, so somewhere around 2.0.0. I don't think they were mandatory until much later.

I suspect that has as much to do with the maturation of the data science community as it does the language environment. There have been pockets of R users who put much effort into reproducibility before the era you cite, such as Bioconductor.

When I think back to the era you're describing what I recall was people winging around hacky scripts being the norm regardless of their environment. While still not something I'd think of as software engineering best practices, what I see now is less Wild West.

I hadn't thought about R as a "worse is better" language, but that's a good way to think about it. Makes sense, too, since it came from the place that inspired worse is better.

R comes from New Zealand, no?

R is an implementation of S. John Chambers worked on it at Bell Labs starting in 1975.

https://en.wikipedia.org/wiki/S_%28programming_language%29

I’m trying to figure out if you were actually asking a question or if it was rhetorical and you were calling shots.

Is there any reason you chose R over Python? Is it just because that’s the go to language?

We asked the people are going to be using it what they'd prefer. Many of them are recent graduates, and they told us they mostly used R during their university courses. It's just a pure familiarity play. The alternative was building a huge system on a mainframe (we're a legacy bank).

Really there wasn't a lot of thought put into the language. We figure that if it ends up being a total failure, we can just pivot.

Bravo! This is exactly right.

This is just a quick example - I would be grateful if people could recreate this brief look at UK COVID figures in another language:

  library(tidyverse)
  library(scales)
  
  download.file(url = "https://api.coronavirus.data.gov.uk/v2/data?areaType=overview&metric=covidOccupiedMVBeds&metric=newAdmissions&metric=newCasesBySpecimenDate&metric=newDeaths28DaysByDeathDate&metric=newPeopleReceivingFirstDose&format=csv", destfile = "./data.csv", method = "wget")
  
  read_csv("./data.csv") %>%
  pivot_longer(names_to = "Data", cols = c(newCasesBySpecimenDate,
                       covidOccupiedMVBeds,
                       newAdmissions,
                       newDeaths28DaysByDeathDate)) %>%
  mutate(Data = factor(Data)) %>%
  mutate(Data = recode_factor(Data, newCasesBySpecimenDate = "New Cases",
         newAdmissions = "Admissions",
         newDeaths28DaysByDeathDate = "Deaths",
         covidOccupiedMVBeds = "Ventilated")) %>%
  ggplot(aes(y = value, x = date, colour = Data))+
  geom_point(size = 1, colour = "gray", alpha = 0.6)+
  geom_smooth(type = "LOESS", span = 0.1)+
  labs(y = "Daily rate", x = "Date", colour = "UK COVID-19")+
  scale_x_date(date_breaks = "months", date_labels = "%b-%y")+
  scale_y_log10(labels = comma(10 ^ (0:5),
                 accuracy = 1),
         breaks = 10 ^ (0:5))+
  theme(axis.text.x = element_text(angle = 45, hjust = 1))

    import pandas as pd
    import matplotlib.pyplot as plt
    import matplotlib.dates as mdates
    import seaborn as sn

    df = (pd.read_csv("/tmp/overview_2022-03-21.csv") # i just used curl beforehand
        .assign(date=lambda x: pd.to_datetime(x["date"]))
        .set_index("date")
        .melt(value_vars=[
                "newCasesBySpecimenDate",
                "covidOccupiedMVBeds",
                "newAdmissions",
                "newDeaths28DaysByDeathDate"],
            var_name="Data", ignore_index=False)
        .assign(Data=lambda x: x["Data"].replace({
            "newCasesBySpecimenDate": "New Cases",
            "newAdmissions": "Admissions",
            "newDeaths28DaysByDeathDate": "Deaths",
            "covidOccupiedMVBeds": "Ventilated"
        }))
    )
    ax = sn.scatterplot(data=df, x=df.index, y=df["value"], hue="Data")
    ax.set(xlabel="Date", ylabel="Daily rate", yscale="log")
    ax.xaxis.set_major_formatter(mdates.DateFormatter("%b"))
    plt.show()

I spend 2 minutes on the pandas part and 20 minutes on the plotting part, which really says it all. Seaborn's support for smoothing is really bad and doesn't play nicely with datetimes for some reason, so if I wanted smoothing I'd need to do it myself. And the other stuff I left out requires going into matplotlib's documentation which I don't want to spend time on.

pandas is as good or better than R's dataframe manipulation, but R's plotting tools are best in class. I hate all the python plotting libraries.

> I hate all the python plotting libraries.

I've given up discussing this. My personal opinion is that MPL is the assembly vs ggplots2's Python.

Yes, maybe there are some things that are doable in MPL but not in ggplot2, but never (and I mean - never -) has any one of my colleagues found a single example that I couldnt recreate in ggplot2 with a more readable code that resulted in a better looking plot.

MPL has this Latex like "once you get the hang of it, you will never use anything else in life for anything" and OOP's "If its good code then it is OOP and if its not, then you're not doing correct OOP" mythisque hanging over it. When confronted with bad MPL code that results in bad plots, its always one of those two.

ggplot2 is the best plotting package out there and imo one of the best "end-user" packages in any language. Also, Hadley is a saint.

Check out plotnine. Really good clone of ggplot for python.

https://plotnine.readthedocs.io/en/stable/

Thanks for providing that. Interesting to see the '.' used like a pipe. Always thought of it used in an OOP context, but interesting that it can be functional too. I also had difficulty fitting a LOESS curve to the plot, but I could do a linear model. The LOESS would have been possible doing it manually I guess.

This is great.

You don't need the curl -- read_csv works with URLs directly.

The lambda can be replaced by passing parse_dates=[“date”]

Thanks

This was fun to play around with. I made some very minor changes and posted at https://gist.github.com/hadley/d54895557fbb0fe0402d2277b9011....

It revealed to me that there's a buglet in `forcats::last()` (https://github.com/tidyverse/forcats/issues/303) and made me wonder if `pivot_longer()` should be able to rename the columns as you pivot them (https://github.com/tidyverse/tidyr/issues/1338)

This might be the most excited I've gotten about a comment in hackernews for a while! So interesting to see your style of code (I was actually on 4.0 so gave me the impetus to upgrade to get the new pipes). fct_reorder - what a hugely useful function I didn't know about. The chicks vignette was nicely illustrative. And label_date_short also super useful. Also curious generally about your bracket style (I just tend to pile them all up when closing, which is something I only do in R and probably shouldn't!).

Renaming factors is one of those things that always seems a bit awkward. I think I've used several methods. Passing a list of named vectors into `levels(x)` allows a many-to-one mapping but was quite dangerous. I've used revalue and mapvalues from plyr. fct_recode is new to me. But yes, renaming while reshaping could be quite convenient. Just looking at fct_recode now, it looks really nice. Seems to support many-to-one and being able to pass it a name vector is very convenient.

Learnt so much today, and this is even before trying out the python and julia examples! Many thanks for this and all your work in R!

If you enjoyed this you might like https://github.com/VictimOfMaths/COVID-19 :)

A Julia solution with Chain and Gadfly might look something like this, although I've translated the R fairly directly so it might not be very idiomatic.

    import CSV
    using Chain: @chain
    using DataFrames
    import Downloads
    using Gadfly
    using Dates

    @chain begin
        Downloads.download(
            "https://api.coronavirus.data.gov.uk/v2/data?areaType=overview&metric=covidOccupiedMVBeds&metric=newAdmissions&metric=newCasesBySpecimenDate&metric=newDeaths28DaysByDeathDate&metric=newPeopleReceivingFirstDose&format=csv",
        )
        CSV.File
        DataFrame
        stack(
            [:newCasesBySpecimenDate, :covidOccupiedMVBeds, :newAdmissions, :newDeaths28DaysByDeathDate];
            variable_name = :Data,
        )
        transform(
            :Data =>
                (
                    x -> replace(
                        x,
                        "newCasesBySpecimenDate" => "NewCases",
                        "newAdmissions" => "Admissions",
                        "newDeaths28DaysByDeathDate" => "Deaths",
                        "covidOccupiedMVBeds" => "Ventilated",
                    )
                ) => :Data,
        )
        subset(:value => ByRow(!ismissing)) # Can't plot Geom.smooth with missings
        plot(
            _,
            x = :date,
            y = :value,
            colour = :Data,
            layer(Geom.smooth(method = :loess, smoothing = 0.1)),
            layer(Geom.point),
            Scale.y_log10(),
            Guide.xlabel("Date"),
            Guide.ylabel("Daily rate"),
            Guide.xlabel("Angle"),
            Guide.colorkey(title = "UK COVID-19"),
        )
    end

Thank you for that, good to see there's an elegant Julia solution! The last time I was using 'pipes' with Julia, I think I was using DataFramesMeta. I also really like this interactive gadfly plot - reminds me of Matlab, but better. It's been a little while since using Julia, so I'd forgotten about the pre-compiling thing, but generally this code looks pretty nice and clear.

I used to use DataFramesMeta.jl, but eventually I found that the mini-DSL that DataFrames.jl has created is really powerful and not overly verbose. Now, going back to the Tidyverse's syntax makes me feel a little uneasy, like there's just too much magic going on behind the scenes, even though I used it for years with no problems.

You don't need the download file command -- read CSV works with URLs directly :-)

Oh nice, that's very useful to know!

I think these kind of common task challenges are great for comparison. You see a few approaches to a single task and you can do a more aligned and detailed comparison.

Unfortunately, it’s also a lot of work. In this case, you’ve posted an intermediate stage artifact from R. If one of the many Python programmers reading this want to produce a comparable artifact they need to understand or run that code. That alone reduces your likelihood of getting any substantial replies.

Maybe add a link to an image of the resulting plot?

Good point, looks like I'm too late to edit, but here's a link [1] (excuse the R style indexing).

Yes, I love these things and very curious to see what hackernews comes up with. Project Euler was an eye opener for how things could be optimised in different languages.

[1] https://i.imgur.com/M8DX98I.png

I also use R for any heavy data manipulation, but I primarily use the data.table package. The efficiency that both of these packages unlock is absolutely unparalleled in any other tabular data manipulation library, in any other language that I have used. And R has the top 2!!

My skin writhes every time I need to type:

table.loc[(table.column > 2) | (table.column2 < 3)].reset_index(drop=True)

when I want to subset a table.

.loc lets you supply a callable, so you can write:

  table.loc[lambda df: df["column"].between(2, 3, inclusive="neither")]

this is useful when your dataframe has a long name, or when you have some long method chain and you need to subset at the end: table.foo().bar().baz().loc[lambda df: ...]

it is still more verbose, but I actually prefer always providing column names as strings. it's more explicit. I don't like R's environment-manipulation metaprogramming magic where you can give column names as symbols.

as for resetting the index all the time, this is something of an antipattern. if you set up your index right beforehand it isn't necessary so often.

Can't get around resetting the index, as far as I know, but for the filtering you can also do,

`table.query("column > 2 and column2 < 3")`

do any IDEs help you autocomplete the string argument? that's a bit part of the non-standard evaluation magic in the tidyverse these days; you can use bare, _unquoted_ names, _and_ get excellent autocomplete, at least in RStudio.

Not to mention the auto complete that comes with RStudio. Is there any way to get equivalent functionality in Jupyter?

If you set up use the Jupyter extension[0] and open your notebooks in VS Code you get Intellisense (code completion, method info and hints etc).

0: https://marketplace.visualstudio.com/items?itemName=ms-tools...

IME this is strictly worse than the RStudio experience; most of the time I hit tab in a VSCode notebook I get way too many options that IMO are clearly not what I want, though at some level it's more pandas' fault (too many methods & attributes even before attaching every column name as an attribute) than VSCode or Intellisense.

In addition to the sibling answers:

If you know the class of an object, let's say Class, but the object has yet to be "constructed" so that IPython can correctly infer its type, you can type `Class.[TAB]` in IPython and look at its methods.

For example, in Sympy, you have a matrix type called Matrix. You can do `(A * B).diagonalize()`, or alternatively you can do `Matrix.diagonalize(A * B)`, which has some advantages because doing `(A * B).[TAB]` does nothing useful because Python can't infer types.

You can also do the same for modules. `ModuleName.[TAB]`

To be honest though, I found the experience smoother in R for some reason.

I use pycharm which has decent autocomplete. Pycharm has its own issue though, it fills out its autocomplete info by looking at the function that created the object, not the object itself. So if a function can return different types, autocomplete won’t work. That’s caused me quite a bit of pain.

My wife is a researcher and started delving into doing her own statistical analysis. It's be fun (and frustrating) learning R with her. I agree that dplyr and the tidyverse are some fantastic packges for a software engineer who thinks about spreadsheets as SQL tables.

I would say the most frustrating part about RStudio is that it is a workbook where you can execute code based on your cursor. For my wife, these workbooks become a total mess because things aren't necessarily run sequentially.

Here are two good strategies:

1. Always run from the top, using the "run previous chunks" button. When this gets too slow, you know that it's time to think harder about your workflow. For a more extreme version of the same idea, regularly restart R using Ctrl-Shift-0, and run from the top. It'll ensure your code is working right.

2. Have a setup chunk that always gets you to the same state. Make sure every other chunk works directly after calling the setup chunk. Then just alternate between "run setup chunk" and "run current chunk".

0. Don’t use notebooks. Neither RStudio nor Jupyter. They prevent people from developing good programming skills and good version control skills, and they encourage making a huge mess.

I mean that's fair, but if you have to write an academic paper, your options are more or less use a notebook, or copy and paste your results. Of course, the question is how much code you should write in the notebook, versus having it in a more organized set of functions and libraries. It's very easy to end up with a huge bloated document which contains thousands of lines of spaghetti.

I absolutely agree with this:

> the question is how much code you should write in the notebook, versus having it in a more organized set of functions and libraries. It's very easy to end up with a huge bloated document which contains thousands of lines of spaghetti.

But this isn't correct:

> your options are more or less use a notebook, or copy and paste your results.

What's wrong with writing scripts that write images to disk? That's how millions of academic papers were written before the advent of notebooks. You could use Makefiles if you like, or you could even use a technology such as Sweave to automatically mix images with LaTeX output.

I mean this as politely as possible but the fact that you think that the options are "use a notebook or copy and paste" I think shows that you've caught a notebook mentality disease! The fundamental point I'm trying to make is that you we don't need to do everything interactively from REPLs. REPLs are great for trying things out, but when it comes to producing the images for your paper, those should be produced by scripts, not by commands entered into a REPL, or notebook. An those scripts should evolve via version control, which is the basis of evolving any good and correct software. And the scripts for producing images for a paper should be good and correct software.

We might be talking at cross purposes. I'm thinking of e.g. a Rmarkdown notebook, which is indeed a script not (necessarily) an interactive notebook like Jupyter. And it can be automatically compiled, via a makefile or something similar, and put into version control.

The point is that it makes sense to mix english prose + code to e.g. produce tables or graphs, even if most of the heavy lifting is done separately in code files.

Ah, OK, yes I was talking at cross-purposes to some extent then, thanks. (Jupyter notebooks are hopeless in version control due to the JSON format, but I'm not familiar with Rmarkdown notebooks -- do you get sane diffs?)

Yep, so what you say makes sense. Isn't it sometimes a bit overly prescriptive to assume all collaborators use Rmarkdown? (Perhaps not! I used to work in biology and statistics and R was very ubiquitous.)

Rmarkdown indeed makes sane diffs, it's just a nice text-based format. You can use python or julia as well (see quarto.org for the latest version of this). Getting collaborators on board... yeah, that varies.

I love the phrase "eviscerate a fresh data set" :D

Great stuff. Could you maybe share a bit of these "3 dozen" scripts? This could be super helpful.

Looks like your notebooks focus on crypto currencies :-) Good use case for data analysis!