chrisparnin

Living papers are useful vision, but it will take a long way to get there.

Even notebooks still are problematic, for example, this study found that only 25% of Jupyter notebooks could be executed, and of those, only 4% actually reproduced the same results.

http://www.ic.uff.br/~leomurta/papers/pimentel2019a.pdf

One compromise is to evaluate the paper separate from it's artifacts, which are reviewed for availability, reproducibility, and reusability. In software engineering conferences, this is becoming a standard, and while there is a huge burden for reviewers to evaluate these things, I think it does take us in the right direction. So in this case, we also submitted our paper for evaluation for its artifacts.

Yes, we took a lot of inspiration from when my PhD student interned with RStudio:

You can build some cool stuff with RMarkdown and LearnR:

* https://rmarkdown.rstudio.com/

* https://rmarkdown.rstudio.com/docs/articles/rmarkdown.html

One interesting distinction is "tutorials" vs. "lessons"... LearnR and datacamp are much better for _lessons_, where you ask a learner to type in some code and check to see if you got it right, with hidden cells with the appropriate test cases backend.

Yes, see: http://howardism.org/Technical/Emacs/literate-devops.html

Nice! I like the "link includes" feature. Funny, because we're displaying the terminal results, we've had the opposite problem, where we need to get and process the ansi-escape codes, so we get colors, and proper formatting, etc.

Thanks for trying it out. We have a bit of work to make bootstrap process easier.

We have a limited set of notebooks published online to demo how they work. For example:

* https://docable.cloud/chrisparnin/examples/basics/script.md

* https://docable.cloud/chrisparnin/examples/tutorials/Git.md

Right now, we're using the notebooks to build better lecture materials + workshops, especially for my DevOps course: https://github.com/CSC-DevOps/Course

We've seen some interest in having support for live documentation + OneOps (simple runbooks for one-off devops tasks), so we'll probably continue to explore this more.

Depending on how you conduct them, they are actually stress tests:

https://medium.com/@gameweld/the-case-for-the-private-techni....

Yes, one use-case is making it easier to setup/teardown clusters for local testing. Two practical scenarios for us: 1) autograding ansible/configuration scripts, 2) CI for instructions/tutorials that involve clusters/devops: https://builds.sr.ht/~ottomatica/job/69644#task-report

Cloud-ready images is an important direction, and on the horizon.

Some limitations in terms of the vms and providers:

* If the size of the initrd is too large, it cannot properly unpack into vm's RAM --- size of RAM must be increased accordingly. We could also change [boot params](https://www.lightofdawn.org/blog/?viewDetailed=00128), or use shared disks, etc.

* For hyperkit, apple's vmnet requires sudo to create a bridge interface on host. We've played with a version that use's vpnkit and port forwarding (like linuxkit/Docker for Mac), but this adds lots of complexity in image, and opted for the simpler approach.

* We would like a better template mechanism for reusing base images and extending. Right now, we support using base image reuse, with extensions through docker buildargs---ideally, we would want something like %include support in Dockerfiles.

* Finally, we're investigating how to make images work well on multiple providers. For example, ubuntu does not play nice with hyperkit out-of-the-box, but works fine for vbox and kvm.

From our discussion with folks at Netflix. We had a long talk about their verification checks. In regards to monitoring:

> Monitoring and alerting is a different beast of a topic. Certainly you can monitor all of these things, but our guidance on alerting strategies is along the lines of finding the top-level metrics for a service (google SREs would call these Service Level Objectives) and only alerting people to dig in when those are impacted. Usually error rates, latency (how many micro/milli/seconds to respond), and throughput (requests per second). A lot of this relies on the people writing the application to instrument their code to store metrics and events. There’s only so much much that you can monitor from outside the application. Some sort of outlier detection is popular too to terminate bad, one-off instances (particularly common in the public cloud)—funny thing is you have to alert on whether you’re killing too many instances too quickly (irony of automation).

I would venture we want to separate testing images and infrastructure on startup before receiving traffic or just making it out of CI vs. monitoring the health of instances. But they also can have things in common!