Does anyone here have thoughts on JOSS? I've reviewed for them once, had the impression the editors take their job seriously, and think I'll review again in the future. The review approach that the journal facilitates has a strong focus on engineering aspects, i.e. it addresses a weakness of other venues, where it often does not matter how messy, unstable, and poorly documented the code is (or even if it compiles). On the other hand, the JOSS reviewers are typically not experts on the problem that the software is solving.

I'm currently reviewing for JOSS, and have done so before. In many ways they're a very strange journal: the paper is nearly an afterthought, and the review is focused on the code. But I like them. As you say, the editors take their role seriously. And it seems to have two valuable contributions.

Firstly, encouraging and structuring code review in academia. My own code is almost entirely solo (and messy), so a venue for structured review and an incentive to robustify public code is good. Secondly, the culture in some disciplines is that code is not citable, only papers - and JOSS is an end-run around this. I hope this second situation is changing, but we're not there yet so JOSS has a valuable role for the moment in simply being a 'journal' assigning DOIs basically for code packages.

[Scholarly] Code review tools; criteria and implementations?

Does JOSS specify e.g. ReviewBoard, GitHub Pull Request reviews, or Gerrit for code reviews?

The reviews for JOSS happen on github[0] but the journal's not prescriptive about how you develop your package as long as the code is public. The criteria for the JOSS review are very clear[1].

I don't want to oversell the depth of the code review possible; not all of the reviewers will be fully expert in whatever tiny cutting-edge area the package is for (making correctness checks difficult beyond the test suite), and most of us are academics-who-code rather than research software engineers. But the fact it's happening at all is a great step forward.

[0]: https://github.com/openjournals/joss-reviews/issues [1]: https://joss.readthedocs.io/en/latest/review_criteria.html

Thanks for the citations. Looks like Wikipedia has "software review" and "software peer review":

https://en.wikipedia.org/wiki/Software_review

https://en.wikipedia.org/wiki/Software_peer_review

I'd add "Antipatterns" > "Software" https://en.wikipedia.org/wiki/Anti-pattern#Software_design

and "Code smells" > "Common code smells" https://en.wikipedia.org/wiki/Code_smell#Common_code_smells

and "Design smells" for advanced reviewers: https://en.wikipedia.org/wiki/Design_smell

and the CWE "Common Weakness Enumeration" numbers and thus URLs for Issues from the CWE Top 25 and beyond: https://cwe.mitre.org/top25/

FWIW, many or most scientists are not even trying to be software engineers: they just write slow code without reusing already-tested components and expect someone else to review Pull Requests after their PDF is considered impactful. They know enough coding to push the bar for their domain a bit higher each time.

Are there points for at least in-writing planing for the complete lifecycle and governance of an ongoing thesis defense of open source software for science; after we publish, what becomes of this code?

From https://joss.theoj.org/about#costs :

> Income: JOSS has an experimental collaboration with AAS publishing where authors submitting to one of the AAS journals can also publish a companion software paper in JOSS, thereby receiving a review of their software. For this service, JOSS receives a small donation from AAS publishing. In 2019, JOSS received $200 as a result of this collaboration.

A low cost of $1/page just shows how much of a scam even prestigious journals have become for all involved parties.

Referees work for free. Submitting authors pay through the nose for presumably typesetting and proofs whose true cost is closer to $1/page rather than hundreds. Editors are overworked.

Sci-Hub was supposed to disrupt that industry, but it seems all it's done is shift burden to authors and created pay-for publishing.

Moderation costs money, too.

Additional ScholarlaryArticle "Journal" costs: moderation, BinderHub / JupyterLite white label SaaS?, hosting data and archived reproducible container images on IPFS and academictorrents and Git LFS, hosting {SQL, SPARQL, GraphQL,} queries and/or a SOLID HTTPS REST API and/or RSS feeds with dynamic content but static feed item URIs and/or ActivityStreams and/or https://schema.org/Action & InteractAction & https://schema.org/ReviewAction & ClaimReview fact check reviews, W3C Web Notifications, CRM + emailing list, keeping a legit cohort of impactful peer reviewers,

#LinkedData for #LinkedResearch: Dokieli, parsing https://schema.org/ScholarlyArticle citation styles,

> keeping a legit cohort of impactful peer reviewers, [who are time-constrained and unpaid, as well]

"Ask HN: How are online communities established?" https://news.ycombinator.com/item?id=24443965 re: building community, MCOS Marginal Cost of Service, CLV Customer Lifetime Value, etc