fiddlosopher

I grew up in Los Alamos and remember shopping at that location when it was a Piggly Wiggly grocery store. There is much more detail here about its subsequent incarnation as the Black Hole and the activism of the man who owned it: https://en.wikipedia.org/wiki/Ed_Grothus

The web app can be found at https://pandoc.org/app . This has almost the full power of the command-line app, subject to the limitations imposed by the WASM sandbox. Thus:

- the app cannot fetch resources using HTTP

- JSON filters are not supported, as they involve external programs

- Lua filters are supported, but only if they don't try to do system IO operations or run external programs

- Output to PDF is supported only via Typst (using the WASM version of Typst).

Pandoc does know how to expand LaTeX macros. For example, given the LaTeX

  \newcommand{\pair}[2]{\langle #1, #2\rangle}
  $$\pair{a^2}{\frac{\pi}{2}}$$

pandoc will give you the Typst

  $ chevron.l a^2 \, pi / 2 chevron.r $

which is correct. Tylax, on the other hand, seems to have problems with this example, producing

  $ angle.l^()frac(pi,)angle.r  $

which does not compile with typst. Going the other direction, pandoc also understands typst scripting. For example, from

  #let count = 8
  #let nums = range(1, count + 1)
  #let fib(n) = (
    if n <= 2 { 1 }
    else { fib(n - 1) + fib(n - 2) }
  )

  The first #count numbers of the sequence are:

  #align(center, table(
    columns: count,
    ..nums.map(n => $F_#n$),
    ..nums.map(n => str(fib(n))),
  ))

pandoc produces this LaTeX:

  The first 8 numbers of the sequence are:

  {\def\LTcaptype{none} % do not increment counter
  \begin{longtable}[]{@{}llllllll@{}}
  \toprule\noalign{}
  \endhead
  \bottomrule\noalign{}
  \endlastfoot
  \(F_{1}\) & \(F_{2}\) & \(F_{3}\) & \(F_{4}\) & \(F_{5}\) & \(F_{6}\) &
  \(F_{7}\) & \(F_{8}\) \\
  1 & 1 & 2 & 3 & 5 & 8 & 13 & 21 \\
  \end{longtable}
  }

With the same input, Tylax produces:

  The first 8 numbers of the sequence are:

  \begin{center}

  \begin{tabular}{|c|}
  \hline
  \hline
  \end{tabular}\end{center}

which is just an empty table.

Pandoc developer here. Two comments: (1) to convert equations pandoc uses the texmath library, which I also wrote (https://github.com/jgm/texmath). Compiling texmath with `-fexecutable` will give you a standalone executable that just converts the equation (and doesn't add the `<p>` element or anything extraneous). Compiling it with `-fserver` will give you a webserver that converts equations. (2) Regarding the bug, note that it's not an empty `<mo></mo>`. There's an invisible U+2061 character ("function application") inside. We don't want to take that out, but it looks like putting the `<mi>` and the `<mo>` together in an `<mrow>` will also solve the problem. I'll fix this.

An even older one is my gitit, started in 2008!

https://hackage.haskell.org/package/gitit-0.15.1.2

It doesn't limit itself to markdown, nor to git (you can use darcs, hg, or even sqlite). A bit long in the tooth, though -- I stopped working on it once spam started to make self-hosted public wikis untenable.

This was my first computer, too. I have fond memories of programming it in 6502 machine language and saving programs on cassette tape. I still have it, along with the power supply my grandfather built from the specs they provided in the user manual. And it still works!

As I explained, if you don't want raw HTML, use the second form:

  pandoc -f html -t gfm-raw_html https://www.fsf.org | grep div

gives no output. (If you get different results, it is possible that you are using an earlier pandoc version and something changed in this regard.)

If you don't want pandoc's fenced divs, just choose a markdown dialect that doesn't support them: e.g.,

    pandoc -f html -t gfm https://www.fsf.org

Or, to get rid of the divs and spans altogether, disable raw HTML as well:

    pandoc -f html -t gfm-raw_html https://www.fsf.org

In fact, both claims are misleading. There are two implementations, and they implement distinct flavors with syntactic differences: https://consolelog.gitee.io/docs-asciidoctor/asciidoc-asciid... Markdown has a smaller flavors/implementations ratio!

Oh, but there is!

    pandoc --shift-heading-level-by=-1 input.md -o output.docx

This will promote level-2 headings to level-1, and promote a level-1 heading at the top of the document to the document's title.

The motivation for this choice is not "convenient parsing" but deeper considerations of language design. As explained at https://github.com/jgm/djot#rationale , this choice follows from two desiderata: (1) "The syntax should compose uniformly, in the following sense: if a sequence of lines has a certain meaning outside a list item or block quote, it should have the same meaning inside it." (2) "The syntax should be friendly to hard-wrapping: hard-wrapping a paragraph should not lead to different interpretations, e.g. when a number followed by a period ends up at the beginning of a line." The document explains the compromise we made in commonmark to avoid the need for blank lines. Djot tries to be more principled.

> Markup language which completely falls over this is Markdown. There’s no way to express generic tree structure, conversion to HTML with specific browser tags is hard-coded.

This isn't really a fair criticism. True, the original Markdown.pl did not produce a generic tree structure, but that's a fact about the program, not the syntax it parses. Many Markdown and Commonmark implementations do support creation of an abstract syntax tree. Pandoc has done this for the last 17 years. It also provides nestable, generic containers as a syntax extension.

> It feels like there’s a smaller, simpler language somewhere

Here's my attempt: <https://djot.net>.

Note also that TeX math can contain \text{..} which can itself contain $-delimited TeX math, e.g. $x = \text{my $y$}$. This currently breaks the GitHub implementation.

In pandoc's extended Markdown you can just write

    # Usage

    See [Usage].
    

https://pandoc.org/MANUAL.html#extension-implicit_header_ref...

In fact, pandoc now does have an ipynb reader and writer. You can convert between ipynb and any of the formats pandoc supports, and you can manipulate the AST using filters.

See https://github.com/jgm/pandoc/issues/1456.

AsciiDoc is a nice project, but I think that pandoc's variant of Markdown has a lot of advantages over AsciiDoc for academic writing. Let's just compare support for math and citations, for example. Pandoc's citation support is output-format independent: you can write the citations, specify a CSL stylesheet and bibliography, and you'll get the same output in every format pandoc supports. In AsciiDoc, as I understand it, if you want automatic citation support you need to use LaTeX, and then you're limited to output in LaTeX and PDF. Similarly for math: pandoc actually converts your LaTeX math to MathML (for formats that like that) or to native Word equations (for docx); in AsciiDoc, as I understand it, your LaTeX math will work if you target LaTeX, but if you want MathML you need to put MathML in the source. If you want to target both, then maybe you need two different source documents?

I don't have much experience with AsciiDoc, but I've encountered other limitations in writing pandoc's AsciiDoc renderer. For example, I didn't see any way to include multiple paragraphs (or other block content) in an AsciiDoc footnote. That's a deal-breaker for many academics.

I also want to emphasize something that is often not mentioned in comparisons of Markdown and AsciiDoc. As John Gruber emphasizes in his Markdown documentation, Markdown emphasizes ease of reading in source format. AsciiDoc has different priorities, and it sacrifices the readability of the source document to get them. Here's an example involving nested lists, from the AsciiDoc manual:

    1. List item one.
    +
    List item one continued with a second paragraph followed by an
    Indented block.
    +
    .................
    $ ls *.sh
    $ mv *.sh ~/tmp
    .................
    +
    List item continued with a third paragraph.
    
    2. List item two continued with an open block.
    +
    --
    This paragraph is part of the preceding list item.
    
    a. This list is nested and does not require explicit item continuation.
    +
    This paragraph is part of the preceding list item.
    
    b. List item b.
    
    This paragraph belongs to item two of the outer list.
    --

Markdown equivalent:

    1.  List item one.
    
        List item one continued with a second paragraph followed by an
        Indented block.
    
            $ ls *.sh
            $ mv *.sh ~/tmp
    
        List item continued with a third paragraph.
    
    2.  List item two continued with an open block.
    
        This paragraph is part of the preceding list item.
    
        1.  This list is nested and does not require explicit item continuation.
    
            This paragraph is part of the preceding list item.
    
        2. List item b.
    
        This paragraph belongs to item two of the outer list.
    

Note that the AsciiDoc is at least as easy to write, perhaps easier, because you don't need to worry about indentation. But the Markdown source is more readable; the indentation makes clear the structure of the list in a way that mirrors how it would be displayed in a browser or on the page.

"For example, they do not plan to add syntax highlighting blocks (the ```some code``` on GitHub) to their implementation, because they believe that it is outside the scope of markdown."

To correct the record, fenced code blocks have been there from the beginning: http://spec.commonmark.org/0.18/#fenced-code-blocks.

Although I'm involved in both these enterprises, let's not confuse their goals.

CommonMark is currently focused on the task of giving a decent spec for the core syntax and robust, efficient implementations; extensions will wait til that project is done, but certainly aren't ruled out.

Pandoc has always been in the game of extending the feature set. Here are just some of the Markdown extensions pandoc supports: LaTeX math (which can be rendered in a variety of formats, including native Word and MathML), LaTeX macros, inline LaTeX, automatically numbered examples and cross-references to these, automatically generated citations (using CSL styles), super and subscripts, strikeout, figures, YAML metadata, definition lists, several styles of tables, fenced code blocks with syntax highlighting, header identifiers, and footnotes. scholdoc just adds a few things on top of all this (and many of them could be implemented in pandoc filters). As noted in one of the other comments on this thread, most of the features scholdoc adds are under active discussion in pandoc as well. So it's not that pandoc and scholdoc have different aims; pandoc just moves more slowly, because it has to worry about how features are implemented in many more output formats, and it operates under some other constraints that scholdoc rejects (e.g. trying to avoid the use of English words like "Figure" for syntax cues).

Your comments (coming from someone who has actually tackled this surprisingly difficult task) are some of the most valuable we've received; having them on the Discourse forum would be great.

We considered writing the spec in the state machine vein, but I advocated for the declarative style. It may be worth rethinking that and rewriting it, essentially spelling out the parsing algorithm. As you suggest, a parallel document could be created for writers.

I'll need to study your spec further to see what the substantive differences are.

If you think there are ambiguities, please comment at http://talk.standardmarkdown.com. This is meant to be a provisional spec, up for comment. There is undoubtedly room for improvement.

The C and javascript implementations use a parsing algorithm that we could have simply translated into English and called a spec. (That's the sort of spec vfmd gives.) But it seemed to us that there was value in giving a declarative specification of the syntax, one that was closer to the way a human reader or writer would think, as opposed to a computer.

Re (3): we have an asterisk which can open emphasis. So, to see if we have emphasis, the rules say to parse inlines sequentially until an asterisk that can close emphasis is reached. The first inline we come to is [b*](url), which is a link. There's no closing asterisk, so we don't have emphasis, but a literal asterisk followed by a link.

Re (1): I believe you are right that the case of a referenc e definition before a setext header line should be clarified. However, the other case seems clear enough. ~~~ starts a fenced code block, which ends with a closing string of tildes or the end of the enclosing container. The underline would be included in that code block either way.

Re (2): I believe the talk of precedence may be misleading here (I thought it would be useful heuristically). The basic principle of inline parsing is to go left to right, consuming inlines that match the specs. This resolves all of these cases. Perhaps the talk of precedence should be removed.

I am no stranger to formal specifications. I wrote what I think was the first PEG grammar for markdown (peg-markdown, which came to be used as the basis for multimarkdown and several other implementations). PEG isn't a good fit, especially for block-level parsing. It almost works for inline-level parsing, but there are some constructs (like code spans) that can't be done in PEGs. It might be worth specifying inline parsing in a pseudo-PEG format to avoid worries like those you've expressed.

For a while I was working on a Haskell-based macro system that compiles TeX-like source to HTML and LaTeX (and whatever else you like) (https://github.com/jgm/HeX). You'd write your macro definitions in a Haskell program, then run this program on the textual source to produce output. What's nice about this is that you can use Haskell's module system to organize your macros, and Haskell's type checker to check for errors. The macros are just Haskell functions, and the types of the functions determine what kinds of arguments are consumed by the macro. I'd like to come back to this some time.

All the vowels in Navajo can have a hook under them (indicated nasal tone) and an acute accent above them (indicating higher pitch). The "ł" is not pronounced as in Polish. It is a voiceless alveolar lateral fricative. Try to pronounce l and s at the same time and you'll be in the right ball park. Navajo vowels also come in long and short duration; a doubled vowel means long. (I know all of this because I designed a font for Navajo in the early 90s, while working on the Navajo reservation. I don't know if it ever got much use, though I once saw a children's book typeset in it. Most of my friends who spoke Navajo did not read the language, probably because there wasn't enough written in Navajo to make it worth while for those who had learned to speak it growing up to learn to read it as well.)

Edit: I didn't comment on ń. According to "An Introduction to the Sound System of Navajo," by Ken Hale and Lorraine Honie (MIT unpublished ms), "the sequences /ní/ and /ni/, when occurring before consonants, are often pronounced without the vowel; instead, the nasal becomes syllabic and carries the tone originally carried by the vowel. When this happens, the tone mark is transferred to the nasal -- thus /ní/ is written [ń]; in the case of low-toned /ni/, a special tone mark [`] is used for the syllabic nasal..." I was excited to find all of this unpublished material from the late Ken Hale here: http://www.swarthmore.edu/SocSci/tfernal1/nla/halearch/halea...

Pandoc doesn't have an "include" feature, but it's easy enough to add this with a preprocessor. See http://randomdeterminism.wordpress.com/2012/06/01/how-i-stop....

Pandoc does provide a way to link to section headings, but it doesn't yet have a generic system for autogenerating numbers and producing references to these, like LaTeX's \label{} and \ref{}. It does, however, have a system for creating running example lists: http://johnmacfarlane.net/pandoc/README.html#numbered-exampl.... This can be used for numbering equations and referring back to them, but it is considerably less flexible than LaTeX.

Pandoc has extensive support for automatic citations using CSL stylesheets: http://johnmacfarlane.net/pandoc/README.html#citations. BibTeX and BibLaTeX files can be used as the database, or YAML citations can be included in the markdown document itself.

LaTeX formulas can be embedded in markdown. They can even use macros defined in the document. Formulas can be converted to MathML or native Word equation objects, depending on the output format.

Pandoc can also be extended using "filters" that operate directly on the parsed AST. Here's an example of a filter that finds tikz diagrams and converts them to embedded images that can be displayed on the web: https://github.com/jgm/pandocfilters/blob/master/examples/ti....

The recently-released pandoc 1.12 (http://johnmacfarlane.net/pandoc) supports reveal.js as an output format. I've been using it for some time to give talks, and it works well. The 2-D navigation structure makes it easier to find slides when there are questions.

pandoc mybook.md -o mybook.epub

This works too:

    pandoc -t markdown http://example.com