The entire US Code is now online in XML

Re:The entire US Code is now online in XML by mjn on Wednesday July 31, 2013 @04:39PM (#6136684)

Git's built-in diff algorithm is particularly bad for text. Since it's aimed at line-oriented code, it does line-based diffs, which is horrible for ASCII text that is reflowed, because every line in a paragraph will show up as changed for a small change.

Example: https://github.com/divegeek/uscode/commit/1fb2d83137dad1c6ca...

What's happened is that "Section 2" was moved later in the sentence, abbreviated as "Sec. 2", "of" was deleted, and "act" was capitalized:

    Section 2 of act July 30, 1947, ch. 392, 61 Stat. 674, provided...

    Act July 30, 1947, ch. 392, Sec. 2, 61 Stat. 674, provided...

The rest of the paragraph is unchanged, but git shows a 6-line diff with the entire paragraph replaced. GitHub attempts to do some word-based highlighting (see the timestamp lines), but it falls down on most of these paragraphs. Wikipedia's diffing tends to work better for this kind of thing; I'm not sure what they use. The upshot is that the number of lines changed may be a 5-10x overestimate.

[ Parent | on HN ]

Re:The entire US Code is now online in XML by gmartres on Wednesday July 31, 2013 @04:53PM (#6136762)

> Since it's aimed at line-oriented code, it does line-based diffs

You can do word diffs with git:

    git diff --word-diff=color

[ Parent | on HN ]

Re:The entire US Code is now online in XML by DannyBee on Wednesday July 31, 2013 @05:24PM (#6136956)

It's still recreating a word based diff from a line based diff.

See diff.c line 793 for how it works.

[ Parent | on HN ]

Re:The entire US Code is now online in XML by azernik on Wednesday July 31, 2013 @06:17PM (#6137188)

It may be doing that conversion, but the conversion works. For example, committing the following text (with line breaks), then joining it all into one line, shows no differences when using 'git diff --word-diff'.

  Test the first. This will check if reflowing
  text actually produces git word-diff weirdness,
  or if it's actually decent.

The line does get reproduced on the terminal (a line diff was seen), but no text is shown in green or red to indicate an actual change.

[ Parent | on HN ]

Re:The entire US Code is now online in XML by mjn on Wednesday July 31, 2013 @06:31PM (#6137258)

Just checked with the old and new versions of Title_09.txt and you're right, --word-diff does the right thing. It echoes all the changed lines to the terminal, but it only marks up (and colors, in color mode) the changed words:

          [-Section 2 of act-]{+Act+} July 30, 1947, ch. 392, {+Sec. 2,+} 61 Stat. 674, provided that

Wonder if there's a way to enable that behavior on GitHub? And/or to generate repository activity statistics based on changed words rather than changed lines?

[ Parent | on HN ]

Re:The entire US Code is now online in XML by DannyBee on Thursday August 01, 2013 @01:21AM (#6138826)

Sure, but this particular test tells you nothing. You need one where the "line diff" has identified the wrong set of changes (IE it has decided two sets of text look close enough that one is really a change into the other, even though that's not what historically happened).

[ Parent | on HN ]

Re:The entire US Code is now online in XML by azernik on Saturday August 10, 2013 @12:24AM (#6190045)

That is never the error I see in line-based diffs; because they're taking a much larger chunk of text to be atomic and treat irrelevant characters as relevant, they tend to give false positives (seeing two lines as completely different when in fact they're slightly modified versions of each other; or seeing two blocks of text as different when one is a reflowed version of the other.)

[ Parent | on HN ]

Re:The entire US Code is now online in XML by stock_toaster on Wednesday July 31, 2013 @06:33PM (#6137269)

If there are sporadic line differences, for that git diff supports different algorithms than the default. patience or histogram may work better.

as far as words in lines, you do have a point.

[ Parent | on HN ]