News for hackers. Stuff that matters.
Re:Awk: `Begin { ` Part 1 by mehrdadn on Friday October 30, 2020 @08:05AM (#24941336)
Tangent, but another interesting problem with date parsing besides lack of standardization is that it's ridiculously context-sensitive.

For example, if I schedule an event for Oct 1-Jan 1 right now, the most likely (least surprising) parse would be Oct 1, 2021 through January 1, 2022. Which is both surprising because parsing Oct 1 depends on the current date, and because the parsing of Jan 1 depends on the parse of Oct 1, and because both of these depend on the fact that you're scheduling (in all likelihood) a future event, rather than describing a past event. So whatever date parser you use (no matter how good it is at handling different notations) will return garbage if it doesn't take all these into account.

Re:Awk: `Begin { ` Part 1 by komali2 on Friday October 30, 2020 @08:34AM (#24941560)
Heh, interesting, I actually assumed it would be oct 2020 to Jan 2021, and that you were describing an event already in progress.
Re:Awk: `Begin { ` Part 1 by mehrdadn on Friday October 30, 2020 @09:26AM (#24941960)
Sure :-) even then, you have even more fun... imagine if the next item was Sep 1-Dec 1 and the list was known to be sorted! The parser would need to be stateful to be able to disambiguate!!

Even worse (well, maybe similar, but more surprising): imagine if on Feb 28, you parse "Feb 29-Mar 1"... both of those dates could land in an entirely different year than "Feb 28-Mar 1" would, depending on whether the current year is a leap year...

I dare say I have not yet seen a single parser in my life that handles such issues. In fact I don't think I've seen a parser that can parse a date interval, or that can even do "parse this string assuming it is after that date", or anything like that.

And all of these problems are before we even consider time zones, leap seconds, daylight savings, syntactic ambiguities, etc... not just how they affect individual dates, but also ordered dates (/intervals) like above...

Re:Awk: `Begin { ` Part 1 by enriquepablo on Friday October 30, 2020 @10:30AM (#24942583)
I would say that you are mixing up "parsing" and "calendaring (or something of the sort)". As far as I understand parsing is syntactic analysis, i.e. going from a linear structure to a more complex structure (usually a tree); it should not add to the tree anything that was not in the linear structure. It shouldn't consider a semantic context (such as the current date) to produce an ast.
Re:Awk: `Begin { ` Part 1 by mehrdadn on Friday October 30, 2020 @06:18PM (#24947402)
By parsing I mean it in the usual sense for a date... strptime, Date.Parse, etc... i.e. turning a string to a date (or multiple dates). You can call it something else if you'd like.
Re:Awk: `Begin { ` Part 1 by monsieurbanana on Friday October 30, 2020 @11:33AM (#24943299)
> I dare say I have not yet seen a single parser in my life that handles such issues. In fact I don't think I've seen a parser that can parse a date interval, or that can even do "parse this string assuming it is after that date", or anything like that.

You can google "nlp date extraction" maybe? I've used libraries in the past that do this.

Note: they're far from being perfect. I ended up not using any, as each had their weird corner cases.

Here's an old one: http://natty.joestelmach.com/try.jsp#

I tried with: "first week of december to end of january"

And it gave me: Tue Dec 01 15:31:30 UTC 2020 Fri Jan 31 15:31:30 UTC 2020

Edit: this one seems more polished: http://nlp.stanford.edu:8080/sutime/process

Re:Awk: `Begin { ` Part 1 by mehrdadn on Friday October 30, 2020 @06:19PM (#24947410)
Cool, thanks for the tip!
Re:Awk: `Begin { ` Part 1 by jandrese on Friday October 30, 2020 @01:21PM (#24944571)
Parsing anything that was originally developed by humans writing it on paper (or clay tablet, or whatever) is a nightmare. Natural means chaotic.

If your CSV file contains any field entered by humans AWK isn't going to be powerful enough to parse it at scale. Someone somewhere is going to have the name 'Mbat"a, Sho,dlo' in some bizarre ass romanization (and this assume you're not accepting Unicode, which is a whole other can of worms that AWK is not prepared to deal with) that breaks your parser.

Re:Awk: `Begin { ` Part 1 by mulmen on Friday October 30, 2020 @02:49PM (#24945572)
I'm saving that as a test case name but making some small adjustments.

'Mbaät\"a, Sho,dló'

"a" followed by "ä" because some suggest encoding umlauts as double characters. When you decode that does it go first or second?

Answer: use Unicode.

Throw an escape character in there, before the quote character just to make it interesting.

This is a good time for everyone to review: https://www.kalzumeus.com/2010/06/17/falsehoods-programmers-...