News for hackers. Stuff that matters.
Re:Rob Pike: "Current Unix tools are weakened by the built-i... by its_so_on on Thursday June 14, 2012 @04:07PM (#4113285)
I like your observation but I don't like your conclusion.

I think the major thing missing from the world is easy piping a la Unix. You can't pipe your list of paying customer's email addresses to paypal at the command line with some switches - really, no matter how many switches you use - and bill each one the amount stated, because "paypal" is not a commandline app. You can't pipe the results of some long-running analysis to Twitter to announce that you've finished computing it, no matter how many switches you add, because twitter is not a commandline app.

The direction you're suggesting we take things is, in fact, a fuller API. These exist. They're slower and worse.

The amazing thing about text is that it's a lowest-common denominator. Think of communicating with a person.

Communication is faster with a mind-meld where you're looking at another person's face and picking up micro-expressions and body language. That's also the easiest thing to misinterpret.

When you pipe through such a 'human-readable' lowest common denominator, you're actually setting the ground for a very dynamic and versatile channel.

I don't know what the easy solution is to the problem you bring up, but I don't think your proposal is it.

Maybe there is no easy solution. Several unicode characters to abstract away the tab character and the newline character to instead n dimensions of characters, and corresponding negative characters so you can put something on a 'line' out of channel, (negative tab comment, tab, text, newline repeat) would probably solve some issues but is too abstract to even discuss. What we have really isn't that bad.

Re:Rob Pike: "Current Unix tools are weakened by the built-i... by haberman on Thursday June 14, 2012 @04:42PM (#4113424)
> The direction you're suggesting we take things is, in fact, a fuller API.

Nothing about my proposal has anything to do with an API. I'm just proposing a structured stream of data instead of an unstructured one.

> They're slower and worse.

I am proposing avoiding a serialize/parse step between every pair of pipeline elements (or using a more structured/optimized format if a serialization step is desired). Doing less work cannot possibly be slower.

> The amazing thing about text is that it's a lowest-common denominator. Think of communicating with a person.

Communicating with a person is an endless process of content negotiation. What are you and I talking about? Using our interface of plain text, we could be talking about literally anything. I could suddenly start talking about kumquat farming in Russia. At that point you could decide to follow suit and weigh in with your opinions on kumquat farming, or you could stop talking to me altogether because I've gone off-topic. If I start speaking complete jibberish, you could try to learn the language that I'm speaking, or you could start doing something equally nonsensical.

You and I can respond to unexpected communications in useful ways because we are fully autonomous, intelligent, sovereign beings that are capable of learning, creativity, and curiosity. I don't know about you, but I don't want my software to be autonomous or react to unexpected situations in unpredictable ways.

Data processing software should be as simple, predictable, and deterministic as possible. To use your example, if I somehow got an email address in my list called "send $1000 to Lucy," I don't want PayPal to decide to get smart and interpret the invalid email address as a command to send money to Lucy.

Re:Rob Pike: "Current Unix tools are weakened by the built-i... by 205guy on Friday June 15, 2012 @07:12AM (#4115926)
> Communicating with a person is an endless process of content negotiation. What are you and I talking about?

Right. So how does grep know it's talking to ls about dir contents with a (disk) size field and not talking to ps about process table contents with a (mem) size field? Some sort of slower content negotiation, I presume.

> cannot possibly

I suggest that every time you think that, you double-check your assumptions. In this case, I think you're just pushing all the extra hard work off onto some other part of the algorithm.

Re:Rob Pike: "Current Unix tools are weakened by the built-i... by darkarmani on Friday June 15, 2012 @12:36PM (#4117332)
His point is that he's pushing off the extra hard work to be done only once, rather than at every pipe. I'm not sure how he needs to recheck his assumptions there since he stated them.
Re:Rob Pike: "Current Unix tools are weakened by the built-i... by Arelius on Thursday June 14, 2012 @08:12PM (#4114274)
> I'm just proposing a structured stream of data instead of an unstructured one.

That's basically what an API is.

Re:Rob Pike: "Current Unix tools are weakened by the built-i... by darkarmani on Friday June 15, 2012 @10:11AM (#4116546)
So things like CSV, XML, and JSON are APIs now?
Re:Rob Pike: "Current Unix tools are weakened by the built-i... by Arelius on Saturday June 16, 2012 @02:45AM (#4120033)
Can any of those not be an interface to an application layer?
Re:Rob Pike: "Current Unix tools are weakened by the built-i... by haberman on Saturday June 16, 2012 @02:54PM (#4121315)
Plain text also "can be an interface to an application layer" (see SMTP, IMAP, POP, etc). Your argument is meaningless.
Re:Rob Pike: "Current Unix tools are weakened by the built-i... by mattdeboard on Thursday June 14, 2012 @10:08PM (#4114592)
You are confusing the "how" (the API specification) with the "what" (the data the API exists to provide access to).
Re:Rob Pike: "Current Unix tools are weakened by the built-i... by Arelius on Saturday June 16, 2012 @02:47AM (#4120036)
> I'm experimenting with the idea of letting people specify the output of command-line utilities as a Protocol Buffer schema, for example:

Looks like an API spec to me

Re:Rob Pike: "Current Unix tools are weakened by the built-i... by nnnnni on Thursday June 14, 2012 @08:40PM (#4114354)
Re:Rob Pike: "Current Unix tools are weakened by the built-i... by xtractinator on Thursday June 14, 2012 @09:11PM (#4114445)
No, they have command line interfaces.
Re:Rob Pike: "Current Unix tools are weakened by the built-i... by wendsday on Thursday June 14, 2012 @05:38PM (#4113637)
I could not agree more with your comment. There's not a problem with text. Why do people pretend there is? They often only achieve making other people's jobs more difficult. Text is what people can read. People do not read binary. When something goes wrong, debugging binary formats becomes insanely cumbersome.

The concept of lines is a human one. It is how humans parse. If humans could parse without needing the concept of a "line" then, e.g., there would be no problems with programming in C which has very shaky support for the concept of "lines".

But there are problems as we all know. It's proof that people do need to think in terms of "lines". Even though the computer does not need them. The only "problem" with this is that people are not computers.

I'm not sure anyone outside of the most unrealistic nerds would agree that this is a "problem".

1. As for the paper, I can think of at least one tool/language to work with text that does keep data in a binary format while one performs a series of transformations. It's not line-based. And I would guess that Mr. Pike does not know how to use it. It's fast and efficient. Probably faster than sam.

Re:Rob Pike: "Current Unix tools are weakened by the built-i... by haberman on Thursday June 14, 2012 @06:00PM (#4113742)
> I could not agree more with your comment. There's not a problem with text. Why do people pretend there is?

My problem is not with text per se, but with unstructured text. I'm fine with JSON in cases where efficiency is not a top concern.

Let me ask you this; how would you do the equivalent of this hypothetical command?

  $ ls | structured-grep 'file.size > 1M'
The answer is that you can't in today's world without writing a parser (or some code that calls readdir/stat manually). That is the problem with unstructured text.

Text formats like CSV that seem simple actually end up being hugely complicated once you push them to their limits. Nothing is worse than software that breaks once something unexpected happens, like a string that contains an embedded comma.

Re:Rob Pike: "Current Unix tools are weakened by the built-i... by e12e on Thursday June 14, 2012 @06:40PM (#4113910)
"ls" prints a list of files. "find" knows about the filesystem and can print a list of files based on file metadata and a path (such as a filename). If you have a list of files (eg from "ls") -- you would need to look up the metadata you want to filter on; it is not part of ls' interface to give them to you directly:

  ls | find -size +1M
On the other hand, if you want to make list of files and their sizes, that can be stored, sent over the network, etc -- and then filer that list, you can do:

  ls --size | awk 'file_blocks=$1 $file_blocks > 1024 { print $2 }'
For most tasks, I think the fact that any human can look at the output from eg: "ls --size" and then produce a valid test dataset in the same format is more valuable than having to explicitly "cast" the metadata while processing.
Re:Rob Pike: "Current Unix tools are weakened by the built-i... by swa14 on Thursday June 14, 2012 @06:40PM (#4113911)
>>Let me ask you this; how would you do the equivalent of this hypothetical command? >> $ ls | structured-grep 'file.size > 1M'

# find . -maxdepth 1 -size +1M

really, flat text is fine. Maybe it's not perfect, but it's good enough that most want to not add any complexity to it that would make it incompatible.

And if you really need that complexity, it's usually worth whipping up a parser for.

Re:Rob Pike: "Current Unix tools are weakened by the built-i... by haberman on Thursday June 14, 2012 @07:02PM (#4114010)
"find" is a poor man's "structured-grep." It provides a bunch of functionality for filtering a result set, but is totally specific to lists of files. You can't use find with ps, netstat, iptables, ifconfig, or any other command-line program that produces a list of records.

> And if you really need that complexity, it's usually worth whipping up a parser for.

No work is worth doing if it could just as easily have been avoided.

The vision in my head has less complexity than the status quo, not more. How many flags does "find" have? One for every field name you can filter/sort by. Done right, a "structured-grep" that can grep on any field sent to it is much, much simpler.

Re:Rob Pike: "Current Unix tools are weakened by the built-i... by swa14 on Friday June 15, 2012 @03:19AM (#4115445)
I think I see your point, and it has merit ... but ... :)

in a way, the 'find' program is like what you envision, except it's just for files. That means that someone somewhere along the road, had the same idea/problem (but limited to files) as you and whipped up a parser to produce that meta-data. That particular parser proved to be so useful to so many people, it became it's own program.

There's more than 40 years of sofware-"evolution" contained in unix, and apparently retrieving structured on the command-line has only proven universally useful for files. Unix has outlived many at the time more modern operating systems, and I think it's partly because it lacked a "grand unifying vision". Instead it has a "small, quick&dirty unifying vision" of which "flat text processing" on the commandline is a central part. It has turned out to be the greatest common denominator for being able to write programs, that might be quick and dirty oneliners, but ultimately they got the job done. And only those tiny little utilities that proved to be universally useful were developed into bigger more stuctured programs.

I'm not saying you idea is without merit, but it does apply the principle of "this concept A is useful for this particular problem-set. Let's apply it natively to all problem-sets so it can be useful there too!" ( in a way like Java did with the OO concept).

When simpler visions and concepts are actually implemented in the end it usually turns out one has been replacing witchcraft with voodoo.

Re:Rob Pike: "Current Unix tools are weakened by the built-i... by wendsday on Thursday June 14, 2012 @09:00PM (#4114416)
The solution I often contemplate is to address the process by which the data is created. (Let's assume we're talking about data you would find on the web.) That is, why don't we impose rules on that process? Why can't we "mandate" that the output be structured data from the get go? Instead we allow vast quantities of unstructered data to be created and then we try to normalize it. I know this is a radical view, but it could make sense in some circumstances.

I disagree with your comment about CSV. But it's impossible to have a meaningful argument unless you provide an example: Give me a job to do, a CSV file and let me have a go at it. I'm serious. Post a link to a CSV file, define a task and let's see what we can do just using plain ole UNIX. Could be a fun exercise.

As for your hypothetical command, I do not understand what is so difficult about this. The stat command is what you want, not ls. No self-respecting UNIX user would parse ls when he can use stat (I recommend the BSD one over GNU.).

But here's what I would do:

1. If your UNIX filenames have spaces in them, rename them. There is no sensible reason to leave spaces in filenames in UNIX. Fix this first before it becomes a problem.

2. Write a one-liner and save it as a function, perhaps in your .profile, or maybe in RCS, or save it as a script. There's so many ways to manipulate output as a stream. Pick one that suits your tastes. That's the beauty of UNIX. Make your own solutions as you go. There is no right or wrong answer. It is a form of customization. My choice will no doubt make some people cringe. Assuming there's no user named "[0-9]M":

   whatever(){ ls -lhS |tr '\011' '\040'|sed '/ [0-9]M /!d' ;}
or save what's between the brackets as a file named "whatever". Maybe you save it in a directory called "x" and add that to your PATH. Then you do

   . whatever
Of course how long the list of files is going to be makes a difference. I might take a different approach if the output was going to be an enormous list.

There are so many ways to get what you're after. The point is that you should be able to tap out a one-liner that does the job. Maybe it takes a few iterations to get the right output. Tweak it until the output is what you want. Viewing command-line history is perfect for seeing the process of creating a one-liner to manipuate output. You can see the line grow incrementally as you build it, until you finally have the output you want. This sort of history allows you to go back to any stage in the process. If you're a vi fan, you can use vi-mode on the command line to move around the line quickly as you edit. Eventually you can hit "v" and edit the thing visually in your EDITOR, then save it. I've built over 700 useful functions this way and the number keeps growing.

I do understand there should be a way to "extract" the file size column the way Pike decribes in the article. To do this I think you have to free yourself from "line-oriented" thinking and imagine another type of structure. And I think using another language you can do it. But for something as simple as this -- manipulating ls command output (cf. manipulating large datasets) -- an "ugly" one-liner suits me fine. The more you use the boring old utilities the more you can get them to do.

Regular Expressions are indeed "crude". But, to me, that is just fine in a lot of cases.

Re:Rob Pike: "Current Unix tools are weakened by the built-i... by mjwalshe on Friday June 15, 2012 @05:27AM (#4115706)
Yes try working on the full OSI Stack you had to learn asn.1 just to beable to read what a concrete decode was doing.

Though my boss did impress me by watching an OSI transaction in flight on our network monitor stop it and point to a dword and say thats wong and its Sprints broken x.400 implimentation.