After over two decades of using Unix and Linux, I ran into jq, a tool for querying and transforming structured JSON pipelines: http://stedolan.github.io/jq/
This can be used to do many of things you demonstrate with PowerShell. Here's an example from the docs:
curl 'https://api.github.com/repos/stedolan/jq/commits?per_page=5' | \
jq '.[] | {message: .commit.message, name: .commit.committer.name}'
{
"name": "Nicolas Williams",
"message": "Add --tab and -indent n options"
}
... more JSON blobs...
I think that PowerShell and jq get this right: More command-line tools should produce explicitly-structured output, and we should have powerful query and transformation tools for working with that output. For day-to-day use, I do prefer jq's use of JSON over a binary format.
https://github.com/raine/ramda-cli
The example from before would look like this:
curl https://api.github.com/repos/stedolan/jq/commits\?per_page\=5 |\
R 'map -> message: it.commit.message, name: it.commit.committer.name'jq looks very interesting though note that it builds upon an underlying notion of text to serialize JSON.
As for binding to .NET, I don't think that's very limiting. A surprising amount of stuff is already easily hosted in .NET.
I would argue that all PS needs to be more competitive is: Ports to Linux/*BSD/OSX and a terminal implementation for Windows that doesn't suck.
Cmd.exe is a piece of shit that needs to die:
- Command editing. Is support for at least ^A, ^E, ^W, and friends too much to ask?
- Completion. Who wants to cycle through 4253 possibilities one at a time?
- Copy/paste. Programs actually STOP executing when you select text, like as if you ^Z in Unix. Even with quick edit enabled so you don't have to ALT-SPACE-E-ENTER<select stuff>ENTER, the fastest way to paste is 4 keys: ALT-SPACE-E-P.
- No SSH. Microsoft is addressing this. It's borderline criminal that Windows doesn't just ship with OpenSSH server already.
- No screen/tmux. I can't even talk about how it deals with resizing without using a lot of profanity.
- Lack of echo by default is seriously annoying.
In short, make the terminal feel like Putty and the editing/completion features feel like bash and I think PS could give all existing shells a run for their money.
As a Linux user / developer, it's surprising to hear colleagues talk about how important it is to separate content from presentation regarding, say, TeX, but ignore the benefits of Powershell doing the same. Unix users actually expel cycles trying to express things like times and file size as regular expressions to operate on strings, rather than dealing with them using their real data types.
A big part of this is that remoting has sucked - most webb app developers don't use Windows - so a lot of people who should have been able to find out for themselves hasn't been able to. Microsoft supporting SSH (as they announcedd recently) should fix that.
This debate never ends because, like the static vs dynamic types debate, it ignores the main tradeoff in favour of a one-size-fits-all solution. In both debates (separation and types) the tradeoff can be expressed most generally as an up-front vs deferred time investment. Static types require more up-front time investment with the advantage that they save time later. The same goes for having separated content from presentation.
Now, in light of the tradeoffs above what is the appropriate choice to make? That depends on how much time you're going to spend writing the script/document/whatever and how much time you're going to spend running/maintaining it later. For scripts, it makes no sense to have a heavyweight type system if you're only going to run the thing once and throw it away. Likewise, for documents it makes no sense to put in the extra time planning all the styles if you're only going to make a document (say, a shopping list) and then throw it away. For a long-term document such as a book or a thesis it makes a ton of sense to use something like TeX.
What exactly is it that saves time? Is it to not have to type as much? In my experience if you use a lang with good type inference you type a lot less with strict typing than without, since you don't have to have unit tests to guard for things like typos/param order/param types.
Static types save time later by giving you the opportunity to do a lot of refactoring with high assurance of correctness. Unit tests bring back some of this ability to dynamic languages but they require extra time to write and maintain the tests.
(Of course, I think text-only Unix tools as a way of interacting with a computer is a fundamentally broken idea). Powershell is an interesting idea, though maybe not the best implementation of that idea.
Everything sharing a common representation does not make types less useful. I would rather my program crash then confuse Bid and Ask prices.
"And for interactive scripring, compilation feels a bit strange too."
I wonder if you've used a REPL for a typed language. I agree that peeling out a compilation step would be odd, but adding types doesn't actually have to change the interaction that much.
Well,
precoding :: Int -> Int -> Int -> (Int, Int) -> Precoding
rsa_dp :: Integer -> Integer -> Integer -> Either RSAError Integer
compile :: String -> String -> String -> Map Name Interface
-> Either String (Interface, String)Linux/Unix can't even get the concept of "one thing" right. Sometimes it's separated by spaces, sometimes by lines (and then you need the IFS gore to make it behave the way we want)
I mean, the shell is great, once you get around all those idiosyncrasies, but we can evolve.
"Do one thing and do it well" is about the goal of the tool, not about its features. With all its options, `ls` is still just about listing directory contents.
And that motto is still just a guideline, not an absolute rule. What makes the *nix shell great is that it follows that motto enough to be sensible and structured, but not enough that it becomes a burden. Just like a human language does.
ls is an alias for Get-ChildItem which is simply "get me a list of objects attached to the path". That's about as orthogonal as you can possibly muster.
It's actually more like plan 9 than unix.
ps: this lisp machine talk mention how 'programs' (I guess functions) exchanged plain old data rather than serialized ascii streams making things very fast. There are caveats of adress space sharing though but it's another nice hint we could investigate other ways.
Also structured output may lead to more relation tools, less regexful ad-hoc parsing, maybe some kind of typecheck so callers can be notified when they need to be rewritten.
ssh myrouter "ls /var/log" | ? Length -gt 50ΚΒ
irm myrouter {ls c:\logs | ? length -gt 50kb}
During serialization, all properties are serialized, recursively to a configured max depth. The local client will see the deserialized objects - ie they are not "live" any more.
Note that the PS team recently announced that they would support SSH as a remoting mechanism. I suspect that they will still send CliXML objects when using powershell to powershell communication.
Powershell uses an industry standard for http based remote commands. It should be adaptable to SSH.
Serializing structured data is a pretty solved problem, and I wonder if decades of UNIX going out of its way to screw this up is confusing the parent poster. XML is certainly a way to do it that works just fine, but there are a million others. None of them require awk or counting spaces.
It's not particularly surprising that a system designed a long time after the original is more consistent (I wonder how the Plan 9 shell fares in this regard?).
Oh, and JSON is a functional subset of YAML, so you'd still be able to output as JSON and a YAML parser will read it.
Text made sense because text was the universal format. Now we have JSON. In the Windows world .net objects are universal (though I'd prefer JSON).
Sure, the fact that Unix chose text meant they created a lot of different formats, some non-standard, but I don't see this being an issue, except for configuration purposes.
Do you ever use regular expressions to parse unstructured text when using a Unix shell? Do you think 'grep' and thinking of a regex is more or less efficient that using 'where'?
By that time, the communication has already happened.
> JSON is a text format
json.org is owned by Douglas Crockford, who should know what JSON is, since he was the first to specify that format.
Sure, it is harder to pattern-match context-free, and we have a convenient syntax for textual regular languages, but we create tools such as jq so that we can pipe JSON and extract data.
When dealing with output from command line apps I usually take the time to parse out the data I want from the strings and turn them into strong types if I'm writing a script...If I'm just trying to get something done with a command prompt I just do whatever gets it done the fastest.
Are you sure? AFAIK you have to use C++/CLI which isn't C++ and the official examples are either C# or VB.NET: https://code.msdn.microsoft.com/site/search?f[0].Type=Topic&...
That may mean that you have to implement at least some part of it as a .Net class. That may mean that they are doing COM components that inherit certain interfaces...it may mean that they are doing PInvoke...to be honest I haven't looked into it. It may be that it's still internal. Huh. I should look that up.
Half way through a powershell script you can switch to C#, VB, JavaScript if you want and implement a pipeline or shell out to a C++ program, talk to something over the network or even Cygwin if you really want.
I imagine to achieve the default functionality your command must be significantly more verbose, if this philosophy is followed through 100%, or you have a lot of per-configured default formatters (and you have to remember what to pipe to which). Maybe I'm misunderstanding this, it's very neat in principle though.
Take for instance Get-Process:
$p = ps powershell
Type
$p
Handles NPM(K) PM(K) WS(K) VM(M) CPU(s) Id ProcessName
------- ------ ----- ----- ----- ------ -- -----------
359 24 57096 61972 611 2,20 9608 powershell
How does the user know exactly which output he is going to get? The formatting program cannot know anything about default expected output - so it either must be specified explicitly, or objects must distinguish between default_for_formatter_to_output fields and more_verbose_hidden_fields - I'm really not a fan of the amount of man <command_name> using Unix involves, - but is the alternative really much better?
It works pretty well. In the end there is also some convention that helps it work more consistently.
In practice, it is kind of cool to be able to construct new objects as the pipeline progresses so that can customize that behavior. I've found that with some aliases for default commands it can be reasonably succinct as well.
So ls, by default, gives Mode,LastWriteTime,Length,and Name.
But if I check all of the properties I get PSPath, PSParentPath, PSChildName, PSDrive, PSProvider, PSIsContainer, BaseName, Mode, Name, Parent, Exists, Root, FullName, Extension, CreationTime, CreationTimeUtc, LastAccessTime, LastAccessTimeUtc, LastWriteTime, LastWriteTimeUtc, Attributes
Didn't realise it wouldn't line-break!
These cmdlets are excessively small in scope and that makes them intrinsically tied to one another. They're useless without their sibling cmdlets. In effect they're one giant program.
So I completely disagree that these are more unixlike.
I do wonder why they use TitleCase for somethings, and lowercase for others ("ls", "gt"). I think it kind of makes this non-obvious and non-predicable. (reminds me of PHP a bit).
ls --json
ps aux --json
If they did that, all that powershell stuff would become trivial.
Wouldn't even be hard.
There has been added in FreeBSD 11 but I am not sure how many of the commands have been converted.
There's a page [1] on the FreeBSD wiki with a list of converted and ongoing conversion. I don't know if the list is complete but the page was last modified 2015-05-22, so it should be fairly up to date i guess.
When you try to ram a one size fits all approach to everything, you end up with abominations like SOAP with XML. I simply don't get the fascination with trying to use the same tool for every job even if it's not applicable.
The whole xml vs json debate feels pretty pointless, they are for all practical purposes equal (in size, complexity etc). Sure xml has some weird design decisions around namespaces and so on, but if you use it as a simple hierarchical markup it's pretty much equivalent to json with different brackets? and often easier to parse for humans because of closing tags instead of }}} and of course, comments.
The xml-hate I think isn't really xml-hate it's the reaction against the xml-y things from 10-15 years ago: the IBM/Struts/Blah we all had to wade through. These days it feels frameworks pick json over xml even when it's clearly an inferior choice (such as for config files). Json is an object/message markup, not a good human readable config format.
> Text streams are just an ad hoc structured format. Even an out of date format is better than one you have to invent or decode on a per case basis.
I won't argue which one sucks more, XML or JSON. They are both inferior to what you call "ad-hoc structured" text files.
XML and JSON are both hierarchical data models. It has been known for fourty years that these are inferior to the relational model because they make presumptions about the access paths of the consuming algorithms.
Put differently, hierarchical data models provide just a view of the information that is actually there. (Implicitly -- consistency and normalization are not enforced). Relational databases on the other hand are concerned with the information and provide much better mechanisms for enforcing consistency and normalization.
By coincidence, the "unstructured" text files in Unix are just miniature relational database tables. Think passwd, shadow, hosts, fstab, ... . Consistency is not technically enforced (that would be a huge overkill at this level of abstraction), but there are even checker programs like pwck.
A sequence of tables in csv with decent encoding support would go a long way towards a good machine parseable relational text output.
It's really two separate discussions though: what is s good input/output format for Unix-like tools, and what makes a good format for a config file.
I don't see where these are not one single problem. Everything is a file.
> A good standardized relational model format would be cool, and I'm sure such formats exist. Feels like we could do better than spitting out randomly (I.e per- tool) formatted data with so-so encoding support!
I'm actually currently trying to realize such a thing in Haskell, for usage in low traffic websites. There are obvious advantages in text DBs compared to binary DBs, for example versioning.
But I doubt we can do better than current Unix text files if we don't want to lock in to some very specific technology.
A very general format could solve more problems but as I said earlier I think the lack of comments in json makes it sub par as a config format for human editing.
If you need more commenting freedom or flexiblity, why not make another indirection and generate the data from some source which is tailored to your needs? After all, relational data is often not suited for manual input. It's meant for general consumption by computers.
For example, as a sysadmin, passwd / shadow / group is not enough to model my business objects directly. I keep my business data in a custom database, and generate the text files from that.
YAML is a far superior format to XML for config files, and is basically JSON in a readable format.
There's literally zero reasons to choose to use XML where you could use JSON/YAML instead.
Text streams are unstructured, so parsing them is a pain in the ass.
JSON is simple, standardized, and not going anywhere.
The "directories of files" abstraction is versatile and useful, simply because it is a versatile and simple data structure. Streams of text are too limited.
In PowerShell, the `ls` command is a simple command that produces output without flags to control output format, ordering or filtering. Like with higher order functional programming, those tasks are delegated to cmdlets like Format-List, Format-Table, Select-Object, Where-Object.
Some PowerShell examples:
To list the files and directories (children) of a directory:
To get just the names or the full path: To discover which properties are available from `ls`: To select only files larger than 200MB: (ls is alias for Get-ChildItem, gm is alias for Get-Member, ? is alias for Where-Object)It is somewhat ironic that PowerShell is more true to the principle of "do one thing and do it well" than the Unix shells.