For a developer writing a tool, exposing an interface as objects is far easier than having to constantly serialize data into a string. Keep in mind that once you serialize data into a string, you can never change the order of the data or else other tools will break. Not so with a object based interface.
In other system-admin type tasks PowerShell hides complexity. For e.g. finding the IP address of a network interface is simply
Get-NetIPAddress | select IPAddress
ifconfig | perl -nle 's/dr:(\S+)/print $1/e'In summary:
I need to pull data from a company we're partnered with, and get it into our systems. They ship it as SQL Server backup files. OK, no big deal, we can mount those up and run queries to dump stuff out to whatever format we like and ingest it.
Except part of that data is files, which were stored in the DB as binary blobs (in a table along with filenames and metadata). Need to get those out of there and into our storage as real files. OK, no big deal, I can just query for filename and file contents, dump out onto disk and then batch-transfer into our archival storage, right?
Except no matter what kind of obvious shell-idiom method I try to use, it doesn't work because PowerShell stubbornly insists that I must be working with text data when I'm using all those handy shell constructs, and so the files end up corrupt.
So now instead of just being able to shove data through pipes like I would on Unix (which, for all its faults, doesn't really care whether I'm sending binary or text data), I have to have my script go and instantiate objects for creating a file and reading and writing binary streams, and remember to clean up after them, and my script ends up way more complicated.
What would have been a pretty simple script anywhere else now has to know a bunch of .NET libraries and how to create and work with their types, and that's less useful to me.
Invoke-Sqlcmd -Query "SELECT ..." and write it out to a file?
ip addr show | grep inet
[kapil@localhost ~]$ ip addr show | grep inet
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
inet 192.168.128.236/24 brd 192.168.128.255 scope global dynamic eth0
inet6 fe80::215:5dff:fe80:501/64 scope link
[kapil@localhost ~]$ ip addr | grep "inet " | cut -f 3 -d " " [kapil@localhost ~]$ ip addr | grep "inet " | tr -s " " | cut -f 3 -d " "
127.0.0.1/8
192.168.128.236/24
[kapil@localhost ~]$
[kapil@localhost ~]$ ip addr | awk '/inet / { print $2 }'
127.0.0.1/8
192.168.128.236/24
[kapil@localhost ~]$
I think the idea of an object pipeline is sound for precisely this reason, but I don't think it should be connected to any one language or platform.
I can only use this feature by using a .NET language. I can't take the strongly typed objects into a Node.js script, or a perl script.
It goes without saying that a binary interface requires use of a system that understands that particular binary interface. With .NET being open source though, anyone is free to write a bridge to using the objects in COBOL if they wanted to.
>I can't take the strongly typed objects into a Node.js script, or a perl script.
Yes, because they have no concept of strongly typed objects. You can convert objects to strings if you're using an incompatible tool/language. I don't know what your point is. Sorry..
Virtually any language can already input and output text. That's the real lingua franca of computing (even if it is messy and dirty), not .Net objects...
The difference between impossible and possible is important. Please read what I was replying to. Using your argument, Anytime a project is open sourced, you could reply with "who cares about potential since nobody has actually done anything with the source". Good job.
>Virtually any language can already input and output text.
Um, you seemed to have missed the obvious point that objects can also be converted to text through powershell if the other side can only accept text. So, I don't quite know what you're complaining about. Could you detail your actual technical complaints with powershell, rather than having a philosophical argument?
>That's the real lingua franca of computing (even if it is messy and dirty), not .Net objects...
I don't agree with your opinion, nor with the idea that only a two-choice system can exist.
But it's not bash, it's coreutils
find ./ -mtime +2 | xargs rm Get-ChildItem -File | Where-Object LastWriteTime -lt (Get-Date).AddDays(-2) | Remove-Item
ls -file|? LastWriteTime -lt ((date)+'-2')|rm
Sorry, I can't agree with you.
> I may not know what the +2 actually means with the -mtime argument. How would I change it to look for the age of two hours instead of days?
If you forget or doesn't know what is mtime, you can open man find. For less granularity than days use mmin.
/Mapped-Temp-folders/Server-Asia/temp
/Server-UK/temp
/Server-Japan/temp
ls|? LastWriteTime -lt (date).AddDays(-2)|rmparsing is actually a huge pain in general. a lot of unixy tools produce different output when their stdout is not a terminal to be more pipeline-friendly. That output is generally easier to parse (often tab-separated or so), but you still need to know the same things about the output (the "type" could be whether it came from `ps` or from `ls`, and the fields are just in some static order that was usually chosen decades ago, and if it's extensible it's cumbersome).
If `ps`, `ls`, etc produced typed output, I would be sooo happy.
When I worked at google I thought about what it would be like to have a unix userland where all the parts communicate with protobufs instead of raw text. There are some problems (like portability of message definitions, which was a solved problem within google, but which is harder when you have multiple parties/entities/companies contributing message definitions).
edit: accidentally a word
What sounded like a very simple thing, quickly ended up being a complete nightmare of bash, python, and an unholy bunch of linux commands.
It was ugly, and a terrible way of doing it, but we got it working.
It would have been amazing if i could pipe PS into something that could sort by the dates on the 4th column, and return the last column in the sorted order.
Failing that, consider the `etimes` field (seconds since process was started) used in conjunction with the -o feature (control output format) so you can put `etimes` up front then simple pipe the output to `sort`.
Those things being said, the situation you describe is indeed hairy. `ps` output was designed for human consumption, that's why the process age field format varies (1w vs 10:15) and that's presumably why you wanted to bring python into the mix!
...You might have had an easier time looking in /proc...
Indeed, this looks promising:
cat /proc/[0-9]*/stat|sort -n|less -S
EDIT #2: But try doing ANY of this on an ancient Windows machine! The "peer" of your old Redhat machine would be what, a Win2k3 machine, or a Win2k? How old is PowerShell? :)
I was more just commenting on the fact that with a "strongly-typed" shell some of this stuff becomes much easier without every single tool needing to add all these options.
It gets much more "Unix Philosophy" when every command doesn't need it's own sorting logic, and doesn't need it's own date display logic, and doesn't need it's own way of displaying it to the user.
It can output an object, which can be piped into another program to use what it needs which can be piped to something else to do some sorting which can pipe it out to another to display it in a nice table.
But let's not throw out the baby with the bathwater... The unix shell ecosystem is powerful magic. There are LOTS of things that can be done there with great ease.
Not to mention--and I understand full well that this stance is falling out of fashion--MICROSOFT ... KILLED... my PAPPY! :|
ps kstart_time -efIt was a mess, and I remember thinking it would be such a simple thing that was so much more complicated than i thought it would be.
For example, the other day I was trying to search the columns of a very large SQL Server table¹ but also needed information about the column type.
…\dbo.site> ls Columns | select Name, DataType, {$_.Properties['Length'].Value} | where {$_.Name -like '*int*'}
Name DataType $_.Properties['Length'].Value
---- -------- -----------------------------
internal_end datetime 8
internal_start datetime 8
interval_code varchar 5
prod_intv_start int 4
prod_intv_stop int 4
working_int decimal 5
¹ SQL Server installs a Powershell extension so that you can essentially treat databases like a fancy object filesystem. It's wicked cool actually.
mysql > SELECT column_name, data_type, numeric_precision FROM information_schema.columns WHERE table_schema = ? AND table_name = ? AND column_name LIKE "%int%"Point is, imagine trying to grep anything tabular in bash. Hypothetically, the output of ls -la for files with execute permission that have been modified in the last day. Or, more realistically, anything involving ps.
When boxed in-between the fearsome foursome--`bash`, `sed`, `awk`, and `emacs`--every problem involving tabular data (2D) is trivial. Hm, I'll give a shout-out to `sort` and `uniq`, too! When you go 3D, you might need to bring in `join`.
But anyway, you'd use
find ~/bin /usr/bin -mtime 1 -executable -type f -ls
:)
PowerShell's integrated query syntax that you can patch into various things is a lot more consistent and reasonable. It is difficult to imagine volunteering people into a world of awk, sed and cut with good intentions in one's heart.
Forgive my being frank, please, but I think that given the context of this conversation, it's appropriate: Most users don't HAVE non-trivial problems!
There aren't features in sed nor awk, nor regular expressions in general that weren't put there for some specific reason in the past.
When you wield your shell, well, you wield a tool honed through the ages by people who think like you! Do so with pride! :)
Some people just want to get shit done because this is just a footnote in a task that's a small part of a project. Utilizing gained proficiency in several tools is cool but if you have a choice between doing that and just taking advantage of a simple tool that streamlines things and lets you move on to your next task quickly the correct choice is obvious.
It could be that these tools are better and may be more valuable in some situations but asking someone to spend time gaining proficiency or mastery when another path is available is just premature optimization.
But of course, it's not true. Many great and lasting tools have been founded out of corporate work in a bid to improve mindshare and ease recruiting by increasing the esteem of corporate engineering efforts. Many of the things we think of as "open source" and "free" were built subsidized either directly or indirectly by capitalistic interest.
But I'm not concerned about keeping it. Ossifying my process so that I can give myself the luxury of time of from learning is not something I'm prepared to do, because I think it will make me soft and less capable as an engineer. I regularly rotate my editors and learn new languages because I don't want to become like the sad specters of the previous generation of software engineers I see now, starched shirts and ties and a fanatical devotion to the dated technologies they stomach in order to collect huge consulting fees from corporations.
I've switched off Fish to Powershell as my primary shell for awhile. It's interesting.
*gulp* I've heard people say that, but I haven't delved into them!! :-S $mysql_cmd|awk -F\| '$2 == "what you want the second row to match" {print $0}'
Extendable tooling!
Fortunately, that's not a world any of us live in.
Then you end up using awk, which is rather like a half-assed version of powershell to work around shortcomings with the unix everything-is-a-string design.
You'd probably need to make an explicit SQL query since you wouldn't have the database integration.
If, hypothetically, bash had database integration, ls Columns would probably return something like
foo varchar 30 NOT NULL
bar decimal 8
baz int 4
I could've used an example involving process lists, but I didn't want to trigger PTSD in anyone who's had to grep the output of ‘ps’ for anything non-trivial. (Plus, the database thing was something I actually did a few days ago.)
/proc is a race condition waiting to happen, so I can't really say I recommend using that.
get-netadapter | get-member
Alternately, you can get all of the current properties of any existing object by piping to 'select' (to get a general subset) or 'select * ' to get all properties. Note that this gives all information for the objects in the pipeline.
get-netadapter | select
get-netadapter | select *
'Select' can also be used to easily filter the properties of items in the pipeline to ones you care about.
get-netadapter | select Name, MacAddress
And ultimately, if you want to use powershell to do the stuff that typed objects are good at, and then leverage command line tools to do the rest, you can if you like.
get-netadapter | select Name, MacAddress | exportto-csv
There's a bunch of subtle distinctions like that which don't sound impressive but end up being really nice when working with it.
I don't know, I never had any problems with text parsing. I love strong types in programming languages, but in system administration I think it would only get in the way. But as I said, never tried it, so I might be missing something.
> Can I take the value, convert it to string, make some magic on it and convert it to another object entirely?
Get-ChildItem *.js | Get-Content | Select-Regex 'require\("(?<Require>.*?)"\)' | Select-Object -ExpandProperty Require
Ultimately, Microsoft provides WMI cmdlets[2]. You'd have strong types from the get-go and wouldn't need resort to this string silliness.
[1]: https://gist.github.com/jcdickinson/cee4582448300c0d404bbff1... [2]: https://technet.microsoft.com/en-us/library/ee176860.aspx
How so?
gci *.js | foreach { gc $_ | foreach { if ($_ -match ..) { $Matches...
sls 'require\("(.*?)"\)' *.js | select { $_.Matches.groups[1] }
I don't see why this would be a problem for PowerShell. Just use Select-String to strip away the characters you're not interested in. The advantages that derive from passing around objects aren't based on all data being automatically in the format you want.
https://technet.microsoft.com/en-us/library/ff730948.aspx
You use the exact same functions (first, last, substring, replace, etc) as you would normally. Then the rest of the pipeline has it available if needed.
There are also details of Powershell that are very annoying, such as the lack of < redirect, and the difficulties it has on binary output sometimes.
(ls)[0] | gm
will list every field and its type on a File object. Wicked easy.
ls|gm
REASON=$(echo "$line"|tr ' ' '\n'|grep reason= |cut -d= -f2)That being said, I can see the appeal and I know people who love it. I have seen some amazing shit written in it, like this security tool that was just open sourced this year:
https://github.com/adaptivethreat/bloodhound
I have used Mono/GTK# for years on Linux and prefer it over other GTK wrappers. Powershell might be worth a try on Linux, but it will take time to build up the libraries and wrapper needed to interface with most of the Linux subsystems (unless Microsoft as ported a lot of useful stuff in this initial release).
People don't (or shouldn't) use bash/zsh/fish for really complicated stuff anyway. I have a feeling most sys admins/devops people will continue to use Python/Ruby for complicated tasks; and they both have a ton of libraries that help avoid the stdin/out/parsing situation you have with chaining together regular shell scripts.