Also for those that haven't come across it https://github.com/dlwyatt/PowershellProfiler is pretty useful.. despite add-content being the obvious offender here..
On that note, what are the obvious ways you've tried?
So many times i run into performance walls with powershell, that calling the .net library directly is basically required.
While ForEach-Object is like an order of magnitude slower, we're talking like 100 ms vs 10 ms to iterate through 10,000 items. (https://blogs.technet.microsoft.com/heyscriptingguy/2014/07/...) In other words, you have to have a good number of objects to make foreach meaningfully faster that when you do, you often find that either a) lose time to powershell.exe allocating more system memory to store the collection, or b) the loop itself is two or three orders of magnitude slower already so it's ultimately a trivial optimization.
Also, the foreach statement waits to allocate the entire set to memory ($y in your example) while ForEach-Object begins processing as soon as the first object comes through the pipeline. If you're using the output of a command for $y that returns output with some lag, you might find that ForEach-Object actually turns out to be faster because the first objects start being processed immediately. If you're returning data over a network or dealing with particularly large objects, ForEach-Object can be better.
Finally, the foreach statement itself doesn't work with pipelines either as input or output, so it's not appropriate for a lot of scenarios.
The pipeline is powerful, but it's slow. There's a meaningful performance gain I've seen with foreach(), and even more so when loops need to be nested. I believe the speed increase is because the runtime/clr/jit can infer the types contained within the collection before it actually starts executing the statement. yes, using generic collections with foreach helps as well, because (i think) powershell can completely avoid redetermining what every type of object is at execution time (as the collection is generic underneath).
The downside is readability and memory usage as you pointed out, but i could argue that piping to foreach-object can also be painful to read, and many times the loop could be refactored into a proper function and given a meaningful name....
Compare:
Measure-Command {
# ~4,000 files
$Files = Get-ChildItem -Path "C:\Windows\System32\DriverStore" -File -Recurse
foreach ($file in $Files) {
$file.LastWriteTime.Date;
}
} | Select-Object -Property TotalMilliseconds
Measure-Command {
Get-ChildItem -Path "C:\Windows\System32\DriverStore" -File -Recurse | ForEach-Object {
$_.LastWriteTime.Date;
}
} | Select-Object -Property TotalMilliseconds
However, let's take something decidedly non-trivial:
Measure-Command {
# ~4,000 files
$Files = Get-ChildItem -Path "C:\Windows\System32\DriverStore" -File -Recurse
$Algorithm = @('MD5','SHA1','SHA256')
foreach ($file in $Files) {
Get-FileHash -Path $file.FullName -Algorithm ($Algorithm[(Get-Random -Minimum 0 -Maximum 3)]);
}
} | Select-Object -Property TotalMilliseconds
Measure-Command {
$Algorithm = @('MD5','SHA1','SHA256')
Get-ChildItem -Path "C:\Windows\System32\DriverStore" -File -Recurse | ForEach-Object {
Get-FileHash -Path $_.FullName -Algorithm ($Algorithm[(Get-Random -Minimum 0 -Maximum 3)]);
}
} | Select-Object -Property TotalMilliseconds
Most of my workloads involve calling commands that involve significant time like Get-FileHash does. I'm doing things like splitting PDFs based on content and inserting them into an SQL database, or fetching 8,000 records from Active Directory and verifying file share permissions. I have found that for nested loops that I tend to use a ForEach-Object in the pipeline and then use foreach statements for operations on arrayed properties in the object and that works well, but for the outer loop, no, it's not usually worth the time to refactor or eliminate the pipeline.
i mean, if the code is 90% I/O bound i agree: micro-optimisations can be a waste of time. But tight, nested loops that are only memory bound i think is worth the effort.. so for me, if i need need to do a lot of field comparisons for 300k accounts or groups from AD, i will download the ad data into memory and use foreach(), hashtables and generic collections wherever possible.. i actually think were making the same point anyway. I now reach for C# before powershell for situations like above where performance matters, but powershell does have advantages for smashing out some quick work. But i honestly cringe when i hear people talk about powershell scripts that take minutes to execute for tasks that should be reasonably quick.
And, yes, generally the easiest path to faster performance is just loading everything into memory and working with it there. In my PDF script I need to validate that the ID numbers are being read are valid, so I pull all 20,000 of them and put them into a HashSet<String>. That's extremely fast to validate with compared to an array and performs a bit better than a HashTable. There are times when I need to use an SqlDataReader or StreamReader and read row by row or line by line (4 GB+ text files suck), but that's only when memory has become a problem.
I've been meaning to really dig into the High Performance PowerShell with LINQ article (https://www.red-gate.com/simple-talk/dotnet/net-framework/hi...), but I just don't have a need for it at present unless I refactor an existing script and they all work great.
In his wine glass script he also uses the PowerShell patten of creating an array ($x = @()) and then appending to it (foreach ($i in $set) { $x += $value * $i }). The problem here is that PowerShell arrays are fixed in size. To append a value to an array, the system creates a new array, copies all the values over with the new one, and then disposes of the old array. It works fine up to about 100 items, but it gets noticeably slow after that. Since it's done with a loop it makes it somewhere between O(n log n) and O(n^2). It's better to just output all the values as a single array ($x = $set | ForEach { $value * $_ }) or to create an ArrayList or List<String> or some C# collection that supports an O(1) append.
He also assigns a lot of variables in his loops and then uses them only once to format a string. He could eliminate those variables and just embed the expressions in the strings.
I got a ~25% performance improvement (210 ms to under 140 ms) with just the script embedded in the article when I switched to StringBuilder instead of Add-Content, and that script doesn't have the poor array pattern. StreamWriter would work, too, with less memory pressure than a StringBuilder. I suspect that with better code you could easily get this down below 3 seconds.