echo@> (2{ARGV) -.&([: <;._1 LF, 1!:1) (3{ARGV)
NB. 2nd arg 3rd arg
NB. g&f execute f for each, then g on both
NB. 1!:1 read file
NB. LF, prepend newline
NB. [: <;._1 split based on first char
NB. -. remove right elements from left array
NB. echo@> echo each line
exit 0
On two ~1.6MB files with ~15k lines (both the same except 3) I had lying around: $ time j9 -c ./pseudo_grep.ijs test_b test_a
…
real 0m0.064s
user 0m0.032s
sys 0m0.017s
$ time grep -vf test_b test_a
…
real 0m5.815s
user 0m5.234s
sys 0m0.576s
Note that most of the script is for loading each file into an array of lines. Most work is done by -. on the two arrays, which is exactly what you asked for, e.g. 0 1 2 3 4 -. 2 4 is 0 1 3. https://code.jsoftware.com/wiki/Vocabulary/minusdot#dyadic set(open('file_b')) - set(open('file_a'))
Slower than J by a factor of 2-3, but still 10x faster than grep: real 0m0.128s
user 0m0.078s
sys 0m0.063s
This would make a good Rosetta Code prompt.I did not know that. Assumed you had to somehow wrap them first. Very useful, thanks!
In my simple test, that resulting in grep running in 44% of the prior amount of time it required (still more than python though).
I tested with files just over 2 MB on a small Digital Ocean VM. Depending on disk speed, based on running time I suspect you ran on files at least an order of magnitude larger. What time did python run in for those? Seeing memory usage from time might be illuminating for these tasks too. Using 4x the disk size in memory is fine for a couple MB file, but less so for a couple GB file (in which case creating a bloom filter or trie might be better, but I really have no idea if Pythons set functions do that already).
A←⊃⎕nget 'C:\test\a.txt' 1
B←⊃⎕nget 'C:\test\b.txt' 1
A~B
┌─────────────────────┬──────────────────┬────────────┐
│AAAsofxxxLqKbYOAwLMHG│AAAdjLalKFapoYAcGE│AAAthGazmnfK│
└─────────────────────┴──────────────────┴────────────┘
Read each file as a nested vector of lines. A without B, like set operations.If the components of your company's system never are longer than a page or two of code, rewrites become trivial, making planning for the future unnecessary. This is the real promise and value of array languages.
Or put more simply, the size of a code base is the dominant factor in its complexity. Less code is less complex code, is better code.
Also, loops imply mutation, which is the second biggest indicator of complexity. Mutation kills code bases and kills projects. When you decide to never use a loop and to never mutate, you're on the right path. Now the next step is to try and remove as many lines of code as humanly possible.
A good way to measure how you're doing is by comparing the size of your gzipped (or zipped or whatever) codebase to its uncompressed form. If they are close in size, you know you are on the right track. Or in other words, you want the size of your code base to be close to its Kolmogorov complexity.
What about someone who thinks "if" is a big source of software complexity? https://www.youtube.com/watch?v=z43bmaMwagI - talk is "If considered harmful: How to eradicate 95% of all your bugs in one simple step - Jules May".
[1] in interpreted languages, anything you can push down for the runtime to do instead of writing that same code in the interpreted language, tends to execute faster. e.g. my_string.count('c') in Python compared to a `for` loop over the characters written in Python.
For that sort of comparison I usually use
diff -y A B comm -13 A B echo AAA_not_in_A_xxx >> B
fgrep -vf A B
Result should be AAA_not_in_B_xxx A_B
The name of this site is "nsl" for No Stinking Loops.
Here is an array programming challenge for the K programmers reading this thread.
There are two large files A and B.
The task is:What are the lines in B that are not in A?
Something like
But without the stinking loops!EDIT: File A,B formatting