News for hackers. Stuff that matters.

A comment on Inside the "3 billion people" national public data breach. Parent comment

Re:Inside the "3 billion people" national public data breach by diggan on Thursday August 15, 2024 @02:29PM (#41258995)
That's not even that big? `cat big_file | grep -v my_term` would go line-by-line and show any lines matching your query. If you're doing a lot of queries, you'd probably want to index it, so you throw it into a sqlite database with the usual SQL utils.

Edit: I missed you said Windows. Probably Powershell have similar utilities, so you can do `ReadFileLineByLine \r \d big_file | ReturnHitBySearchTerm \v \t \s my_term` or something similar.

[ Parent | on HN ]
Re:Inside the "3 billion people" national public data breach by kaiwenwang on Thursday August 15, 2024 @03:21PM (#41259441)
Using sift on a 100GB txt file still takes multiple minutes. I haven't tried ag, but grep is supposedly slower.
[ Parent | on HN ]
Re:Inside the "3 billion people" national public data breach by kelnos on Thursday August 15, 2024 @04:03PM (#41259814)
If the desire is just to grep for your name, email address, whatever, and then throw the rest of the data away, I don't think waiting multiple minutes is a big deal.
[ Parent | on HN ]
Re:Inside the "3 billion people" national public data breach by 6031769 on Thursday August 15, 2024 @04:54PM (#41260273)
You absolutely do not want to use "-v" with that grep.

Nor do you want to use cat (UUoCA) but that's very much a minor point in comparison.

[ Parent | on HN ]
Re:Inside the "3 billion people" national public data breach by SahAssar on Friday August 16, 2024 @09:36AM (#41266303)
UUoCA: https://porkmail.org/era/unix/award

I hadn't heard of it before.

[ Parent | on HN ]
Re:Inside the "3 billion people" national public data breach by Vicinity9635 on Thursday August 15, 2024 @05:52PM (#41260781)
>ReadFileLineByLine \r \d ssn.txt | ReturnHitBySearchTerm \v \t \s trampas ReadFileLineByLine : The term 'ReadFileLineByLine' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verify that the path is correct and try again. At line:1 char:1 + ReadFileLineByLine \r \d ssn.txt | ReturnHitBySearchTerm \v \t \s tra ... + ~~~~~~~~~~~~~~~~~~ + CategoryInfo : ObjectNotFound: (ReadFileLineByLine:String) [], CommandNotFoundException + FullyQualifiedErrorId : CommandNotFoundException

:(

All I know about powershell I just learned by accident: ls works

[ Parent | on HN ]