Re:Inside the "3 billion people" national public data breach
by Vicinity9635 on Thursday August 15, 2024 @01:43PM
(#41258553)
I dug into this a little and one of the files is 164GB. How do you even work with these files? That is, how would I search for my SSN on my windows box?
[ Parent
| on HN ]
Re:Inside the "3 billion people" national public data breach
by diggan on Thursday August 15, 2024 @02:29PM
(#41258995)
That's not even that big? `cat big_file | grep -v my_term` would go line-by-line and show any lines matching your query. If you're doing a lot of queries, you'd probably want to index it, so you throw it into a sqlite database with the usual SQL utils.
Edit: I missed you said Windows. Probably Powershell have similar utilities, so you can do `ReadFileLineByLine \r \d big_file | ReturnHitBySearchTerm \v \t \s my_term` or something similar.
Re:Inside the "3 billion people" national public data breach
by kaiwenwang on Thursday August 15, 2024 @03:21PM
(#41259441)
Using sift on a 100GB txt file still takes multiple minutes. I haven't tried ag, but grep is supposedly slower.
Re:Inside the "3 billion people" national public data breach
by kelnos on Thursday August 15, 2024 @04:03PM
(#41259814)
If the desire is just to grep for your name, email address, whatever, and then throw the rest of the data away, I don't think waiting multiple minutes is a big deal.
Re:Inside the "3 billion people" national public data breach
by 6031769 on Thursday August 15, 2024 @04:54PM
(#41260273)
You absolutely do not want to use "-v" with that grep.
Nor do you want to use cat (UUoCA) but that's very much a minor point in comparison.
Re:Inside the "3 billion people" national public data breach
by SahAssar on Friday August 16, 2024 @09:36AM
(#41266303)
UUoCA: https://porkmail.org/era/unix/award
I hadn't heard of it before.
Re:Inside the "3 billion people" national public data breach
by Vicinity9635 on Thursday August 15, 2024 @05:52PM
(#41260781)
>ReadFileLineByLine \r \d ssn.txt | ReturnHitBySearchTerm \v \t \s trampas
ReadFileLineByLine : The term 'ReadFileLineByLine' is not recognized as the name of a cmdlet, function, script file,
or operable program. Check the spelling of the name, or if a path was included, verify that the path is correct and
try again.
At line:1 char:1
+ ReadFileLineByLine \r \d ssn.txt | ReturnHitBySearchTerm \v \t \s tra ...
+ ~~~~~~~~~~~~~~~~~~
+ CategoryInfo : ObjectNotFound: (ReadFileLineByLine:String) [], CommandNotFoundException
+ FullyQualifiedErrorId : CommandNotFoundException
:(
All I know about powershell I just learned by accident: ls works