News for hackers. Stuff that matters.
Re:533M Facebook users' phone numbers and personal data have... by b212 on Saturday April 03, 2021 @06:25PM (#26685294)
Could you please tell me how did you convert it to sqlite? I've got a huge 1 GB txt file that crashes my comp every time I try to search for myself there :( Thank you!
Re:533M Facebook users' phone numbers and personal data have... by banana_giraffe on Saturday April 03, 2021 @06:37PM (#26685375)
Super hacky python script I used to turn the text files into a sqlite database:

https://pastebin.com/gBWhCVGz

Re:533M Facebook users' phone numbers and personal data have... by kuyan on Sunday April 04, 2021 @12:47AM (#26686960)
Thanks, this worked for me! Here's my contribution; I tweaked your script a little for memory usage. https://pastebin.com/4hA8VACe
Re:533M Facebook users' phone numbers and personal data have... by shazzdeeds on Sunday April 04, 2021 @04:04PM (#26692412)
Thanks. I tweaked the script to include an incremental printout across files and also creating an index on phone num to easily search.

https://pastebin.com/AxY5PeDQ

Re:533M Facebook users' phone numbers and personal data have... by bobbylarrybobby on Saturday April 03, 2021 @07:02PM (#26685516)
I suggest using a shell program like grep (or even the search feature of less), as shell programs are notoriously good at lazily seeking through files to keep memory use low.
Re:533M Facebook users' phone numbers and personal data have... by knolan on Saturday April 03, 2021 @06:48PM (#26685427)
Firstly don’t do something like open it in notepad. 1GB text files are not exactly difficult to work with once you use a proper text editor or parsing tools.
Re:533M Facebook users' phone numbers and personal data have... by throwmusic2366 on Saturday April 03, 2021 @07:19PM (#26685607)
Does it open with Geany?

https://www.geany.org/

It's FLOSS. Available on Windows, Mac and Linux.

Re:533M Facebook users' phone numbers and personal data have... by cable2600 on Sunday April 04, 2021 @01:56AM (#26687229)
I used to use Borland's Brief to edit large files. But it is commercial. There is a free version in beta here https://sourceforge.net/projects/grief/ Grief. It pages the file in and out of memory to save space.
Re:533M Facebook users' phone numbers and personal data have... by datavirtue on Saturday April 03, 2021 @06:45PM (#26685411)
Try Ultra Edit, free trial. It can read and search massive text files without crashing. Quite responsive.on 10GB files.
Re:533M Facebook users' phone numbers and personal data have... by max_hammer on Sunday April 04, 2021 @02:23AM (#26687336)
You can use `awk` for any sql like operations on file.
Re:533M Facebook users' phone numbers and personal data have... by swiley on Saturday April 03, 2021 @11:54PM (#26686766)
sqlight can ".read" a number of formats. It's one of the more convenient ways to do joins on ugly tables.