News for hackers. Stuff that matters.
Re:Text Editor: Data Structures by mpweiher on Monday October 02, 2017 @08:41AM (#15383335)
Emphatically: Yes! KISS.

Moby Dick, at its slender 752 pages, is 1.2MB of text. You can save the entire text to disk on every keystroke on just about any system today and keep up with typing just fine.

Assuming you are actually dealing with text in a text editor, you should be fine.

If you have 100MB+ files, chances are they aren't actually text.

Re:Text Editor: Data Structures by caconym_ on Monday October 02, 2017 @03:01PM (#15386825)
Many times in my career I've had to open very large files (usually CSV data) and inspect/edit them by hand.

Of course there are other ways to do this work, but a text editor that can efficiently open and edit very large files (e.g. Vim) is a great tool for the job. I will put up with otherwise excellent code-specific editors/IDEs that cannot work on such files but my default general-purpose text editor is always going to be something that can.

I appreciate that the design and implementation of text editors is basically an art form (one I've dabbled in myself) in which simplicity has aesthetic value, but efficiency and flexibility are very important for an editor that's going to be used for real work.

Re:Text Editor: Data Structures by terminalcommand on Monday October 02, 2017 @10:50AM (#15384191)
Textual dumps of SQL databases also fall in this category. I remember having downloaded a huge sql dump. I couldn't open it in most editors because they all buffered into RAM. The file was about 13G in size and did not fit into the RAM of any machine I own. But in this case using ropes wouldn't have helped either, I guess.

Text editors should also take huge files into account and provide sequential reading from the disk to memory. Even on Emacs, I couldn't work with the file. I ended up fixing it and importing it to Postgresql. And then I spent hours on indexing the necessary fields :).

Re:Text Editor: Data Structures by MetaDark on Monday October 02, 2017 @01:00PM (#15385591)
I believe you can use the vlf package (https://github.com/m00natic/vlfi) for dealing with large files in Emacs. I haven't used it myself, so I am not sure how stable it is.
Re:Text Editor: Data Structures by terminalcommand on Monday October 02, 2017 @02:30PM (#15386470)
Thanks for the pointer. I guess I could also have written a small program or script to go through the file. Regular fopen and fread should work. However the vlf package does exactly what I looked for. Only prerequisite I can see is that one must compile Emacs with bignum support on 32bit systems to read files greater than 512M.
Re:Text Editor: Data Structures by weavie on Monday October 02, 2017 @10:38AM (#15384111)
Log files..
Re:Text Editor: Data Structures by mpweiher on Monday October 02, 2017 @10:51AM (#15384195)
Fair point, but do you actually edit log files in a text editor?

Because if you're just viewing, the "array of characters" representation is going to beat just about any other hands down, especially if you just mmap() the whole thing.

Re:Text Editor: Data Structures by weavie on Monday October 02, 2017 @12:08PM (#15384979)
True. Generally no, but sometimes I will insert several empty lines around a particular line of interest..
Re:Text Editor: Data Structures by gkya on Thursday October 05, 2017 @05:37AM (#15407806)
I've used vi where I meant the pager quite often. I was late at making the use of head, tail, grep, wc, du, stat, etc. to peak into data to become muscle memory.
Re:Text Editor: Data Structures by scott_s on Monday October 02, 2017 @10:48AM (#15384173)
Yes. Occasionally I'll open a 100MB+, or even a GB, log file without realizing it, and it will take vim many seconds to give me control.
Re:Text Editor: Data Structures by maxxxxx on Monday October 02, 2017 @10:49AM (#15384182)
I am dealing with gigabyte sized log files almost every day.
Re:Text Editor: Data Structures by scaryclam on Monday October 02, 2017 @02:27PM (#15386441)
Why not send them into a log aggregator for search and long term storage and logrotate on the server so you don't get this problem? Hell, why not logrotate anyway and keep multiple smaller files?
Re:Text Editor: Data Structures by maxxxxx on Monday October 02, 2017 @03:12PM (#15386942)
Sur4e. My response was to the comment that 100MB+ files are most likely not text. Like others I just tried to express that there are plenty of use cases for a text editor that can handle very large files. It's just a very practical thing.
Re:Text Editor: Data Structures by macintux on Monday October 02, 2017 @04:27PM (#15387624)
I was consulting on a project the other day; turned out they never rotated their main Apache log file, and after 5 years it's now 32GB.