The OP talks from experience gathered over many years of "getting things done". I would listen very closely to him.
local resource_name_to_address = {}
local n_resources = file:read_int()
for i=1,n_resources do
local name = file:read_str()
resource_name_to_address[name] = file:read_int()
end
is easy to write indicate that I have not spent any time programming?This code allocates a lot of memory, needlessly. Why not just do a single `fread' to read in the TOC as one big lump, followed by a loop calling `strncmp' on each entry each time the offset of a particular resource is required? (You don't even need to malloc in this case; just decide that the TOC will never be more than a certain size, and put the buffer in the BSS.)
I personally would prefer the `fread' approach. It doesn't allocate lots of small pieces of memory, so the list of memory allocations is easier to read. You waste less space on malloc bookkeeping overhead, and/or you don't need to go to the effort of writing a specialised allocator for small allocations. If your hash table is closed, you won't end up with a big hole from where it had to rehash.
I've spent a fair amount of time getting game projects ready for release and it always, no matter how cunning everybody is (or tries to be), ends up with you - that'd be me, by the way - sitting there staring at a big long list of allocations, because something weird is happening. There are big holes appearing in the middle of memory, exactly where you don't want them, or there's just too much being allocated, or things are leaking, and the debugging infrastructure (if any) isn't being helpful for whatever reason. Probably those zillions of tiny little allocations exhausting the pool available to store the debug info.
This sort of thing is an easy way to eat up the man-hours saved by writing code like the above. Though I suppose that if you (and this would not mean me this time) can get somebody else to do the cleanup work, it probably doesn't matter.
Of course, once you've done all the hard work of following my advice, you could just pre-generate the hash table in the file already, maybe using a perfect hash system, and use that instead. (One of my colleagues used to like this.) That's fine, but... what's the point? You're about to LOAD A FILE. You have to assume it will take at least 0.1 seconds before the read can even begin to start. The difference between getting to the entry quickly, and getting to the entry really quickly, will just get lost in the noise.
The points implied by this post are all there in his talk, I think ;)
That is, writing 90,000 lines of the product in C++ is likely a case of premature optimization, or precisely what I was just told I shouldn't do. If not, it is at least a decision to spend hours upon hours of my life waiting for things to compile and days more messing with bugs that only show up if I don't compile with debug flags, which doesn't seem very life-efficient either.
This could also happen if the programmer simply never spent the five minutes necessary to get a library with a hash table implementation in it, opting instead to spend a third as much time on hundreds of occasions. I suspect that this is closer to what's going on...