Its been running for over 6 months :), ended up being a side project. We are running tokyo tyrant (disk based), so that has slowed down over time. Will tune that up!

As for the breaks and paragraphs, planning to add more logic when lines are picked up from different para. Thanks for the feedback!