I do a lot of that sort of thing, and my go-to tools are grep (with full regular expressions), sort, uniq, head, tail, sed, and, the 200 pound gorilla (not as heavy as it could be) awk. Then if all else fails, Python