Seq is great because you can set up your own instance very near to your servers for low-latency/high-bandwidth logging, which really changes the game in terms of what you can feasibly (perf/financially) log. It also has some decent visualization options, and it's got some great integrations, with a decent plugin architecture to create your own real-time log processing code.
Logg.ly has some amazing GUI/search options.
Log.Info("Customer# {customerNumber} completed transaction {transactionId}", customerNumber, transactionId);
Then using the Seq log viewer you can simply click on "transactionId" in the log line and filter by "transactionId = 456" or whatever. It's one of the most exciting advancements I've seen in the .Net logging world.
EDIT: I realized I didn't really answer OPs question regarding space. If you used Serilog, you can set up different sinks to export to, with different options. For example, you could send all your logs to mongodb, and just recent 1 week rolling logs to the Seq server.
Log.Info("customerNum={customerNumber} transactionId={transactionId} state=completed", cN, tID)
or the ever popular logstash-y format: Log.Info(LogState.Add("state", "completed").Add("customerId", customerId).Add("transactionId", transactionId));
where `LogState` would build up a key-value dict and its `ToString` would emit the logstash JSON format.I guess the version that works best depends on the tool that is consuming the log text.
Right now at work everyone just logs to a single CSV with an inconsistent format and it makes me cringe every time I look at it. It's also really difficult to parse.
Your current way does sound like a headache. If your logging lines are in the standard NLog format, you should be able to drop in serilog without many changes.
Using a system like Ganglia (or the Etsy inspired statsd) is an important idea since the OP's original question included how to limit the size of logged data. These systems provide a natural way to aggregate data.
The most commont logging formats I've come across in production environments are:
1.log4j(java) or nlog(.NET)
2.json
3.syslog
Tools that I've used to search ,visualize and analyse log data have been:
1.Elasticsearch, Logstash and Kibana (ELK) stack
2.splunk (commercial)
3.Logscape (commercial)
Changes to the fields representing your data with the database approach is expensive because you are locked in by the schema. The database schema will never fully represent your full understanding of the data. With the tools I've mentioned above you have the option to extract ad-hoc fields at runtime.
Hope this helps.