News for hackers. Stuff that matters.
Re:Why Mongrel2 Doesn't Use INI Files (Zed Shaw) by tptacek on Sunday October 10, 2010 @10:01AM (#1777093)
Bad call. The canonical version of the server configuration doesn't play nice with version control, and so staff has to work with a facsimile and hope that nobody makes out-of-process changes.

(INI files though? What year is it?)

Re:Why Mongrel2 Doesn't Use INI Files (Zed Shaw) by gaius on Sunday October 10, 2010 @11:13AM (#1777167)
Because programmers are expensive, but sysadmins are free.

If you're a programmer working for a large organization, despite sitting in an open plan office surrounded by dozens of other programmers, and knowing that there are hundreds in the building, you probably nevertheless believe that yours is the only code running in production, and the entire operations staff exists in order to support it...

Re:Why Mongrel2 Doesn't Use INI Files (Zed Shaw) by zedshaw on Sunday October 10, 2010 @12:17PM (#1777286)
I think me and you have some different definition of canonical:

http://www.wordnik.com/words/canonical

All I see are things about religion, dressing like the status quo, and other things that say "old way of doing things". I think you mean to say "official" or something like that. Let's use official.

In this case, the config file you use is the "official" configuration. If you use the m2sh format than that's the default official format. If you put those into version control then you have solved your problem.

But then, I'm sure you have some witty reply to this reply to your canonical one-liner:

"Conforming to orthodox or well-established rules or patterns, as of procedure."

Re:Why Mongrel2 Doesn't Use INI Files (Zed Shaw) by tptacek on Sunday October 10, 2010 @12:24PM (#1777303)
I used the word in the sense of "authoritative"; M-W might say "the most solemn and unvarying part". Which seems like the right term to use for "the form of the config the server actually honors".

You chose this as a design feature worth highlighting; you asked for people's opinions about it. I think it's flawed. Other tools I like are flawed in similar ways (tinydns, for example). Don't worry, I'm not leading a movement to overthrow you.

Re:Why Mongrel2 Doesn't Use INI Files (Zed Shaw) by zedshaw on Sunday October 10, 2010 @12:49PM (#1777369)
So you meant "official" then. I actually don't mind your rebuttals and over throw attempt, I live for that kind of debate. You just tend to write one-liners that really have no meaning and provide nothing useful as feedback.

For example, the problem you describe of a lack of an official configuration exists not because of the storage mechanism but because you have a storage mechanism. If you use a VCS then you have the same problem since the stored version can be different from the actual version running on any one server.

That begs the question how what's on the server gets to be different from the "official" stored version, and in this case it's people. It's always people.

So, let's look at the two mechanisms and compare:

1. VCS and just a config file means you have no idea who changed that config file. It's not in the VCS remember? Post crash forensics are then difficult and you really only have a diff.

2. VCS, a config file, and a sqlite storage with commit logs (we have those, you can see who changed stuff) means you have diffs of the config file from the VCS, and logs from the sqlite database. Diff the config in the VCS just like before, and if someone is going in the super back door to run SQL behind the scenes (which I don't get), then you know they did it. If they use the tools then you have the commit log in the datbase. You've now got way more data than you did with just config files.

But, if I were to extrapolate your real complain it is probably this:

"When the server breaks, I can not run 'diff' on the sqlite3 database config and see what changed some jackass could run raw sql against it and I would never know. Can I please have a way of comparing a given database to what a config file would produce? Kind of a config delta?"

With that in mind, I present to you the m2sh diff command ticket:

http://mongrel2.org/tktview/9e57f999d5cf23d899858521260b104b...

Thanks man!

Re:Why Mongrel2 Doesn't Use INI Files (Zed Shaw) by tptacek on Sunday October 10, 2010 @12:57PM (#1777395)
I'm glad I was able to help you clarify your design.

The "diff" command you propose seems like a lot of machinery just to make up for the fact that your design --- which is, in essence, a Unix take on the Windows Registry --- doesn't have a canonical human-readable form you can just pop open in a text editor.

In the interests of full disclosure: I'm very unlikely to ever use Mongrel2. My goal isn't to provide you with useful feedback. Instead, you opened up a conversation about this design decision, and I think the fact that it's prone to a particular failure mode is relevant --- especially to the (presumably numerous) HN readers who might think of emulating you.

I hope you read the "what year is it" comment as a judgement on INI files though, and not your code. We appear to share an opinion about INI files.

Re:Why Mongrel2 Doesn't Use INI Files (Zed Shaw) by zedshaw on Sunday October 10, 2010 @01:26PM (#1777454)
I'm glad I could help clarify your FUD.

> which is, in essence, a Unix take on the Windows Registry

Ah, the windows registry canard. You are the canard king man. Let's see you've pulled out djb and windows registry. I think your next linux fear buzzword will be "embrace and extend" or possibly "free as in freedom". That'll really enhance the F in your effing statements.

The flaw of the windows registry is not that it wasn't a text file, but that it was a massive monolithic database with all software configurations in it and no enforceable schema that couldn't be examined. Our database is only for mongrel2 and has a solid clean schema everyone can go look at with "sqlite3 .dump". So that's a pointless comparison.

And you keep using this word "canonical" where it doesn't apply, so let's use official. The "official" configuration file is loadable into a text file, but again, I think that's irrelevant for the use cases the design targets. Either you're doing your own setup and m2sh with mongrel2.conf is fine, or you're doing large scale automation and the database is better.

In my design you get both, plus more choice. In your design you only get the status quo configuration which sucks.

Also, if I do m2sh diff it'll be just for fun. Nobody really needs that since most people don't hire Ninjas.

Re:Why Mongrel2 Doesn't Use INI Files (Zed Shaw) by tptacek on Sunday October 10, 2010 @01:49PM (#1777509)
I responded to this comment here:

http://news.ycombinator.com/item?id=1777507

Re:Why Mongrel2 Doesn't Use INI Files (Zed Shaw) by bradleyland on Monday October 11, 2010 @12:31PM (#1780205)
I choked on the whole SQLite configuration aspect of Mongrel2 as well, but I continued to read further and started to see how it ties in to the entire mongrel2 philosophy: "Mongrel2 is an application, language, and network architecture agnostic web server that focuses on web applications using modern browser technologies.". Agnosticism is important to Mongrel2. If you open your mind to the concept, you can begin to see how using SQLite for configuration becomes a bike shed in the entire scope of the project.

If I'm a shop looking for a single web server to host apps written in a variety of languages, chances are that I'm going to have an opinion about the process used to configure and administer that server. That opinion is going to include the structure, process, and implementation of the entire server and management tool stack. The same thing happens with servers that use plaintext configuration formats as well. Have a look at the differences in Apache administration on Redhat based distributions versus Debian based distributions. Look at the number of tools that exist for managing Apache conf files. True, not everyone uses them, but the manageability of editing these text files by hand only scales so far. At some point, you turn to a tool that abstracts away the underlying (canonical?) configuration file. At that point, the underlying format -- text or SQLite -- becomes less important. Hence, Zed's repeated referral to this whole issue as a "bike shed".

What I think is really cool about this entire approach is that there will likely be a wide variety of tools built to manage different types of Mongrel2 hosting environments. The power of SQLite will come to bear when implementing these tools. SQL is a powerful tool for collecting and updating information. There are plenty of text libraries that perform similar tasks, but text is a lowest common denominator. At the end of the day, I think it's a pretty interesting choice.

Re:Why Mongrel2 Doesn't Use INI Files (Zed Shaw) by protomyth on Sunday October 10, 2010 @02:39PM (#1777633)
"My goal isn't to provide you with useful feedback."

Shouldn't that actually be the default goal of any reply?

Re:Why Mongrel2 Doesn't Use INI Files (Zed Shaw) by tptacek on Sunday October 10, 2010 @02:52PM (#1777666)
It would be if I was commenting on Shaw's blog. It's not, by default, on HN. Here, the goal is: discuss.
Re:Why Mongrel2 Doesn't Use INI Files (Zed Shaw) by yummyfajitas on Sunday October 10, 2010 @10:39AM (#1777131)
To get a canonical server configuration in raw text:

    $ echo '.dump' | sqlite3 mongrel2conf.sqlite > mongrel2conf.human_readable
    $ git add mongrel2conf.human_readable
    $ git commit -m "Added mongrel2 configuration"
Then type the following before invoking mongrel2:

    $ rm mongrel2conf.sqlite
    $ cat mongrel2conf.human_readable > sqlite mongrel2conf.sqlite
I agree with you - I prefer a human readable/git friendly config file. That's why I invested 10 minutes coming up with this solution and putting it into run_mongrel2.sh.

[edit: earlier version of this post was unnecessarily snarky.]

Re:Why Mongrel2 Doesn't Use INI Files (Zed Shaw) by mechanical_fish on Sunday October 10, 2010 @11:54AM (#1777235)
Congratulations, you have invented the "facsimile" that tptacek spoke of. Because your "canonical" dump file is just a dump file. At any given moment your server cannot be guaranteed to be in a configuration that is specified by your dump file, or by any version of your dump file. Why? Because the server exposes a tempting non-file-based interface to its internal state, and people who are trying to fix the servers at three in the morning will tend to use this to make "out-of-process changes".

If the server is broken, do you dare to naively overwrite its configuration with the version that is in git? The general answer is "no". Because two weeks ago somebody did an INSERT into the server's internal database to make Customer X's site work, and afterwords they didn't take time to properly dump the config and commit the dump to git. Because it was 3AM, and the edit was only one of seventeen switches that they were desperately flipping, and they just forgot. When you restart the server that config setting will be lost and the customer will crash and you will spend hours trying to recapture the lost state of the server.

You can prevent all this by making it difficult or impossible to alter the server config without altering the config file. But then why did we bother exposing the non-file-based interface in the first place?

Re:Why Mongrel2 Doesn't Use INI Files (Zed Shaw) by yummyfajitas on Sunday October 10, 2010 @12:15PM (#1777279)
The non-human readable interface is exposed for experts who wish to have python/ruby/whatever edit the sqlite interface. If you don't like this interface, don't use it. If your sysadmins are desperate to screw things up, they will.

If you really want to make it hard, your runmongrel2.py script can store the actual sqlite file in os.path.join('/tmp',str(uuid.uuid4())), thereby making it more difficult for sysadmins to find/edit.

Lastly, if you are truly paranoid, you can put something along these lines into your server restart script:

    $ echo '.dump' | sqlite /tmp/b5497e39-f55a-4234-8f5c-99ea8c1e8ee8 > /tmp/oldconfig.human_readable
    $ diff /tmp/oldconfig.human_readable main-git-repo/conf/config.sqlite_human_readable
Re:Why Mongrel2 Doesn't Use INI Files (Zed Shaw) by zedshaw on Sunday October 10, 2010 @12:21PM (#1777293)
> At any given moment your server cannot be guaranteed to be in a configuration that is specified by your dump file, or by any version of your dump file.

It's a database, and one that's used by the entire world to reliably store data guaranteed. So this is just false on technical merits.

> Why? Because the server exposes a tempting non-file-based interface to its internal state, and people who are trying to fix the servers at three in the morning will tend to use this to make "out-of-process changes".

Yet, you say people store things in version control. You have the exact same problem then. People can go and make changes out of band that aren't in version control and forget about them. Happens all the time, and that's a people problem.

What you seem to not understand is that you just need to take people out of the equation. Automation is the future of operations, and a system that allows any programming language with sqlite3 bindings manage the config is going to help with automation.

Once you don't have humans logging onto machines like it's the 90's then you've actually solved the problem. A config file won't do that.

Re:Why Mongrel2 Doesn't Use INI Files (Zed Shaw) by tptacek on Sunday October 10, 2010 @12:27PM (#1777312)
Your last sentence is hard to argue with, but it's worth pointing out that there's a cost to designing systems as if the universe you want to live in is the universe we actually live in. Let's call it "the Bernstein tax".
Re:Why Mongrel2 Doesn't Use INI Files (Zed Shaw) by zedshaw on Sunday October 10, 2010 @01:13PM (#1777428)
Ah the old DJB canard. I will counter your DJB with Alan Kay who's work inspired and/or created a huge amount of our modern technological world.
Re:Why Mongrel2 Doesn't Use INI Files (Zed Shaw) by tptacek on Sunday October 10, 2010 @01:16PM (#1777433)
Huh? I have no idea what this is intended to mean.

For what it's worth: I'm an ardent admirer of Bernstein, and happily pay the Bernstein tax on my systems. All I'm saying is that I recognize that I'm paying it.

Re:Why Mongrel2 Doesn't Use INI Files (Zed Shaw) by zedshaw on Sunday October 10, 2010 @01:32PM (#1777466)
Whenever someone wants to shoot down a new system that is weird, they pull out djb. "DJB did this thing with qmail and everyone hated it, but he wanted it his way and that's why nobody uses qmail."

It's another fear buzzword for sysadmins. It's effectively saying, "If you use Mongrel2 then it'll be like qmail. BEWARE!"

Re:Why Mongrel2 Doesn't Use INI Files (Zed Shaw) by tptacek on Sunday October 10, 2010 @01:48PM (#1777507)
If Mongrel2 was like qmail, it would in fact be very likely that I would use it. If you think I'm making up my admiration for Bernstein's work for the sake of argument, here's what 1 minute of Google searching turns up:

http://news.ycombinator.com/item?id=890613 http://news.ycombinator.com/item?id=890558

(These are a year old).

Not myself being a sysadmin, I couldn't tell you what the fear buttons are. I had that job in '96-'97 (the same time period where I read qmail, which I deployed in early beta versions at the rather popular ISP that employed me, because I love qmail), became a developer, and never looked back.

Re:Why Mongrel2 Doesn't Use INI Files (Zed Shaw) by danudey on Sunday October 10, 2010 @04:02PM (#1777807)
As a sysadmin for about ten years, this argument makes no sense to me.

Zed has created a new, flexible configuration method using sqlite, which lets me easily automate changes to the web server configuration. If version control is an issue, I can easily automate taking a human-readable, source-controllable dump of the database however often is reasonable.

Bernstein, on the other hand, decided that everything about the UNIX system was wrong, and built his system around what he asserted was a better way. All of qmail went into /var/qmail, which meant you had to either symlink everything in place anyway or add it to your $PATH, which caused unnecessary confusion.

He encouraged a different (non-human-readable) timestamp for log files (http://cr.yp.to/libtai/tai64.html), making it a huge hassle to take a quick glance through your log files for something without piping it through at least one extra process.

While we're talking about config files, he also used one config file for every variable (allowed hosts, virtual domains, etc), meaning that checking to make sure you configured something right involved cat'ing all kinds of files back and forth and scrolling up and down in your terminal.

He designed the qmail build process to hard-code the user IDs into the binaries, meaning you couldn't move a compiled version of qmail from one server to another unless all their UIDs/GIDs were identical. He also made qmail use inode numbers as file names for messages in the mail queue, meaning you couldn't move a queue from one server to another without renumbering each file.

Finally, he prevented people from shipping modified copies of his source code, meaning that adding any new features to qmail required patches, with a very significant chance that two patches for different features would interact in unexpected or inconsistent ways. It also meant that if you didn't have the original source tree and wanted to add one single feature (or even change the UIDs qmail ran under) there was a significant chance that you would lose features and/or your config would break.

Presumably this last rule was put in place to prevent people from fixing all the brain damage that went into qmail's ridiculous design. It caused nothing but massive headaches, and linux distributions had to jump through hoops to provide qmail as a package, since one single change to the source required the installer to download, compile, and install it for you.

qmail was a system administrator's nightmare because DJB had in his head what he believed to be a much better way to manage software on a UNIX system, and he went out of his way to make his software use that and force people to deal with it if he wanted to use that software (making qmail open source for most of its lifetime). The problem is that that might well have been a great system, if everything was using it, but since no one but him used his wildly varied system it just made qmail a headache. For as long as sendmail was the only alternative, it was a headache worth dealing with, but once Postfix came along most (open-minded) sysadmins jumped ship at the first opportunity.

Qmail is horrid, non-free software, but, like Windows, it was horrid, non-free software we all had to deal with for a long time. Now there are other, better options.

So I'll reiterate: qmail was a horrible mess that made my life, as a system administrator, a huge hassle. Mongrel2 and its DB config file, on the other hand, are ideas I'm ridiculously excited about, and can't wait to put into production, because the flexibility and power available in this sort of config philosophy are things I can do a great deal with.

Don't compare Zed to DJB. DJB looked down his nose at anyone who didn't like his pure, pristine system, and if you didn't like it, too bad for you. Zed, on the other hand, has gone out of his way to make his software work with whatever workflows we, as sysadmins, need to implement. That's worth some high praise, as far as I'm concerned.

Re:Why Mongrel2 Doesn't Use INI Files (Zed Shaw) by tptacek on Sunday October 10, 2010 @05:59PM (#1778071)
I skimmed this briefly, saw the sentence "qmail is horrid, non-free software", and didn't bother to read closely, knowing that it was written by someone who hasn't even taken the time to learn that qmail is in the public domain. You probably aren't even engaging the argument, which does not revolve around Bernstein's software being bad.
Re:Why Mongrel2 Doesn't Use INI Files (Zed Shaw) by nailer on Sunday October 10, 2010 @03:31PM (#1777753)
Djb did try and get users to replace init and the fhs in order to use his mail app. It's not a buzzword, it's a genuine issue that made people not want to use qmail.
Re:Why Mongrel2 Doesn't Use INI Files (Zed Shaw) by Goladus on Sunday October 10, 2010 @05:09PM (#1777974)
A configuration management system like cfengine or puppet is probably distributing files directly from revision control. Changes made to files on the system will likely be reverted unless someone specifically disables the agent.

> a system that allows any programming language with sqlite3 bindings manage the config is going to help with automation.

It's just a different config language, there's nothing that inherently allows more automation. The guy writing a gui interface might find that sqlite3 more useful. The guy using puppet and erb templates will probably prefer the text file.

The point of a configuration language is to provide a human interface. If you don't need the human interface then why do you expose configuration at all?

Re:Why Mongrel2 Doesn't Use INI Files (Zed Shaw) by Goladus on Sunday October 10, 2010 @11:43AM (#1777219)
Now imagine if everyone had their own version of this great idea and you had to make similar modifications to 100 different startup.sh scripts.
Re:Why Mongrel2 Doesn't Use INI Files (Zed Shaw) by yummyfajitas on Sunday October 10, 2010 @12:18PM (#1777288)
Contributor: "Please pull from me."

You, after examining diff outputs: "No, you are adding a second config file. Use the existing config file and deploy script."

Contributor: "Ok, fixed and rebased, now there is no trace of my stupid misconfiguration in the repository. I'll be sure to learn about the project conventions before committing in the future. My mistake."

Re:Why Mongrel2 Doesn't Use INI Files (Zed Shaw) by cdavid on Sunday October 10, 2010 @10:31AM (#1777120)
I don't follow: surely, a sql dump is possible with sqlite, and you could use just that in your VCS ?

As for ini file, the format is supported in the python stdlib, which makes it a natural default if you don't need too much. If .ini is not enough, I think in general you need something with a bit some logic, at which step using the database itself makes sense in mongrel2 case.

Re:Why Mongrel2 Doesn't Use INI Files (Zed Shaw) by tptacek on Sunday October 10, 2010 @11:58AM (#1777248)
Ew. I used to write a lot of Python, but I didn't know there was a standard INI parser. Gross.
Re:Why Mongrel2 Doesn't Use INI Files (Zed Shaw) by zedshaw on Sunday October 10, 2010 @12:24PM (#1777302)
What? You should totally be into .ini files. I mean, one incomprehensible difficult to automate bizarre ass user interface file is no better than any other.
Re:Why Mongrel2 Doesn't Use INI Files (Zed Shaw) by tptacek on Sunday October 10, 2010 @12:30PM (#1777320)
It's a false dichotomy to suggest that because INI files are incomprehensible, difficult to automate, and have bizarre asses, all text configuration files must be the same.

But, like your comment, that's just message board geekery. I wouldn't want to suggest that it's worth arguing about.

Re:Why Mongrel2 Doesn't Use INI Files (Zed Shaw) by zedshaw on Sunday October 10, 2010 @01:35PM (#1777474)
You keep using the logic phrases that you seem to not understand. It's not a false dichotomy, it's a generalization that has no external validity. It has internal validity though.
Re:Why Mongrel2 Doesn't Use INI Files (Zed Shaw) by tptacek on Sunday October 10, 2010 @01:54PM (#1777522)
Vocabulary really seems to be tripping us up today. Let's just cite the Nizkor list when critiquing arguments. For reference, I'm invoking #24, "False Dilemma":

http://www.nizkor.org/features/fallacies/false-dilemma.html

Specifically:

Bill: "We'll have to keep our configurations in a database and avoid flat text files." Jill: "Why?" Bill: "Because otherwise we'll be stuck with incomprehensible text files with bizarre asses."

Re:Why Mongrel2 Doesn't Use INI Files (Zed Shaw) by cdavid on Monday October 11, 2010 @03:51AM (#1779102)
I wonder: what kind of format is easy to automate ? What makes configuration file difficult to "automate" is their multiple format (especially on unix), the fact that it is not easy to reload a configuration (most applications don't even allow it) and that there is no easy way to check that your configuration matches how the app is configured (especially for bad softwares). The format only helps for the first point.
Re:Why Mongrel2 Doesn't Use INI Files (Zed Shaw) by spc476 on Monday October 11, 2010 @02:11AM (#1778962)
Cisco configuration files don't play nice with version control either. Anyone (with access) can log into any Cisco router, make changes and forget to write the running config to the startup config. Even if they do that, they might forget to copy the new startup config to the TFTP server. And even if they do THAT, they might not even bother doing the "svn commit" or "git commit" or "whatevervcweareusing commit". I view the sqlite database as the running configuration of Mongrel2. And from where I sit, it seems vastly simpler to do a diff of a configuration file stored in sqlite than it is to do a diff of a configuration from a Cisco router.