News for hackers. Stuff that matters.
Re:How to build a news app that never goes down and costs yo... by danso on Monday February 25, 2013 @12:02PM (#5279972)
I had the pleasure of working with NPR's data news chief, Brian Boyer, who taught me a lot about actual good software practices.

I agree with his preaching the power of flat-files. Not that flat-files should be used to do things that they inherently can't...but that too many projects (or hobby apps) don't consider them and then spend as much time figuring out how to keep their server from crashing. I find it pretty amazing that they have only one small EC2 instance for their news apps (this is separate from the NPR.org site overall) just to do cron jobs.

Flat files, of course, require good planning...not least of which involves an accurate gauging of how often an app's data needs to be refreshed. But I like that kind of planning and thinking more than I do the kind it takes to maintain a stable server.

Re:How to build a news app that never goes down and costs yo... by ericcholis on Monday February 25, 2013 @12:27PM (#5280153)
Recently I re-engineered an ecommerce site that takes many things from the "flat-file" montra. It's amazing how little database interaction you actually need for many sites.

I think the database-driven content comes primarily from kitchen sink packages like Wordpress. More often than not, it's better to get an overall look at your structure and decide what needs to be dynamic and what can be "static".

When talking about it internally, we refer to "flat-file" as "generated". Simply meaning that it's dynamically created by a task or user interaction.

Re:How to build a news app that never goes down and costs yo... by wting on Monday February 25, 2013 @12:48PM (#5280292)
The file system is the original "database". I solved an interview question recently using a few Linux utilities, pipes, and flat files. It was a bash one liner that impressed the interviewer, but then he asked me to solve it the "real" way.

Both databases and file systems use B-trees to implement fast read / writes, it's just that SQL databases enforce structure. OTOH, using flat files moves data checking into application space and losing certain ACID properties.

Similar trade offs are made between NoSQL and SQL, dynamic and static languages, but I digress...

Re:How to build a news app that never goes down and costs yo... by ewang1 on Monday February 25, 2013 @01:03PM (#5280402)
I would imagine "writes" are still being written to the database, which then triggers the re-generation of the appropriate flat-file(s).
Re:How to build a news app that never goes down and costs yo... by wting on Monday February 25, 2013 @01:17PM (#5280525)
No, given a bunch of Apache logs I needed to find the top 10 queries that met conditions A and B.

1. grepping for conditions and extracting the query with sed

2. appending a char to a flat file (i.e. increasing query count by 1)

3. sort by largest files and get the top 10

Parsing log files[0]:

  for ./*.log -type f -exec echo 1 >> $(grep cond_A {} | grep cond_B | sed -E "s:.*(query).*:./results/\1.txt:") \;
Finding top 10 results:

  ls ./results/*.txt -Sl | head
[0]: Untested code, and it only grabs one line per log file instead of grepping all matching lines in the log file. I'd have to move the grep to the outside and loop the `echo 1 >>` command, but you get the point.
Re:How to build a news app that never goes down and costs yo... by mbreese on Monday February 25, 2013 @03:31PM (#5281564)
You probably could have done the same thing in one line:

    cat *.log | grep condA | grep condB | sed 'some regex to get rid of dates, etc' | sort | uniq -c | sort -nr | head -n 5
Or something like that... Not very efficient, but it would work in a pinch. I actually do something like this all the time for large datasets. In the time it would take me to write something better, this set of commands is already done.
Re:How to build a news app that never goes down and costs yo... by lotsofcows on Tuesday February 26, 2013 @03:34AM (#5284603)
Those that don't know awk are doomed to complex pipe lines.
Re:How to build a news app that never goes down and costs yo... by legutierr on Monday February 25, 2013 @03:04PM (#5281393)
Maybe I shouldn't be, but I am kind of surprised that you can do an ecommerce site using mostly generated files. The two things that I wonder about are (1) displaying cart/user information on every page (is it all javascript?) and (2) csrf in the forms.

Also, I would assume that javascript becomes a requirement for the functioning of the site; I always considered ecommerce to be the one part of the web were you wanted to be able to operate without javascript, so as not to turn anyone away.

I found this blog post that says that in 2010, 2% in the US had JS disabled:

http://developer.yahoo.com/blogs/ydn/posts/2010/10/how-many-...

Re:How to build a news app that never goes down and costs yo... by pjungwir on Monday February 25, 2013 @04:28PM (#5281936)
You can also integrate with a third-party shopping cart then load a small iframe to show "2 Items in the Cart." Not my favorite way to build a site, but it is very inexpensive to build, and for small vendors the site becomes entirely static. Even if you build it in Rails, you can use page-level caching so after the first hit it's served as a static html file.
Re:How to build a news app that never goes down and costs yo... by jasonwatkinspdx on Monday February 25, 2013 @05:24PM (#5282285)
Page level caching is one of the most underused features of rails. Most applications show anonymous users identical content, and can tolerate that content being cached briefly. Setting this up gives you slashdot protection as well as access to a last chance degraded mode if there's a chain of catastrophic failures further into the app stack.
Re:How to build a news app that never goes down and costs yo... by ericcholis on Monday February 25, 2013 @08:44PM (#5283309)
Exactly, this is the general idea of what I do in the ecommerce app I mentioned.
Re:How to build a news app that never goes down and costs yo... by ericcholis on Monday February 25, 2013 @04:38PM (#5282012)
I guess I should have been more specific about my strategy for "generated" content in an ecommerce environment. The traditional ecommerce elements that should be real-time are; such as carts, pricing, availability, etc. Everything else is generated on update via our [custom] inventory management software.

The closest analogy I can give is Varnish's Edge Side Includes. It's not exactly the same, but it's very similar.

Re:How to build a news app that never goes down and costs yo... by subpixel on Monday February 25, 2013 @12:44PM (#5280256)
FYI I'd be totally interested in more info about how you achieve this in an e-commerce environment. Blog post?
Re:How to build a news app that never goes down and costs yo... by snowwrestler on Tuesday February 26, 2013 @08:41AM (#5285428)
Serving flat files...isn't that what a caching layer is for? I don't understand why someone would go to extra trouble architecting an app around flat files on the back end, rather than building it the easiest way and sticking Varnish or nginx in front it.
Re:How to build a news app that never goes down and costs yo... by stcredzero on Monday February 25, 2013 @02:27PM (#5281111)
> I had the pleasure of working with NPR's data news chief, Brian Boyer

I found the unexplained use of the word "Boyerism" in this article to be confusing and unprofessional. I have nothing against the use of slang, or the expectation that readers will need to Google terms: this is the reality of our culture in the age of the Internet and excellent search. However, the casual and unexplained use of a private group's inside joke reveals a lack of awareness of how search affects interacts with culture. Is it a deliberate attempt to confuse and snub the larger audience, or is in unintentional?

Re:How to build a news app that never goes down and costs yo... by gottagetmac on Monday February 25, 2013 @03:38PM (#5281620)
I think you are bothered too easily.
Re:How to build a news app that never goes down and costs yo... by danso on Monday February 25, 2013 @04:25PM (#5281918)
I agree with you...I didn't even realize that Boyer wasn't introduced in the OP by name. I mentioned his name because he is well known in news circles and frequently evangelizes about flat files. The OP should have said who Brian Boyer was even if the NPR apps blog is meant as insider baseball for news devs.