News for hackers. Stuff that matters.
Re:Heroku Postgres by sgrove on Tuesday August 30, 2011 @01:42PM (#2942438)
This is really exciting - the team behind this really cares about developers and their experience. They see a way things could be better, and regardless of technical difficulty, say, "why not?" Forking a db? Brilliant. Offload expensive task to passive followers? Obvious (in hindsight). Instant, brain-dead db provisioning? Of course.

Personally, the only terrifying thing about this is that when Heroku releases something, it's been in the works for a good long while, and they usually have a stream of incredible releases waiting right behind it. The mind boggles at what they're going to be doing next.

But we'll be first in line for it :)

Re:Heroku Postgres by jinushaun on Tuesday August 30, 2011 @02:49PM (#2942704)
When I read "forking" and "following" on there, it was a revelations: of course, that's so obvious! Those terms are much friendlier than "replication" or "master-slave". It describes the desired result, instead of the underlying technology.
Re:Heroku Postgres by pvh on Tuesday August 30, 2011 @02:53PM (#2942719)
If you had any idea how long it took to get that language right...
Re:Heroku Postgres by sgrove on Tuesday August 30, 2011 @03:40PM (#2942923)
I'd be curious to hear the alternatives, and the overall process used to arrive at the current language.
Re:Heroku Postgres by pvh on Tuesday August 30, 2011 @05:55PM (#2943445)
Hmm, this would make a good blog post. I'll draft something up.
Re:Heroku Postgres by DavidChouinard on Tuesday August 30, 2011 @08:17PM (#2943742)
Please do. There seems to be valuable lessons learnt there.
Re:Heroku Postgres by samstokes on Tuesday August 30, 2011 @05:01PM (#2943231)
I seem to remember the term "manumission" being floated around during the beta process... :)
Re:Heroku Postgres by joshuacc on Tuesday August 30, 2011 @05:08PM (#2943254)
Seconded.
Re:Heroku Postgres by boundlessdreamz on Tuesday August 30, 2011 @02:28PM (#2942607)
If there is anyone from heroku reading, please pay some money to http://www.sequelpro.com/ developers to develop postgresql compatibility. All GUIs available on OS X for postgres are horrible
Re:Heroku Postgres by pvh on Tuesday August 30, 2011 @02:49PM (#2942706)
We love Sequel Pro and would love to see Postgres support. Unfortunately, nobody has stepped forward with a credible claim to do the work and the existing developers are all, for obvious reasons, much more familiar with MySQL. If anyone is interested in working on that patch, I would personally chip in to a promising effort, and I would encourage anyone else interested to do the same.
Re:Heroku Postgres by boundlessdreamz on Tuesday August 30, 2011 @02:59PM (#2942750)
Offer a public bounty ? Just the basic support for viewing data would be really helpful.

This is my biggest pain point while working with heroku.

Re:Heroku Postgres by tednaleid on Wednesday August 31, 2011 @12:06AM (#2944260)
I'm a big fan of DBVisualizer (http://www.dbvis.com/), and have used it almost daily for ~6 years now on OSX. It's not a perfect OSX app (it's cross platform Java, but quite good looking for what that normally implies). Lots of power, doesn't crash, decent looking, and it works for pretty much every relational database out there (including MySQL, PostgreSQL, and Oracle).

I own a SequelPro license (as well as licenses for a couple of other non-free tools, RazorSQL and Querious) and I always come back to DBVisualizer.

Re:Heroku Postgres by jinushaun on Tuesday August 30, 2011 @02:43PM (#2942680)
I love using Sequel Pro for MySQL, but also wish it had support for SQLite and PostgreSQL. It's a great app and better than all the other OSX clients that I can find on the internet, including the non-free ones.

It's open source, so nothing's stopping anyone from adding support for other SQL servers.

Re:Heroku Postgres by jabkobob on Wednesday August 31, 2011 @04:57AM (#2944847)
I have contributed a little bit to Sequel Pro, and I'm a bit familiar with the codebase. Adding support for PostgreSQL is not trivial, a lot of MySQL is hardcoded into the user interface. There have always been many requests for PostgreSQL compatibility, but it would require A LOT of refactoring.

(Disclaimer: I haven't been involved much with SP for almost a year now, so possibly there has been progress in that time)

Re:Heroku Postgres by stu on Thursday September 01, 2011 @11:21AM (#2950357)
As one of the current Sequel Pro developers I can backup jabkobob on this one, it's not an easy task. It's not that we're ignoring the requests we get to add support for PostgreSQL or any other database system for that matter, it's simply a matter of time, of which most of the core devs don't have just now. We have a lot of plans around this area (as well as others) for the app, but it's going to take time.

Thanks for everyone's support.

Re:Heroku Postgres by jasonjei on Tuesday August 30, 2011 @03:53PM (#2942976)
Is Navicat OK? I've been using it for a while simply because it was one binary that did SQLite, MySQL, Oracle, MS-SQL, and PgSQL.
Re:Heroku Postgres by dwillis on Tuesday August 30, 2011 @08:05PM (#2943711)
I've used Navicat Premium with Postgres and have had no issues. Pricey, but it works.
Re:Heroku Postgres by dewiz on Tuesday August 30, 2011 @08:07PM (#2943712)
have you considered Navicat ? there is also a free version and it runs on windows, osx, linux
Re:Heroku Postgres by smickie on Tuesday August 30, 2011 @05:25PM (#2943327)
I would pay good money for a postgresql client as superb as sequelpro, and I think a lot of other developers would. Someone is going to cash in on this gap if sequelpro don't beat them to it.
Re:Heroku Postgres by pbreit on Tuesday August 30, 2011 @08:22PM (#2943750)
How about a http://kickstarter.com project?
Re:Heroku Postgres by jackowayed on Wednesday August 31, 2011 @12:12AM (#2944277)
But it's kinda kickstarter in reverse since you have people wanting to give money but no one picked to do the work. It's a bounty.

I worked on and almost launched an open source bounty site a couple years ago. But the closer I got to launch the more I realized that I hadn't adequately solved all of the hard UX problems (mostly around a: picking who's going to work on it, trying to avoid both having two people separately do it due to know central coordination and having one person say "yeah, I'm going to do this!" and then not get around to it and hold the project back, and b: how to handle it when someone does technically address the bounty, but in a way that is unsatisfactory either because bounty-placers did not perfectly state their needs or because of poor quality.

Re:Heroku Postgres by knotty66 on Tuesday August 30, 2011 @03:17PM (#2942832)
What's wrong with pgAdmin ?
Re:Heroku Postgres by sirn on Tuesday August 30, 2011 @04:06PM (#2943031)
Ugly on any non-Windows platform, too complicated for simple use (e.g. enter a query here and show me a result or show me a schema), etc. Well, just compare Sequel Pro screenshot with pgAdmin3 :)

Disclaimer: I actually like pgAdmin3, especially the visualized explain page.

Re:Heroku Postgres by mahmud on Tuesday August 30, 2011 @08:03PM (#2943703)
PgAdmin3 on Ubuntu here: gorgeous!
Re:Heroku Postgres by sandGorgon on Tuesday August 30, 2011 @03:19PM (#2942842)
No can afford macbooks - something with Ubuntu please..
Re:Heroku Postgres by swilliams on Tuesday August 30, 2011 @02:07PM (#2942529)
It would seem that their acquisition by SalesForce hasn't slowed them down at all, in fact I'd go as far to say that they've been even more productive since that happened. I'm curious about how that whole process has gone; has SalesForce provided more support/funding/resources? Or just removed distractions and allowed them to focus on their product?

Nice job Heroku.

Re:Heroku Postgres by pvh on Tuesday August 30, 2011 @02:52PM (#2942715)
Suffice it to say, any rumours of our demise were greatly exaggerated. So much more to come and yes, we are hiring, including for our Postgres team. (Resumes/github accounts/etc to jobs(at)heroku.com.)
Re:Heroku Postgres by olefoo on Tuesday August 30, 2011 @01:42PM (#2942436)
Just a little usability nit, is the pricing per month, per year, lifetime?

It's not clear from the pricing page, and I'm assuming it's per month, in which case it's reasonable, especially given the value added services being offered, especially the ability to fork the database and the automated creation of read slaves.

It looks like a solid offering. It's probably not right for companies that are subject to HIPAA or PCI compliance requirements, and there's no information on the use of compiled extensions in the db which may limit it's utility if you're wanting to use PostGIS or other specialist datatypes.

Re:Heroku Postgres by pvh on Tuesday August 30, 2011 @03:02PM (#2942761)
I would say that at this point you're right, we're probably not right yet for companies subject to HIPAA or PCI regulation.

The advanced Postgres datatypes are pretty much my favorite thing in the world. We use HStore internally all over the place, as well as some more exciting projects that we haven't said too much about yet. As I mentioned elsewhere in the thread, if you're interested in trying out some of those extensions, contact me directly.

Re:Heroku Postgres by olefoo on Wednesday August 31, 2011 @01:00AM (#2944397)
I'm a real fan of PostGIS and like ltree and hstore a lot. Heroku isn't quite right for my needs at this time, but I would definitely consider it if I had a use case that needed an extremely fast public facing postgres installation.

I am curious as to whether you do pro-rated pricing?

Re:Heroku Postgres by sabman on Wednesday August 31, 2011 @11:00AM (#2945810)
@Olefoo you can get PostGIS on Heroku via SpacialDB. SpacialDB is cloud hosted PostGIS and a convenient RESTful API. We have a heroku addon too currently in private beta. Check out our devcenter here: http://devcenter.spacialdb.com/ and let us know if you would like a beta invite.
Re:Heroku Postgres by pvh on Wednesday August 31, 2011 @02:03AM (#2944506)
As of today, we only charge for the time you use. I don't imagine we'd ever need to change that, but never is a long time.
Re:Heroku Postgres by olefoo on Wednesday August 31, 2011 @03:13AM (#2944630)
That is interesting, there are times when I want to use PG as an ETL tool and take a midsized ( < 2tb data ) set and load it up and slice and dice it and produce some output for other programs and then have it go away. So, I'll keep you guys in mind.
Re:Heroku Postgres by simonw on Tuesday August 30, 2011 @03:28PM (#2942877)
That confused me too - the https://postgres.heroku.com/pricing page should definitely mention that it's per month. I'm also not clear on the difference between the Ronin and Fugu plans, both of which are 1.7 GB of RAM but one which is twice the price of the other.
Re:Heroku Postgres by wisty on Wednesday August 31, 2011 @02:25AM (#2944556)
They are built on different EC2 instances.

Ronin = Small Instance: 1.7 GB of memory, 1 EC2 Compute Unit (1 virtual core with 1 EC2 Compute Unit), 160 GB of local instance storage, 32-bit platform. ~$60/month spot.

Fugu = High-CPU Medium Instance: 1.7 GB of memory, 5 EC2 Compute Units (2 virtual cores with 2.5 EC2 Compute Units each), 350 GB of local instance storage, 32-bit platform. ~$120/month spot.

Ikea = Large Instance: 7.5 GB of memory, 4 EC2 Compute Units (2 virtual cores with 2 EC2 Compute Units each), 850 GB of local instance storage, 64-bit platform. ~$240 / month spot.

Zilla = High-Memory Extra Large Instance: 17.1 GB memory, 6.5 ECU (2 virtual cores with 3.25 EC2 Compute Units each), 420 GB of local instance storage, 64-bit platform. ~$360/month spot.

Baku = High-Memory Double Extra Large Instance: 34.2 GB of memory, 13 EC2 Compute Units (4 virtual cores with 3.25 EC2 Compute Units each), 850 GB of local instance storage, 64-bit platform. ~$720/month spot.

Mecha = High-Memory Quadruple Extra Large Instance: 68.4 GB of memory, 26 EC2 Compute Units (8 virtual cores with 3.25 EC2 Compute Units each), 1690 GB of local instance storage, 64-bit platform. ~$1440/month spot.

I think Heroku will pay less than the spot price for the raw server, but fork out bandwidth, storage, EBS IO, backups and so on. Amazon sells its basic servers cheap, but virtually nothing is included.

It looks like their margins go up at Zilla, but maybe the high-RAM instances need more bandwidth and backups.

Re:Heroku Postgres by msmith on Tuesday August 30, 2011 @02:22PM (#2942586)
I'm sure it's monthly. The price tiers are the same as the full Heroku platform (http://heroku.com/pricing), which is billed monthly.
Re:Heroku Postgres by wtn on Tuesday August 30, 2011 @02:16PM (#2942563)
Basic plan is $200/month.
Re:Heroku Postgres by firemanx on Tuesday August 30, 2011 @02:44PM (#2942681)
Do they still limit the usage of user defined functions and contrib modules, or did that open up? I would love to use Heroku for some lightweight data warehousing that I've got going, but it's still pretty dependent on functions/sprocs for performance reasons.

I looked for any more documentation about that, but the only official word I have from them in the past is the support ticket I filed last year stating that they don't support "additions" like user defined functions or the various contrib modules.

Re:Heroku Postgres by pvh on Tuesday August 30, 2011 @02:47PM (#2942700)
Our production service has always supported UDFs, though the free sandbox service does not.

Unfortunately, Postgres makes it difficult to install extension modules without superuser. That said, we've wanted to support contrib modules for ages. Lately, encouraged by the work Dimitri Fontaine has been doing on extensions, we have a pilot project going which includes support for hstore, pgcrypto, pg_trgm, and of course, postgis. Feel free to contact me at pvh(at)heroku.com if you're interested in taking it for a spin.

Re:Heroku Postgres by samstokes on Tuesday August 30, 2011 @03:08PM (#2942789)
We're running user-defined functions on Heroku right now (using a functional index on a custom hash function to get fast space-efficient queries on a string column of a large table).
Re:Heroku Postgres by andypants on Tuesday August 30, 2011 @01:47PM (#2942458)
This is really awesome, just a few downsides (for me):

I don't use postgres - I hope heroku expands this kind of service to more databases (although I don't think it's likely in the near future).

The smallest database is also pretty expensive. I wouldn't mind a cheaper plan for less resources.

Only being able to create one database of each size is just weird, especially if forking or following. Can anybody confirm that this is a limitation? (I only read about this from one of the other comments here)

Re:Heroku Postgres by pvh on Tuesday August 30, 2011 @03:06PM (#2942780)
We are unlikely to offer MySQL at any point in the near future. We feel that across the board, Postgres has the best data robustness guarantees, the most powerful query optimizer, the strongest replication solutions, the most useful datatypes, and the greatest community driving, developing and supporting it.

We hear you on the cheaper plan; watch this space.

Last, there is no restriction on creating only one database of each size.

Re:Heroku Postgres by latortuga on Tuesday August 30, 2011 @03:39PM (#2942920)
Honestly when you list the merits of Postgres like that you sound like a used car salesman. "Of course you think you want that but really you want all of this." We all get that Postgres is great. Saying "no we won't implement a feature you, our customer, wants" like this just rubs me the wrong way.

Is there some technical reason that you can't offer both?

Re:Heroku Postgres by socratic on Tuesday August 30, 2011 @03:51PM (#2942971)
What would you like Heroku to do with a MySQL on EC2 offering that Amazon is not currently doing with their MySQL on EC2 offering (Amazon RDS)? They are both local, use the same underlying infrastructure, and have Heroku add-ons, no?

Presumably, their decision to only support Postgres is partly competition (many others provide MySQL on EC2), and partly technical. How would you justify the technical merits of the decision, without sounding "like a used car salesman"?

Re:Heroku Postgres by pvh on Tuesday August 30, 2011 @04:02PM (#2943013)
Sorry, I thought you wanted to know why we chose Postgres over MySQL.

Those are the specific features I think that make Postgres a better choice for both use as my day-to-day database and why we chose Postgres over MySQL for building a database service.

Certainly you could build a similar service on top of, say, RDS, but without the features listed above, I think it would take a lot more work and not come out as well.

Hope that helps.

Re:Heroku Postgres by bad_user on Tuesday August 30, 2011 @01:58PM (#2942498)
In case you're using Mysql, give Postgres a try -- it's really awesome.

I do agree about them being expensive. Their services in general are too expensive for me -- personally I prefer to manage my own EC2 instances. But they do provide value worth paying for depending on your needs.

Re:Heroku Postgres by khoffrath on Tuesday August 30, 2011 @02:45PM (#2942687)
For a similar service using MySQL you can try http://xeround.com/. (Disclaimer: I haven't tried it myself and have no association with them)
Re:Heroku Postgres by rbranson on Tuesday August 30, 2011 @04:00PM (#2943001)
Is this still using the EBS RAID that you guys mentioned in a blog post a while ago? If so, how do you avoid any slow I/O requests (which plague EBS) stalling all I/O to the volume?
Re:Heroku Postgres by pvh on Tuesday August 30, 2011 @04:30PM (#2943115)
Actually, Postgres WAL support helps mitigate this reasonably well, and a well tuned application has something like a cache hit rate of >99% during normal operation.

That said, yes, EBS can have unstable performance. We monitor for several kinds of related problems and in the most dire of straights can perform a hardware migration to a quieter node which generally clears issues up.

Re:Heroku Postgres by jvehent on Tuesday August 30, 2011 @01:38PM (#2942409)
"Forget daily backups, Continuous Protection redundantly archives data to high-durability storage as it is written, ensuring that it is safe no matter what."

I'm sorry, but daily backups are not only for high availability, but also for point in time recovery.

What if $dev drops the user table by mistake ? Do they provide backup for that ?

Re:Heroku Postgres by pvh on Tuesday August 30, 2011 @03:08PM (#2942790)
We have found that large production databases do not appreciate having frequent full SQL dumps taken from them. We offer an automatic daily snapshot option because we believe that for a production database it's really not wise to take dumps more frequently than that, though there's nothing stopping you from rolling your own.
Re:Heroku Postgres by olefoo on Tuesday August 30, 2011 @01:46PM (#2942452)
You can set up Point In Time Recovery (PITR), and replay recovery logs. It's not clear from Heroku's pages whether their solution is set up that way.
Re:Heroku Postgres by jpitz on Tuesday August 30, 2011 @04:21PM (#2943081)
It sounds like their solution is set up that way.

From https://postgres.heroku.com/#protect : "Every change to your data is written to write-ahead logs, which are shipped to multi-datacenter, high-durability storage. In the unlikely event of unrecoverable hardware failure, these logs can be automatically 'replayed' to recover the database to within seconds of its last known state."

Re:Heroku Postgres by moe on Tuesday August 30, 2011 @05:44PM (#2943408)
It sounds like their solution is set up that way.

I think the question is whether the customer will be able to perform a point in time recovery, and what the size and resolution of the backlog is.

Re:Heroku Postgres by pvh on Tuesday August 30, 2011 @05:56PM (#2943448)
Not yet.
Re:Heroku Postgres by jpitz on Wednesday August 31, 2011 @08:52AM (#2945288)
Thank you for that clarification.
Re:Heroku Postgres by joevandyk on Tuesday August 30, 2011 @01:46PM (#2942451)
You can make snapshots of the data.
Re:Heroku Postgres by Fluxx on Tuesday August 30, 2011 @04:21PM (#2943080)
This is really awesome! A couple questions though:

* What do I do if I need to tune/configure Postgres for my workload.

* How is the performance of transferring all of the queries and responses across the internet?

Keep in mind I've never used postgres before so these may be moot points.

Re:Heroku Postgres by pvh on Tuesday August 30, 2011 @04:26PM (#2943102)
We've done a lot of tuning focused on web applications already, our experience is that mostly people with problems just need to add an index.

Performance depends on where you are querying from. We've found that if you are inside EC2 network overhead is typically <1ms.

Re:Heroku Postgres by joevandyk on Tuesday August 30, 2011 @01:35PM (#2942390)
Although they only let you create one database per plan, you can use schemas to emulate multiple databases.

http://www.postgresql.org/docs/9.0/static/ddl-schemas.html

Re:Heroku Postgres by pvh on Tuesday August 30, 2011 @03:28PM (#2942880)
Just to clarify: You're almost right. You can, in fact, create multiple databases of any plan you please, but yes, within each of those databases you can use schemas to implement something akin to multiple databases without resource isolation, but with the ability to share data.
Re:Heroku Postgres by pbh on Tuesday August 30, 2011 @03:54PM (#2942980)
Any chance of PL/python or similar?

I guess I'm probably one of like three users of PL/python, especially since it's untrusted. Worth a check, though!

Re:Heroku Postgres by pvh on Tuesday August 30, 2011 @04:22PM (#2943084)
No plans to support untrusted languages as yet. Too many foot-guns built into that. You couldn't even believe some of the evil, insane things that we've done with that.

Trusted languages are far more likely. This one looks really, really cool: http://code.google.com/p/plv8js/wiki/PLV8

Re:Heroku Postgres by drewcrawford on Tuesday August 30, 2011 @02:26PM (#2942601)
This is ridiculous, but I would love a REST interface to a SQL server. I do a lot of AppEngine work and although I love the datastore and write super-optimal queries, I would really love the ability to pull a SQL datasource in every once in awhile.
Re:Heroku Postgres by pvh on Tuesday August 30, 2011 @03:02PM (#2942763)
Check out the extremely cool HTSQL from Clark Evans!
Re:Heroku Postgres by volkadav on Tuesday August 30, 2011 @08:48PM (#2943796)
Not exactly RESTful but you might find some utility in this, developed at and open-sourced by the NYT: http://code.nytimes.com/projects/dbslayer

(edit to add: this is mysql-only, unfortunately :) I forgot about that part when I remembered it off-hand.)

Re:Heroku Postgres by hopeless on Tuesday August 30, 2011 @03:01PM (#2942753)
I just realised that Heroku have a problem: when I looked at the page, and looked at the subdomain URL, I couldn't tell if this was a real offering or a fake app put up by someone else.
Re:Heroku Postgres by pvh on Tuesday August 30, 2011 @03:11PM (#2942802)
Yes, it's the real deal, just like blog.heroku.com and all our other services.

In order to reduce this ambiguity we are gradually moving towards placing user apps on a different domain; because we approach these kinds of issues very gradually this will probably take quite some time to fully complete, but as of today all new Cedar stack applications are hosted on herokuapp.com.

Re:Heroku Postgres by hopeless on Tuesday August 30, 2011 @04:30PM (#2943117)
Ah, good to know. I haven't got around to creating a Cedar-based app yet
Re:Heroku Postgres by mberning on Tuesday August 30, 2011 @07:57PM (#2943696)
This is cool, seems to have a pretty fair price structure. I still don't understand their shared database pricing. 5mb for free, then $15 for 20gb. I need like 100 megs for 5 bucks.
Re:Heroku Postgres by laran on Tuesday August 30, 2011 @06:20PM (#2943498)
This blew my mind. Amazing stuff. Heroku is absolutely my favorite platform.
Re:Heroku Postgres by francoisdevlin on Tuesday August 30, 2011 @01:42PM (#2942437)
I think the Fugu plan is a typo, if you're reading this Heroku guys...
Re:Heroku Postgres by wccrawford on Tuesday August 30, 2011 @01:45PM (#2942446)
'has' a typo, you mean.
Re:Heroku Postgres by njharman on Tuesday August 30, 2011 @10:27PM (#2943977)
need GIS

And confused by "one database only". Does that mean CREATE DATABASE databases? If so why?

Re:Heroku Postgres by abhishektwr on Tuesday August 30, 2011 @10:40PM (#2944011)
Ya me too, no mention of PostGIS.
Re:Heroku Postgres by kashifr on Wednesday August 31, 2011 @04:21AM (#2944772)
Hi co-founder of SpacialDB here. Cloud hosted PostGIS is already available as a Heroku plugin, stand-alone and an easy to use API. Check out http://devcenter.spacialdb.com/ for documentation and you can sign up via the Heroku addons page or http://beta.spacialdb.com/
Re:Heroku Postgres by pvh on Wednesday August 31, 2011 @02:05AM (#2944509)
PostGIS is currently in a pilot phase. If you're serious, please contact me as described elsewhere on this thread.