News for hackers. Stuff that matters.
Re:At 22 years old, Postgres might just be the most advanced... by kev009 on Wednesday December 05, 2018 @02:29PM (#18611617)
I'm a big fan of Postgres and it is basically the only DB I use but the title is naive and the post is low content. SQL Server, Oracle, and DB2 are crown jewels products and each have substantial and difficult to implement niche or extreme scale features or other sweet spots that no open source databases come close to after two decades, and they weren't sitting still during that time either.
Re:At 22 years old, Postgres might just be the most advanced... by judgardner on Wednesday December 05, 2018 @03:31PM (#18612213)
Agree. I have been using Postgres and Oracle in production since ~2000 (oracle 8i). Postgres is an amazing project but the oracle core database has billions of R&D dollars. Don't confuse the oracle core database with the awful applications they hang off of it. Oracle will still destroy pg for most OLAP workloads, now you have to decide if the insane licencing cost is worth it.

Also just want to add an oracle developers post that I find hilarious...

https://news.ycombinator.com/item?id=18442941

Re:At 22 years old, Postgres might just be the most advanced... by shdh on Friday December 07, 2018 @07:52PM (#18633082)
> https://news.ycombinator.com/item?id=18442941

This was a gem. Really makes you appreciate not having to work on such a codebase.

Re:At 22 years old, Postgres might just be the most advanced... by rolleiflex on Wednesday December 05, 2018 @03:03PM (#18611891)
One thing Postgres definitely doesn't do, and I wish it did, is embedded. This is usually the place that people use SQLite, for example, apps that are clients, but still need to do substantial work on the client side.

The problem is, SQLite does have a few limitations that it doesn't make obvious: some of the more complex SQL syntax that Postgres supports are missing, but more importantly, it can only ever do one write or read to the database predictably. (Don't believe me? Try this: https://gist.github.com/mrnugget/0eda3b2b53a70fa4a894)

I know that SQLite supports concurrent writes and reads technically, but it predictably leads to 'database is locked' issues. In practice you have to use it in a fashion that it only ever does one thing, read, or write. No multiple writes. No single write and multiple readers, no single write and single reader. Just read — or write. (Edit: WAL doesn't help with this either.)

Or you're in a world of hurt. I'd love to be able to move to Postgres just to get away from that.

Re:At 22 years old, Postgres might just be the most advanced... by coleifer on Wednesday December 05, 2018 @03:44PM (#18612338)
> some of the more complex SQL syntax that Postgres supports are missing, but more importantly, it can only ever do one write or read to the database predictably

I don't know...it's recently added upsert compatible with Postgres syntax as well as support for window functions.

Additionally, sqlite has a sophisticated json extension.

> it can only ever do one write or read to the database predictably

WAL mode addresses this, allowing an arbitrary number of readers with a writer.

For concurrent applications just use a dedicated write thread and enqueue writes...your example is contrived. Connection management, transaction scope management, etc, can all work in your favor. Writes occur in a few milliseconds, meaning you only need to hold the exclusive lock very briefly in most situations.

More information here: http://charlesleifer.com/blog/going-fast-with-sqlite-and-pyt...

Re:At 22 years old, Postgres might just be the most advanced... by rolleiflex on Wednesday December 05, 2018 @04:42PM (#18612896)
The gist I linked is extremely basic, just insert and read, really, and it does use WAL by default. It'll still end up with database lock, and from there it's all shallow waters — reverting from a crashed database lock is not trivial and can cause data loss if handled improperly.

Not all writes are couple milliseconds. I regularly have transactions that take 15< seconds, and it's not because the data size that's being input is large, it's just that the SQL is recursive and complex enough to warrant recursion (storing graph data).

Re:At 22 years old, Postgres might just be the most advanced... by manigandham on Wednesday December 05, 2018 @03:09PM (#18611951)
This is almost always a driver/wrapper problem, which seems to be the case here. SQLite supports multiple readers without issue that only block slightly when a write transaction is committed, and there's also WAL mode with even better concurrency that avoids locks altogether for reads.
Re:At 22 years old, Postgres might just be the most advanced... by rolleiflex on Wednesday December 05, 2018 @04:46PM (#18612926)
I've had the same problem with Python, not just Go. If it's a driver problem, it's a very widespread one, which would point out to SQLite driver API being in need of some love if so many folks who write SQLite drivers are making the same mistake.
Re:At 22 years old, Postgres might just be the most advanced... by blattimwind on Wednesday December 05, 2018 @05:04PM (#18613117)
The Python driver is buggy-by-default. This has been documented in recent versions [1], and is kinda sane for the undo log, but not so good if you want to use snapshot isolation. I guess many bindings would make the same decision. Basically they delay the emission of BEGIN until the first DML/DDL is executed.

If you fix this (which is easy), then WAL works just fine. Note that transactions, as is standard, span from BEGIN/COMMIT to COMMIT/ROLLBACK.

See e.g. https://docs.sqlalchemy.org/en/latest/dialects/sqlite.html#t... and https://docs.sqlalchemy.org/en/latest/dialects/sqlite.html#p...

[1] Really it's more like "encoded": https://docs.python.org/3/library/sqlite3.html#controlling-t... -- you already need to know the issue to understand that this section specifies the root cause.

Re:At 22 years old, Postgres might just be the most advanced... by rolleiflex on Wednesday December 05, 2018 @05:10PM (#18613159)
Interesting! The last time I used this with Python was 2014, and this, alongside the botched migration to 3 was the reason I moved on from Python to Go. Glad to hear things have been improving.
Re:At 22 years old, Postgres might just be the most advanced... by manigandham on Wednesday December 05, 2018 @05:39PM (#18613429)
SQLite has billions of global installs and is tested and certified under incredibly stringent conditions for use in applications like aircraft avionics. Unless there is serious evidence to the contrary, it's pretty much guaranteed that it's not the cause of the issue.

How many drivers have you actually tested? There are 2 comments from 2016 on that thread saying other drivers do work fine.

Re:At 22 years old, Postgres might just be the most advanced... by combatentropy on Wednesday December 05, 2018 @10:32PM (#18615141)
> it predictably leads to 'database is locked' issues

Do you mean your application actually throws the error "database is locked," and dies without simply waiting for the lock to release? This behavior definitely goes against the grain of the documentation. From the SQLite website:

"SQLite supports an unlimited number of simultaneous readers, but it will only allow one writer at any instant in time. For many situations, this is not a problem. Writers queue up. Each application does its database work quickly and moves on, and no lock lasts for more than a few dozen milliseconds." --- https://www.sqlite.org/whentouse.html

I have to wonder, with the others here, whether the bug is in the driver, or just anywhere but SQLite. With such fragility it would have blown up by now. SQLite is ubiquitous: the iPhone, iTunes, Android, Chrome, Windows 10, McAfee, the Redhat Package Manager (RPM), commercial flight software, and many other pieces of software --- https://www.sqlite.org/famous.html

Re:At 22 years old, Postgres might just be the most advanced... by justinclift on Wednesday December 05, 2018 @04:54PM (#18613010)
As a data point with SQLite, there's a branch called the "Server Process Edition" which has support for 16 concurrent writers:

https://www.sqlite.org/src/tree?ci=754ad35cd26da361

https://www.sqlite.org/src/artifact/b98409c486d6f028

Haven't yet played around with it myself, and I tend to expect it'd be more experimental than something to use in production.

Re:At 22 years old, Postgres might just be the most advanced... by EGreg on Wednesday December 05, 2018 @02:35PM (#18611672)
Can you be more specific? I always wondered why people use them
Re:At 22 years old, Postgres might just be the most advanced... by manigandham on Wednesday December 05, 2018 @02:51PM (#18611805)
SQL Server has columnstores, in-memory tables with compiled procedures (which can be combined with columnstores), graph processing, built-in machine learning extensions, AlwaysOn availability/replication groups, Stretch and Polybase to read from external sources, integrated encryption, fantastic tooling, and the best optimizer with batch/vectorized processing.

It still doesn't have proper upsert or JSON columns, so PostgreSQL wins on small usability features, but SQL Server is a serious workhorse when it comes to large scale operational and analytical systems with unmatched performance. PG barely has parallel queries or decent connection management and is a decade away from catching up to SQL server in these features.

Re:At 22 years old, Postgres might just be the most advanced... by BrentOzar on Wednesday December 05, 2018 @03:11PM (#18611981)
> It still doesn't have proper upsert or JSON columns

Upsert (MERGE) was added in SQL Server 2008 (although it's chock full of known bugs): https://www.mssqltips.com/sqlservertip/3074/use-caution-with...

JSON was added in SQL Server 2016, although indexing it is still tricky: https://docs.microsoft.com/en-us/sql/relational-databases/js...

Re:At 22 years old, Postgres might just be the most advanced... by manigandham on Wednesday December 05, 2018 @03:13PM (#18611998)
That's what I meant by "proper". It can do it, but PG does it better. It's one area that I wish they would invest time into considering the large developer impact it would have.
Re:At 22 years old, Postgres might just be the most advanced... by SigmundA on Wednesday December 05, 2018 @05:54PM (#18613570)
My understanding is they got burned by all the work put into the XML datatype and don't want to repeat it with JSON, hence the more conservative approach.
Re:At 22 years old, Postgres might just be the most advanced... by fanf2 on Wednesday December 05, 2018 @05:18PM (#18613243)
How do Stretch and Polybase compare to Foreign Data Wrappers?

Is the integrated encryption comparable to pgcrypto?

Re:At 22 years old, Postgres might just be the most advanced... by manigandham on Wednesday December 05, 2018 @05:55PM (#18613577)
FDWs are much more flexible, as they are basically an endpoint based on the SQL/MED standard that can be implemented by any module and there are dozens of options out there for connections.

Stretch and Polybase are much more limited to Azure storage and Hadoop systems with deep integration into T-SQL, and scale-out using clustering.

Re:At 22 years old, Postgres might just be the most advanced... by krylon on Wednesday December 05, 2018 @03:06PM (#18611932)
At work, we run SQL Server, because we use several third-party applications that use it. And once it's there and paid for, if we need another database, it is easier to just dump it on the server that is already there, rather than run a second RDBMS server.

I am a little torn about this - on the one hand it strikes me as laziness that all these Windows developers will just pick SQL Server because it is so convenient (good integration with Visual Studio, drivers are present by default in .Net Framework). On the other hand, in terms of performance and reliability, I cannot say a single bad word about SQL Server. Another thing I like is that one can use accounts from an Active Directory domain as database accounts, so you do not have to duplicate group structures and such, if you use elaborate database permissions.

I have come to like T-SQL, which integrates some procedural features into SQL, so SQL Server does not need a separate language for writing stored procedures, triggers, and so forth. Where T-SQL is not enough, one can use basically any .Net language, but there are some restrictions to it, and I have never used this feature.

Postgres offers many features SQL Server lacks, e.g. the array of data types available in Postgres is amazing, or the fact that it has enums. SQL Server has some features Postgres lacks, for example the Service Broker, which is very interesting, but apparently non-trivial to use correctly.

Re:At 22 years old, Postgres might just be the most advanced... by gaius on Wednesday December 05, 2018 @04:27PM (#18612759)
Where T-SQL is not enough, one can use basically any .Net language, but there are some restrictions to it, and I have never used this feature.

Embedded R in SQL Server as well.

Postgres is my default database choice, or SQLite if it's small, but SQL Server is a solid, high performance product and I'll gladly run it for the most demanding workloads.

Re:At 22 years old, Postgres might just be the most advanced... by kev009 on Wednesday December 05, 2018 @02:42PM (#18611722)
I'm not advocating their use and the default should be right and smallest fit for the job, which means sqlite or Postgres nearly always. But too often people get in this mindset that old commercial products are meritless or whatever.

An example for Oracle would be RAC. This is a mature shared data architecture to provide high availability and horizontal server scaling to a DB. Deploying this is a fairly standard Oracle DBA activity. Getting something similar with open source tools is a big lift and permanent support commitment and will be for a while. Meanwhile these three commercial DBs have provided continuity for decades for features like this.

Re:At 22 years old, Postgres might just be the most advanced... by devy on Wednesday December 05, 2018 @02:47PM (#18611761)
> But too often people get in this mindset that old commercial products are meritless or whatever.

There is a lot of that assumption. And then, there is the vendor lock-in licensing terms that commercial DB vendors utilize, in that database switching cost is just prohibitively high and impractical.

Re:At 22 years old, Postgres might just be the most advanced... by manigandham on Wednesday December 05, 2018 @02:55PM (#18611829)
The lock-in is not from licensing, it's from the features and performance. If you aren't using those capabilities then you didn't need that product anyway, and can use any number of migration tools to switch to a different database.
Re:At 22 years old, Postgres might just be the most advanced... by devy on Tuesday December 11, 2018 @03:39PM (#18658636)
Yes, most of factors for lock-in are from features and ecosystem and replacement cost. But in licensing negotiations, license up-sale, auxiliary products & services are definitely making the lock-in tighter - since you've just made the big investments, why switching?
Re:At 22 years old, Postgres might just be the most advanced... by TylerE on Wednesday December 05, 2018 @02:42PM (#18611726)
Scaling/replication is the usual thing. Postgres certainly has a better story here than it used to, but if you want to do serious scale replication it's very much in roll-your-own territory.

Like, imagine you have a few hundred terabytes of data you need to keep in-sync globally, between multiple datacenters, with many servers in each DC.

Re:At 22 years old, Postgres might just be the most advanced... by deepsun on Wednesday December 05, 2018 @02:53PM (#18611816)
I was under impression that only Spanner-like databases (CockroachDB and FaunaDB) are the only ones that can do cross-datacenter multi-row transactions at scale. Because they were designed exactly for this, otherwise tradeoffs taken are just too bad for the that use-case (e.g. too slow or too brittle in availability).

By cross-datacenter I mean datacenters far away from each other, in different geo regions.

Re:At 22 years old, Postgres might just be the most advanced... by madhadron on Wednesday December 05, 2018 @03:20PM (#18612111)
You can do single master, many read replicas, which works fine for many, many use cases.
Re:At 22 years old, Postgres might just be the most advanced... by akerro on Wednesday December 05, 2018 @02:36PM (#18611683)
Oracle SQL Server is like 50 years old
Re:At 22 years old, Postgres might just be the most advanced... by pkaye on Wednesday December 05, 2018 @02:40PM (#18611710)
I thought Oracle was founded in 1977 so it would be closer to 40 years.
Re:At 22 years old, Postgres might just be the most advanced... by kevindqc on Wednesday December 05, 2018 @03:13PM (#18612007)
Correct! First available release was in 1979 according to the wiki, so almost 40 years.

https://en.wikipedia.org/wiki/Oracle_Database#Releases_and_v...