News for hackers. Stuff that matters.
Re:SQL is 43 years old – Here’s why we still use it today by fnord123 on Tuesday May 02, 2017 @09:00AM (#14245962)
>The second annoyance I have is the push for schemaless. Schemaless does not exist. There is always a schema, except in a schemaless data store, the schema has been moved to app code and then my earlier comment applies.

This is a great point. It's like types for programming languages. They always exist; it's just a matter of whether you have the compiler managing it or whether you need to keep it in your head when you're hacking.

> I think the lack of understanding of SQL and bad ORM experiences (Hibernate WTF) is what led people to think SQL/RDMBs were the problem when in fact they were not.

Again, like types in programming languages, I think the ability to iterate quickly without having to predict how the data will be used in the future leads to 'schemaless' approaches which can be kicked out the door more quickly (maybe) than ponderous schema laden tables.

Re:SQL is 43 years old – Here’s why we still use it today by kbenson on Tuesday May 02, 2017 @10:23AM (#14246555)
> It's like types for programming languages. They always exist; it's just a matter of whether you have the compiler managing it or whether you need to keep it in your head when you're hacking.

While true to a degree, it's worth noting that some languages also reduce the number of types considerably at the same time, and maybe even provide some other conveniences when dealing with types (e.g. automatic interpretation as numeric when used in a numeric operation).

You still have to keep types in your head, but the reduction of types generally makes this non-problematic. Although, as your code grows and you define your own types (either formally through an included system or as ad-hoc data structures), you start needing to keep more types in your head, and this starts eating away at any benefit you had from not having to define them in the first place.

Re:SQL is 43 years old – Here’s why we still use it today by dllthomas on Tuesday May 02, 2017 @11:04AM (#14246922)
You and the parent are not using quite the same definition of "types", I think.
Re:SQL is 43 years old – Here’s why we still use it today by kbenson on Tuesday May 02, 2017 @01:14PM (#14248306)
I think we are, if you look at it from a high level. C has some well known types most people are familiar with. Perl has a few types too, scalar, array and hash (and a few more lesser known or used ones). This makes Perl very easy to use initially, because you can just treat strings as string and numbers as numbers, and Perl will mostly do what you want based on the operators involved. You create complex data structures by just creating them (ad-hoc) or by actually defining a class, which actually gives you a new type.

As you create any non-trivial program, you'll likely create some types, whether in C or Perl (as classes). What was initially a benefit in Perl, since you didn't have many typed to keep track of, becomes a liability, as there are more types but still no formal way to ensure they are being used in the correct locations and as expected arguments without runtime checking (and manual checking at that, unless you use a good module to handle a lot of that for you).

That some languages keep types fairly low level (C, with typed relating fairly closely to machine representation) and some are high level (Perl, with each type being a higher level container) is really irrelevant. In the end, they are just labels that codify behavior (size, acceptable methods of use).

In SQL and NoSQL systems, you have the same trade-offs. Do I define everything up-front so I know (presumably) when there's a problem, or do I create the structure ad-hoc as needed and enforce the structure through the function of the application that creates the data?

Re:SQL is 43 years old – Here’s why we still use it today by fnord123 on Tuesday May 02, 2017 @01:50PM (#14248715)
Yes, we're talking about the same thing. But in Perl, do we make classes or hashes with keys as the field names? :)
Re:SQL is 43 years old – Here’s why we still use it today by kbenson on Tuesday May 02, 2017 @02:04PM (#14248880)
Such is the eternal question in Perl. :)

Also, sometimes we make classes out of scalars, arrays and globs as well. It was interesting the first time I looked into the source of IO::Handle (and IO).

Hmm, I bet there's a module to create classes that uses pack and unpack to store byte level representations in a scalar as the blessed type of a class.

Re:SQL is 43 years old – Here’s why we still use it today by dllthomas on Tuesday May 02, 2017 @07:31PM (#14251672)
Heh, then I think I read you as saying something more interesting than you'd intended :-P
Re:SQL is 43 years old – Here’s why we still use it today by fnord123 on Wednesday May 03, 2017 @08:49AM (#14255182)
What did you think I was talking about?

I guess there could be a few senses of types for a programming practitioner:

1. integral types (int, float, etc) 2. product types (structs, objects) 4. sum types (enums) 5. containers (vec, dict, list, etc).

Typeless types would be dicts of lists of dicts. i.e. shitty product types. e.g. `x = dict(y=1, dict(z=[1,2,dict(u='good data')]))`. Passing this to functions that want to get 'good data' out of the structure basically end up treating this as a type, if that makes sense.

If I understand kbenson's point, it's that sometimes this is better than a tower of babel of type hierarchies that you may have confronted in Java or C++ and you end up needing to ponder how to convert your IWiggleWoggleSupplierProvider into an IWoggleWiggleProviderSupplier even though they do almost the same thing and have the same data but somehow they have different inheritance hierarchies.

Re:SQL is 43 years old – Here’s why we still use it today by dllthomas on Wednesday May 03, 2017 @02:22PM (#14258593)
There's a sense of types of "anything (computable?) you can statically say about your terms". Representation is just a part of this, and often not a very important one. "How can I use this?" is usually much more interesting.

From that perspective, "treating the shape of your dicts as its type" certainly makes sense - it determines how you can use it correctly. It's incidental that you're not encoding this information in the "type system" your language provides (and that may or may not be a good idea, depending - there are always tradeoffs and we've left an awful lot of context unspecified).

From the same perspective, if you can use your IWWSProvider and your IWWSSupplier identically, we can say that they're arguably not different types and only being treated so because someone has given the Java or C++ "type system" an incorrect specification.

Re:SQL is 43 years old – Here’s why we still use it today by mindcrime on Tuesday May 02, 2017 @10:17AM (#14246496)
I think the lack of understanding of SQL and bad ORM experiences (Hibernate WTF)

I don't understand this comment. I've never had a bad experience with Hibernate. If anything, Hibernate has saved me tons of time over the years.

Re:SQL is 43 years old – Here’s why we still use it today by hibikir on Tuesday May 02, 2017 @11:47AM (#14247345)
I, on the other hand, have spent an inordinate amount of time, including waking up in the middle of the night for incident response, due to Hiberanate misuse, which is so easy that I'd not touch Hibernate with a ten foot pole.

There is one case where Hibernate can save you a significant amount of time vs plan SQL, and that's arbitrary update statements. But the moment there are two joins in there, predicting hibernate performance is not necessarily easy, and differentiating a query that will run well from one that will kill your system at scale is far harder than in plan SQL. In practice, the only way to use Hibernate responsibly is to examine the likely queries that it will generate.

My least favorite example was a situation where manual SQL gave me a very fast query that took a couple of milliseconds to run on a large database, but Hibernate's answer was not even just one query: It was an accidentally quadratic monstrosity, when the original programmer wanted was one row: Hibernate just wasn't smart enough to figure out which row it wanted in the DB, so it grabbed half the database. After I saw the beautiful queries that Hibernate had created, I had to give the finger to the existing data layers and do my own thing, which I am told remains to this day.

The way Hibernate is built only makes sense if it can hide the database behind it. In reality, it doesn't, so I not only have to make sure everyone knows SQL well, but they also have to learn hql and hibernate's most common pitfalls. All to save a little bit of time on the rare cases where I might have very dynamic queries that somehow hibernate won't screw up.

Java EE was a pretty unfortunate thing, but under duress, I'd much rather use EJB3 and straight SQL over the Spring + Hibernate stack that many of us had to deal with between 2000 and 2010.

Re:SQL is 43 years old – Here’s why we still use it today by matwood on Tuesday May 02, 2017 @10:58AM (#14246857)
I'd be curious about the size of the systems you have used with Hibernate. Do you know off the top of your head what will cause Hibernate to issue a flush? How about the order of operations in a batch? I've read the entire Hibernate book multiple times while trying to squeeze out speed, and finally decided enough.

The problem with Hibernate is I have to learn all of these Hibernate quirks to maybe get decent speed out of a system, when I could have just written SQL and been done. This isn't even including debugging when Hibernate decides to issue N+1s.

Re:SQL is 43 years old – Here’s why we still use it today by mindcrime on Tuesday May 02, 2017 @12:46PM (#14248014)
I'd be curious about the size of the systems you have used with Hibernate.

I wouldn't even know what metric to use to quantify that. Number of tables? On the order of hundreds, probably low hundreds... definitely not thousands.

It may just be that I've been lucky in that the data models I've worked with have been things that map well to using Hibernate. I will say that I haven't worked with a lot of data models where you need more than 3 or 4 joins in any given query. I know some people write far more complex queries than that, and maybe those are the kinds of scenarios that Hibernate doesn't handle well?

The problem with Hibernate is I have to learn all of these Hibernate quirks to maybe get decent speed out of a system, when I could have just written SQL and been done

Do you hand-roll your own caching mechanism when just writing plain SQL?

Re:SQL is 43 years old – Here’s why we still use it today by matwood on Tuesday May 02, 2017 @01:18PM (#14248348)
> Do you hand-roll your own caching mechanism when just writing plain SQL?

Using something like jOOQ has meant I rarely need any caching, unlike Hibernate where caching almost seems required.

Re:SQL is 43 years old – Here’s why we still use it today by mindcrime on Tuesday May 02, 2017 @01:22PM (#14248399)
Interesting. I've never used jOOQ, but I'll give it a look. Hibernate has worked well for me, but I'm not dogmatic about it. Always interested to try new tools.
Re:SQL is 43 years old – Here’s why we still use it today by matwood on Tuesday May 02, 2017 @01:39PM (#14248600)
I've been writing SQL for almost 20 years, so the pain for me is the grunt mapping of data to and from. jOOQ does much of the grunt work of mapping, and lets me write typed SQL that it smartly translates to queries. More advanced features like various types of fetch make my job even easier.

https://www.jooq.org/doc/3.9/manual-single-page/#arrays-maps...

Re:SQL is 43 years old – Here’s why we still use it today by cygned on Tuesday May 02, 2017 @11:26AM (#14247134)
I had to use Hibernate in a project. We were loading an entity from the database, someone invoked a setter and the set value was updated in the database automatically. Totally blew my mind.

Never touched it again since then (not doing much Java anyway).

Re:SQL is 43 years old – Here’s why we still use it today by cakes on Tuesday May 02, 2017 @11:42AM (#14247287)
This mirrors the experience I have as well. I (currently) don't know what will cause Hibernate to flush but I certainly do know that I had to know this exactly and had to do some really painful debugging at points. The worst one I was on was just early prototyping of an app that used Hibernate and we had so many problems with performance and Hibernate(isms) that the whole thing seemed destined to fail even though our use case was exactly what ORM/Hibernate are for...yet we still have to control when changed flushed because otherwise things would get wonky really quickly.
Re:SQL is 43 years old – Here’s why we still use it today by dunham on Tuesday May 02, 2017 @01:02PM (#14248173)
Yeah, profiling helps a little. It taught me that queries flush (and you can easily go n^2 from that if you're not careful).

More recently, we've had n+1 issues (putting a printing breakpoint in EntityType.loadByUniqueKey helped me track down where). My hack to get around that was to do a bulk Criteria query up front to pre-load the session cache with the entities that were being fetched one at a time.

There were a lot of quirks that I've had to learn over the last decade, and I'm still discovering more.

Re:SQL is 43 years old – Here’s why we still use it today by gerbilly on Tuesday May 02, 2017 @11:10AM (#14246974)
Speed? I found Hibernate is great for CRUD.

Like where you have dozens and dozens of domain objects that you want to persist, and you don't want to spend all your time writing DAO code with SQL in in that is all the same but different.

Enterprise applications are rife with those.

For speed write some hand written SQL to handle those few cases (like batch imports maybe).

Also I found a handy rule:

Only create an association between entities when the entity makes no sense without the other.

For all other cases split them up and perhaps create aggregates by hand.

Re:SQL is 43 years old – Here’s why we still use it today by mindcrime on Tuesday May 02, 2017 @12:47PM (#14248029)
That jibes with my experience. We don't have tons of complicated associations, and do a lot of fairly straightforward CRUD operations. Maybe that's just the sweet spot for using Hibernate.
Re:SQL is 43 years old – Here’s why we still use it today by singham on Tuesday May 02, 2017 @11:45AM (#14247318)
I have used Nhibernate in my project at a company. But beyond simple queries we hit a performance issue. For a complex query, it took about 10 seconds to load the data to the frontend. So we had to rewrite the "code logic" ( the Nhibernate part ) into an "custom sql query" (Nhibernate allows you to write custom queries ) and then write custom mappers to map the results back to our application. The reason is Nhibernate does its processing objectwise and inmemory.
Re:SQL is 43 years old – Here’s why we still use it today by rwj on Tuesday May 02, 2017 @10:21AM (#14246543)
This should be emphasised more. Schemaless is good for productivity during prototyping (like dynamically typed languages), but long term a well-defined schema will help maintenance (like statically typed languages).
Re:SQL is 43 years old – Here’s why we still use it today by exclusiv on Tuesday May 02, 2017 @02:09PM (#14248936)
Sometimes it really is schemaless, although usually only schemaless for certain fields. Imagine a case where users can create custom fields or store arbitrary data in some fields.