News for hackers. Stuff that matters.
Re:I'm All-In on Server-Side SQLite (2022) by erulabs on Friday September 22, 2023 @01:00PM (#37614511)
I hope fly is able to make it. I’m rooting for them - however - I’m starting to wonder if the SQLite push isn’t more “this is fun and interesting to build” and less “customers want this”.

Don’t get me wrong - this is neat - but I’d never suggest anyone to actually use this outside of a fun experiment. The problem with existing SQL dbs isn’t really the architecture - its the awful queries that do in memory sorting or make temporary tables for no reason or read-after-write, etc, not network latency. SQLite won’t fix your current production problems.

If it turns out they’re building this for customers throwing cash at them, awesome. I just somehow doubt it. I think Planetscale has the better approach: a drop in replacement for MySQL/RDS with a smarter query planner. As a production engineer that’s what I want to pay for!

Re:I'm All-In on Server-Side SQLite (2022) by jjtheblunt on Friday September 22, 2023 @01:14PM (#37614702)
I think you're skipping the replicated read only use case, which is our use case, and it's super handy there. but i understand this is a restricted scenario where little coudl really go wrong, and it could be done other ways.
Re:I'm All-In on Server-Side SQLite (2022) by sodapopcan on Friday September 22, 2023 @01:15PM (#37614720)
Lots of smaller businesses could do fine with this if they don't have a write-heavy workload. Like an ecomm shop, for instance.
Re:I'm All-In on Server-Side SQLite (2022) by endisneigh on Friday September 22, 2023 @01:38PM (#37615063)
Any self respecting e-commerce site would want fault tolerance and strong consistency even with potential network partitions, so definitely not SQLite as described in article
Re:I'm All-In on Server-Side SQLite (2022) by sodapopcan on Friday September 22, 2023 @01:46PM (#37615179)
They aren't necessarily going to have all this. Lots of smaller ecom shops can run on a single server per region. If you're a North American company selling a few hundred t-shirts per day in NA and EU, it could probably be fine, no? I'll admit I'm not speaking from experience. Rather, I have experience in everything I just said only I was using pg, not sqlite. But I've been very interested in sqlite recently.
Re:I'm All-In on Server-Side SQLite (2022) by endisneigh on Friday September 22, 2023 @01:59PM (#37615364)
It’s not about the size. Issues result in lost sales which means lost revenue. You would want fault tolerance regardless of how big you are.
Re:I'm All-In on Server-Side SQLite (2022) by bootsmann on Friday September 22, 2023 @02:19PM (#37615660)
There is a cost to fault tolerance as well, if you lose $100 per month due to fault tolerance thats worth about 30 developer minutes. Do you really get good fault tolerance for 30m of monthly work?
Re:I'm All-In on Server-Side SQLite (2022) by erulabs on Friday September 22, 2023 @03:12PM (#37616366)
You do with RDS and/or other hosted database services, hence their popularity.
Re:I'm All-In on Server-Side SQLite (2022) by sodapopcan on Friday September 22, 2023 @02:59PM (#37616206)
That's something for me to look into. Like what would it would look like in each? I've never been the dba-type in past jobs. I'm relatively well-versed in SQL but not administration.
Re:I'm All-In on Server-Side SQLite (2022) by emodendroket on Friday September 22, 2023 @01:52PM (#37615264)
Why even build your own e-commerce Web site at all in that case? It’s undifferentiated work.
Re:I'm All-In on Server-Side SQLite (2022) by bcrosby95 on Friday September 22, 2023 @02:27PM (#37615764)
I don't know about nowadays.

8 years ago we built our own ecommerce site + warehouse app (inventory tracking, fulfillment, receiving) for a few hundred orders per day. The goal was to be able to better see profit margins by product, track where the money was going/coming from in detail, along with cleaning up the inventory management part of the operation.

The warehouse people loved the change because it really streamlined the fulfillment process and it basically reduced their errors to zero. Owners loved it because of the all the cost/price tracking. It took about 6 months of work by 2 devs.

Before this they were on woocommerce. It was slow as shit. We looked at shopify but the integrations with 3rd party warehouse stuff was bad. The other solutions we looked at were overly complicated swiss army knives.

Re:I'm All-In on Server-Side SQLite (2022) by sodapopcan on Friday September 22, 2023 @02:41PM (#37615962)
This is very close to my experience as well. Unfortunately, it was in reverse for me :( We had a custom solution and people were happy. A new tech lead came in and didn't like that the custom solution was PHP that still had some legacy spaghetti in it, so we switched to an off the shelf solution. It was very painful. People were unhappy.
Re:I'm All-In on Server-Side SQLite (2022) by pmarreck on Friday September 22, 2023 @09:59PM (#37619984)
> that still had some legacy spaghetti in it

I'm going to take a wild ass guess here that you are wildly understating the "bit of legacy spaghetti"

Re:I'm All-In on Server-Side SQLite (2022) by sodapopcan on Saturday September 23, 2023 @10:36AM (#37623613)
Your wild guess is flat out wrong. When the previous tech lead came in the entire thing was spaghetti. His strategy was to slowly start bringing in some order to the chaos which we did, diligently, over two years making tons of progress. It was a homebrewed framework but it was really nice (he'd had years of experience contributing to Drupal). All the business critical functionality had been converted to this. The spaghetti parts were features that were essentially "complete". Since they never needed updating, they worked fine as they were.
Re:I'm All-In on Server-Side SQLite (2022) by pmarreck on Saturday September 23, 2023 @09:15PM (#37629250)
Drupal is the one piece of technology that I did 1 project in and would never work with again no matter what the pay was.
Re:I'm All-In on Server-Side SQLite (2022) by sodapopcan on Saturday September 23, 2023 @11:17PM (#37629859)
I wouldn’t know I’ve never used it.
Re:I'm All-In on Server-Side SQLite (2022) by evantbyrne on Friday September 22, 2023 @02:15PM (#37615593)
As with anything, the typical business can use something off-the-shelf, but a certain percentage are doing things differently enough that custom development becomes practical. I had to custom build the billing system for Beaker Studio to properly meter customers. Even Stripe's metering API wasn't flexible enough to handle per-hour metering.
Re:I'm All-In on Server-Side SQLite (2022) by emodendroket on Friday September 22, 2023 @02:17PM (#37615625)
I find it hard to imagine a reason a shop selling 100 tee shirts in a day would need any custom functionality since this is basically the exact use case all these OOB e-commerce tools are built for.
Re:I'm All-In on Server-Side SQLite (2022) by sodapopcan on Friday September 22, 2023 @02:34PM (#37615872)
I've worked for such a place. Off-the-shelf solutions are trying to be everything to everyone and you can end up customizing the crap out of them to the point where it can become more onerous than just building your own. The place I worked was also print-on-demand service and had hundreds of thousands of SKUs as well as allowed customers to make custom products and we also hosted some peoples' shops. Shoehorning that into a custom solution was painful.

I work at a very similar place now that uses Shopify. Managing that many SKUs on Shopify is crazy painful.

The thing is is that custom ecom solutions really aren't that hard. The off-the-shelf ones are complex because, as stated above, they are trying to be everything to everybody.

Re:I'm All-In on Server-Side SQLite (2022) by evantbyrne on Friday September 22, 2023 @02:20PM (#37615668)
Yeah, you're probably right about that. Main thing that would likely be custom would be the design.
Re:I'm All-In on Server-Side SQLite (2022) by sarchertech on Friday September 22, 2023 @03:02PM (#37616250)
One of the first commercial projects I worked on nearly 20 years ago was a tshirt shop. And the precious company I worked for was a comparatively huge logistics startup.

I’m fairly confident I could spend a month or so writing a custom solution for a shop selling and shipping a few hundred tshirts a day that would save them enough money to break even on the software in a few years (compared to off the shelf solutions).

If I was starting my own tshirt company, I’d definitely do it.

Re:I'm All-In on Server-Side SQLite (2022) by MyneOutside on Friday September 22, 2023 @03:44PM (#37616773)
In our B2B space there are tons of custom business rules. For example, you place orders per manufacturer and each manufacturer has its own minimum and reorder amounts and reqs on an order, promotions, business rules, etc. This is for business that have been around 25-30 years.

We did an evaluation on several out of the box ecommerce solutions and none of them were able to meet the requirements that absolutely had to be there. Shopify flat out said no, they can't help us, etc.

Re:I'm All-In on Server-Side SQLite (2022) by parineum on Friday September 22, 2023 @02:37PM (#37615906)
> the typical business can use something off-the-shelf

I think we generally have this problem. It's not the typical business that can use that stuff, it's the average business. Unfortunately no business is the average business.

Re:I'm All-In on Server-Side SQLite (2022) by pmarreck on Friday September 22, 2023 @09:57PM (#37619974)
https://rqlite.io/ at least looks like an interesting option, here
Re:I'm All-In on Server-Side SQLite (2022) by geysersam on Saturday September 23, 2023 @01:11AM (#37620722)
Why would they need strong consistency? What's shown on the page the user sees is never strongly consistent with the db anyway.
Re:I'm All-In on Server-Side SQLite (2022) by unshavedyak on Saturday September 23, 2023 @02:21PM (#37625886)
I have this same discussion at work frequently. Everything thinks they need instantly up to date, yet our customers work for 10 minutes on a stale document and no one cares.
Re:I'm All-In on Server-Side SQLite (2022) by lib-dev on Friday September 22, 2023 @01:30PM (#37614940)
Yeah it seems to make a lot of sense in ecomm. Product search and filtering on tables in the 1000s rather than the millions.
Re:I'm All-In on Server-Side SQLite (2022) by hinkley on Friday September 22, 2023 @03:36PM (#37616677)
Lots of small businesses invent write heavy workloads and don't realize how many thousands of dollars they are spending a month on being nosey.
Re:I'm All-In on Server-Side SQLite (2022) by nitwit005 on Friday September 22, 2023 @04:38PM (#37617446)
If you're a smaller e-commerce business, your whole site can probably be cached aside from auth, checkout, and order history.
Re:I'm All-In on Server-Side SQLite (2022) by sodapopcan on Friday September 22, 2023 @06:16PM (#37618475)
I can say with authority that this is true! We used to store our whole massive catalog in Varnish.
Re:I'm All-In on Server-Side SQLite (2022) by Xeoncross on Friday September 22, 2023 @01:44PM (#37615154)
Replace SQLite with Excel and read it again.
Re:I'm All-In on Server-Side SQLite (2022) by Varriount on Friday September 22, 2023 @01:55PM (#37615326)
> It's the awful queries that do in memory sorting or make temporary tables for no reason or read-after-write, etc.

Are there any viable alternatives though? I often wonder what an SQL-like language built from the ground up would look like.

Re:I'm All-In on Server-Side SQLite (2022) by erulabs on Friday September 22, 2023 @03:08PM (#37616318)
The alternative is to write good queries! Alternatively, a better query planner that has a more liberal approach to "I know what you want, not what you're asking for". A great article on this re: Planetscale's approach is at https://vitess.io/blog/2021-11-02-why-write-new-planner/
Re:I'm All-In on Server-Side SQLite (2022) by mattgreenrocks on Friday September 22, 2023 @02:15PM (#37615595)
I’m excited about SQLite for web apps if only because it is one less moving piece in my stack, which focuses on prototyping and finding product market fit.

If I start hitting hundreds of writes per sec, then, thats either awesome or I wrote some horrible code.

Re:I'm All-In on Server-Side SQLite (2022) by erulabs on Friday September 22, 2023 @03:05PM (#37616289)
This is exactly where I think SQLite shines! Unfortunately, I don't get to do much green-fielding in my career.
Re:I'm All-In on Server-Side SQLite (2022) by teaearlgraycold on Friday September 22, 2023 @02:41PM (#37615954)
If you want a read only database that’s too large to simply be a JSON file hosted on a CDN it makes sense. That’s kind of niche but not unheard of.
Re:I'm All-In on Server-Side SQLite (2022) by laurencerowe on Friday September 22, 2023 @02:53PM (#37616136)
> The problem with existing SQL dbs isn’t really the architecture - its the awful queries that do in memory sorting or make temporary tables for no reason or read-after-write, etc, not network latency. SQLite won’t fix your current production problems.

In my experience SQL databases are pretty poor at executing the kinds of deeply nested joins needed to return all of the data needed to render more complex UIs. It almost always ends up being faster to simply make nested selects in batches. So latency ends up mattering in these cases.

You can work around this by denormalizing the data, but this often explodes your data size.

Re:I'm All-In on Server-Side SQLite (2022) by ngrilly on Friday September 22, 2023 @03:01PM (#37616232)
Which is exactly where SQLite shines, but I guess that was your point: https://www.sqlite.org/np1queryprob.html
Re:I'm All-In on Server-Side SQLite (2022) by WuxiFingerHold on Friday September 22, 2023 @03:13PM (#37616380)
I can't imagine many suitable production use cases too. Alternatives like Planetscale, Supabase or Neon are even easier to use and at the same time much more powerful. If latencies around 50 ms are too much for the specific use case, SQLite with Litestream could be a great solution. Otherwise I'd go with managed Postgres/MySQL solutions.
Re:I'm All-In on Server-Side SQLite (2022) by oooyay on Friday September 22, 2023 @03:44PM (#37616775)
I recently implemented SQLite for a metadata application at work. My constraints in design were pretty concise; the application will only ever need to scale vertically, my data persistence story is a matter of speed and accessibility over longevity, I have plenty of options for scaling disk IO, and my read performance is much more paramount than write performance.

The outcome is that my persistence later is not adding $200/m immediate service overhead cost and my deployment was easy to manage, which is a strong promise made to the rest of the team. I think there's a place for SQLite, but just like any tool you need to know that your design constraints match the constraints of the tool.

Re:I'm All-In on Server-Side SQLite (2022) by eek2121 on Friday September 22, 2023 @04:55PM (#37617652)
The first iteration of my website used sqlite. I only switched away because the growing audience made me nervous.