News for hackers. Stuff that matters.
Re:SQLite as an Application File Format (2014) by lbutler on Saturday June 13, 2020 @12:22PM (#23510511)
I have an application that uses sqlite as their file format but they've cleared the header so you can't open it directly.

I'd love to be able to make secondary applications like you've described but being enterprise software they don't want to make it too easy.

They obviously want to keep people locked in with their $40k per seat application!

I guess the first step is figuring out the page size and other bits the other meta data you set in the header [1].

I know I just have to sit down and understand the format better and I will eventually figure it out...

[1] https://www.sqlite.org/fileformat.html

Re:SQLite as an Application File Format (2014) by TheRealPomax on Saturday June 13, 2020 @12:24PM (#23510528)
That's mean, but you'll just have to live with that. If someone takes active steps towards preventing people from making secondary apps/utils, then they're (a) jerks, and (b) allowed to do that.

Might be worth actually contacting them to ask why, if you can make the case that secondary applications will increase the value of their app, not decrease it.

Re:SQLite as an Application File Format (2014) by hamburglar on Saturday June 13, 2020 @05:41PM (#23513009)
I'm surprised that nobody's pointed out that there are actually valid reasons other than greed to obscure your file format. It's an implementation detail, not a contract. If customers begin relying on the implementation details, you end up with angry customers when you change the implementation details. A SQLite db without the header is basically a statement saying, "we are using the obvious file format here for our convenience, not for general purpose access. Screw around in here at your own risk."

If you modified their app's internal state db and screwed it up because they have designed their software with certain assumptions that aren't clear from just reading their db schema, that would be a nightmare for them to support. The easiest thing for them to do is just to try to discourage tampering with their internal state.

This is especially true if there's a chance that a market for secondary apps/utils will spring up. If that's to happen and be viable, they absolutely would want to put thought into what their supported interfaces are for those apps/utils, otherwise they will end up painted into a corner and unable to change their architecture without destroying a marketplace.

Re:SQLite as an Application File Format (2014) by Eremotherium on Sunday June 14, 2020 @12:06PM (#23518771)
I really don't get this line of reasoning. If I do something with a product that is (maybe even explicitly) unsupported by the manufacturer I don't have a reasonable leg to stand on. We've recently had this with a customer where they used the concrete calls of our app to automate some of their stuff (we had had no public API for that action at that point because we hadn't had the need so far and so did no other customer or to be precise: no customer was prepared to shell out the cash for us to develop it) and after we changed something they suddenly weren't able to script the creation of customer entries in their installation. We had told them at least two years ago that the way they interacted with our system was but supported (we only noticed them automating stuff back then because it was throwing exceptions in our backend) and while we were nice enough to fix it this time because the fix was trivial we recommended that they should switch to our supported API. Two weeks ago theyir stuff broke again and we told them to use the API or fuck off.
Re:SQLite as an Application File Format (2014) by hamburglar on Sunday June 14, 2020 @04:52PM (#23521263)
So you see the problem and you've wasted time on it, and now you've come to the point where you're ok with being put in a situation where you have to tell your users to fuck off.

Just because they don't have a leg to stand on doesn't mean this isn't potentially a huge pain in your ass that you'd probably rather avoid. Imagine if you had a LOT of customers contacting you with this type of problem to the point where you felt like you were painted into a corner and had to support some ad-hoc APIs that weren't designed to be customer-facing and which you might have been planning to remove altogether because they're part of a design that's changing.

This is exactly the situation the obfuscation is attempting to avoid. They're just doing it with a technical solution rather than a human telling another human "don't do that."

Re:SQLite as an Application File Format (2014) by EvanAnderson on Saturday June 13, 2020 @07:50PM (#23513874)
I've never understood the "We can ship this but it's not contract" mentality. To my mind if you ship something it's contract. I think developers keeping that in mind would make the world full of more resilient code. You're making a contract w/ your future self, if nothing else.

To the point re: modifying internal state and screwing-up the application - If you're writing anything out to persistent storage you should assume that it's untrusted data when you read it back in. If for no other reason than physics itself is a malicious (or, at best, ambivalent) actor.

Re:SQLite as an Application File Format (2014) by hamburglar on Sunday June 14, 2020 @05:05PM (#23521370)
If your app ships with a dynamically loaded library and I dig through the exposed functions and find something undocumented that looks useful and I figure out how to use it, that's a contract to you? That's utterly insane.

Re: the concept of untrusted data, this is off in the weeds argument for argument's sake IMO. Do reasonable validations of the state data, sure, but picking nits about the nature of trust and internal application state is an infinite hole I'm not jumping into with you.

Re:SQLite as an Application File Format (2014) by catalogia on Saturday June 13, 2020 @11:44PM (#23514918)
Sometimes a programmer may like to create such a contract, if only with their future self, but have reason to fear their boss will force them to break that contract in some not unlikely future.
Re:SQLite as an Application File Format (2014) by scottlamb on Sunday June 14, 2020 @09:15PM (#23522925)
+1. I recently made a variation of this argument on my FOSS app. [1] If it were commercial software with support I'd feel even more strongly.

Philosophically, people should be able to do what they want on their machines. But expecting support (eg figuring out how their third-party software has corrupted my database) is another matter, so I can see why people would install a speed-bump or two...

[1] https://github.com/scottlamb/moonfire-nvr/issues/44#issuecom...

Re:SQLite as an Application File Format (2014) by traverseda on Saturday June 13, 2020 @01:00PM (#23510809)
>That's mean, but you'll just have to live with that

I mean I know we're all on board with the idea of intellectual property actually being a thing now, but surely there are limits? I've seen people take the hard-line stance that if something is your property you should be able to dictate exactly under what situation it can be used, but there have to be limits to IP holders rights on some level, and I feel like reverse engineering a file format is a pretty reasonable place to draw that line.

Re:SQLite as an Application File Format (2014) by matheusmoreira on Saturday June 13, 2020 @02:16PM (#23511402)
> we're all on board with the idea of intellectual property actually being a thing now

We most certainly are not. I personally believe that intellectual property as a whole doesn't make sense in the 21st century and should be abolished.

> there have to be limits to IP holders rights on some level

There are. The laws generally recognize fair use and reverse engineering for interoperability.

> I feel like reverse engineering a file format is a pretty reasonable place to draw that line

Absolutely. Unfortunately, in the US it seems corporations can force people to give up their rights by making them agree to it. Therefore, "you must not reverse engineer our software" is a standard clause in every contract and it's not negotiable.

Re:SQLite as an Application File Format (2014) by kstrauser on Saturday June 13, 2020 @03:13PM (#23511869)
I'd be more on board with it if it were taxed like other property. That patent's worth $1B, you say? Better let Accounting know. Conversely: you're suing for $1B for a patent violation but you only paid taxes on it being worth $300? $300 max it is, then.
Re:SQLite as an Application File Format (2014) by kgwgk on Saturday June 13, 2020 @03:19PM (#23511915)
> taxed like other property.

Most property owned by companies is not taxed.

Re:SQLite as an Application File Format (2014) by patmorgan23 on Saturday June 13, 2020 @07:36PM (#23513800)
Actually physical property is at least in my state both physical plant and inventory are taxed.
Re:SQLite as an Application File Format (2014) by kgwgk on Saturday June 13, 2020 @08:04PM (#23513949)
Interesting, I was not aware of those taxes.

https://taxfoundation.org/does-your-state-tax-business-inven...

Re:SQLite as an Application File Format (2014) by mcny on Sunday June 14, 2020 @07:26AM (#23516869)
Nice introduction. Thank you.

I think taxes are very difficult. Personally, I think when Google or Facebook places an ad on its own (public) platform, they should have to pay a tax similar to a sales tax as they have sold themselves. Probably shouldn’t apply to the ads on google.com homepage as they don’t sell those as far as I understand.

The article mentions neutrality in taxes which is either silly or disingenuous though. Our tax code is not neutral. We openly use tax code as a way to motivate public behavior.

Ad space as inventory would be an interesting concept. It is tricky though. What is the value of something before it is sold? What is a fair tax rate? How often does this inventory adjust as you have more or fewer users? Messy.

Re:SQLite as an Application File Format (2014) by chii on Sunday June 14, 2020 @02:27AM (#23515556)
Yep, it's a great idea - treat intellectual property like any other property. However, corporate lobbyists would shoot this sort of legislation down, so i don't have any faith that it can ever come to fruition.
Re:SQLite as an Application File Format (2014) by traverseda on Saturday June 13, 2020 @09:16PM (#23514316)
> we're all on board with the idea of intellectual property actually being a thing now

Admittedly I was being a bit facetious about that.

Re:SQLite as an Application File Format (2014) by derefr on Sunday June 14, 2020 @12:23PM (#23518951)
> I personally believe that intellectual property as a whole doesn't make sense in the 21st century and should be abolished.

Do you believe that models should have no right to be compensated if some corporation takes a picture of them off the internet and uses it in their own ad campaign?

Do you believe that McDonalds should be free to make and sell happy-meal toys of whatever kids’ movie is hot lately, even using the logo of the movie in their advertising, with no obligation to compensate the people who made the movie?

Do you believe that if an inventor comes up with a new system for drug delivery and tries to sell it to some pharma company—but the pharma company turns around and does industrial espionage to get access to the technique themselves—then the pharma company should be able to just walk away with the new technology, with the inventor left with no legal recourse?

Those are all “intellectual property”, too. IP isn’t just software patents and overextended copyright terms. A world truly without any IP law wouldn’t be a utopia for innovation; it’d be a dystopia of every middleman having the legal right to produce and sell their own fakes of everything, to the point that brands cannot exist.

It’d be a world where every store, even the brick-and-mortar ones, even the ones selling things like drugs, work like shopping on Wish/AliExpress.

It’d be a world where e.g. the Coca-Cola company makes fake Pepsi products that looks exactly like the ones PepsiCo makes but which taste much worse, and puts them in stores beside the real PepsiCo products, to get Pepsi drinkers to buy those, taste them, think “Pepsi isn’t the same any more”, and switch.

It’d certainly be a world where every drug is just a generic, because you couldn’t maintain a drug brand in the face of identical dups—but it’d also be a world where even the generic store brands of drugs could be switched out at every step of the supply chain for cheaper/nastier alternatives, with no legal consequence (as long as the resulting drugs still met FDA standards.) Without IP law, there’d be no legal recourse to the suppliers who did that. It’d be, at best, a contractual dispute; and so would effectively always come down to the relative depth-of-pockets of the buyer vs. the elements of their supply chain.

Is that a world you want to live in?

Certainly, we need IP reform. But not even the most hardcore libertarian really wants to live in a world where every kind of IP right is unilaterally abolished. Having global capitalism entirely unfettered by IP rights, is like having a car entirely unfettered by brakes. You don’t get a faster car; you get a car that crashes into trees a lot.

Re:SQLite as an Application File Format (2014) by zzzcpan on Sunday June 14, 2020 @01:39PM (#23519603)
More competition can't lead to nastier alternatives, more competition leads to both higher quality and lower prices. And power of copyright that protects pretty much just the wealth of rich corporations doesn't simply disappear. Taking it away creates new power for everyone else equally distributed, leading to more competition, more independent creators and small companies, not more rich corporations, as those won't be able to concentrate wealth without it.

The world you are describing is impossible and part of it already exists precisely because of the power of copyright, not the other way around (models are already screwed and have to give up rights to corporations, they certainly aren't powerful enough to monitor where their images are used, inventors too have to give up rights to corporations and do get their stuff stolen, remember how Google did that? And it was just one public occurrence).

Re:SQLite as an Application File Format (2014) by derefr on Monday June 15, 2020 @11:44AM (#23528621)
> More competition can't lead to nastier alternatives

Pre-FDA "patent medicines" beg to differ. Austria's wine industry (https://en.wikipedia.org/wiki/1985_diethylene_glycol_wine_sc...) begs to differ. Those were competitive markets! And they certainly increasingly optimized for something as competition increased. What they increasingly optimized for, though, was the naive experience that made people buy the product (e.g. taste; "feeling good"), at the expense of health/welfare outcomes. It turns out that some poisons taste good, and make your product more popular!

Over the long term, statistically, yes, companies that do bad things in the name of profitability probably get boycotted and die.

In the medium term, though, the people that bought the products die as well. That's a necessary step in that long-term equilibrium. The "state of nature" of a capitalist market is one where companies cut corners until the corners kill people, and then people get mad and get together to kill those particular companies.

This is a dynamic equilibrium though. It's not that you end up with corporations afraid to cut corners. You end up with a constant stream of new corporations forming, going well for a while, cutting corners, killing people, and then being taken apart. It's the reverse of P.T. Barnum's "a sucker is born every minute": an unethical entrepreneur is born every minute, to take advantage of those suckers. On average, the set of corporations in existence at any given moment would be half "one step away from killing people", and half "already killing people but nobody's noticed yet."

And indeed, this is how things were for much of the Victorian era (with cyanide-based paints, nitrocelluloid plastics, and other such already-well-known hazards continuing to be sold on the open market) up through to the 1950s.

Being humans rather than animals, we have the unique capability (if not often the motivation) to learn an object lesson without having to personally suffer its negative consequences even once. We can see someone else burn their hands on a stove, and then make a rule about not touching stoves, such that nobody ever has to actually burn their hand on a stove to personally re-derive that rule again.

Re:SQLite as an Application File Format (2014) by matheusmoreira on Sunday June 14, 2020 @03:47PM (#23520723)
I apologize but I cannot continue this thread. I've been warned before about ideological discussion on HN so I shall limit myself to expressing my opinion without elaborating.
Re:SQLite as an Application File Format (2014) by jtbayly on Saturday June 13, 2020 @01:04PM (#23510848)
I didn't take "you'll just have to live with that" as a claim that the user couldn't reverse engineer it. It simply means that they've made it harder for the user to do so, and so the user has to live with it being harder.
Re:SQLite as an Application File Format (2014) by TheRealPomax on Saturday June 13, 2020 @03:28PM (#23511983)
this.
Re:SQLite as an Application File Format (2014) by jstanley on Saturday June 13, 2020 @01:41PM (#23511134)
> I know we're all on board with the idea of intellectual property actually being a thing now

I don't know that that's true at all! I'd say the pendulum is swinging in the opposite direction.

Re:SQLite as an Application File Format (2014) by deathgrips on Saturday June 13, 2020 @01:56PM (#23511248)
>we're all on board with the idea of intellectual property actually being a thing now

No. Intellectual property is not genuine property. It is a state granted monopoly and is antithetical to free market principles.

Re:SQLite as an Application File Format (2014) by rswail on Sunday June 14, 2020 @02:33AM (#23515572)
"Intellectual property" isn't a thing, "Intellectual Property Rights" are. They're not "antithetical to free market principles" if you accept that the "free market" requires regulation and that there are benefits to society by the cost of temporarily restricting free use with the subsequent benefit of ideas and expressions not being lost on the death/loss/bankruptcy etc of the IPR holder.

IPRs are (in general) things that are protected by law and open to licensing and civil suits for damages if used outside those laws and licenses:

* Trade Secrets and NDAs

* Copyrights

* Patents

* Trademarks

Re:SQLite as an Application File Format (2014) by Eremotherium on Sunday June 14, 2020 @10:16AM (#23517776)
NDAs, etc. are contract law. Contracts need to be entered in without duress as they would be felonies otherwise. The others are societal contracts which operate much more akin to taxes from a philosophical point of view. But calling today's copyright law a temporary restriction is cynical at best.
Re:SQLite as an Application File Format (2014) by deathgrips on Sunday June 14, 2020 @04:11PM (#23520901)
Every government protection starts as a temporary relief. Then the corporations that benefit from those regulations use the extra revenue to lobby for more exclusively beneficial regulations that harm their competitors. The result is copyrights that last for infinity minus one years, infantile industries requiring temporary protections yet never leaving infancy, and regulatory boards staffed by CEOs from the industries they regulate. Copyright law, much like communism and eating ice cream for dinner every night, sounds good only in isolation without a historical perspective.
Re:SQLite as an Application File Format (2014) by cortesoft on Saturday June 13, 2020 @01:47PM (#23511177)
They aren't saying you have to live with the legal restriction, but rather the technical restriction of having to reverse engineer it without the headers
Re:SQLite as an Application File Format (2014) by Gibbon1 on Saturday June 13, 2020 @07:00PM (#23513587)
I think the limit is when they are using their IP to prevent you from owning your own data.
Re:SQLite as an Application File Format (2014) by II2II on Sunday June 14, 2020 @10:28AM (#23517851)
Even when a person is on board with intellectual property rights, there is still a distinction between the vendor created program and the user created data. Encoding the data in an undocumented or obfuscated file format may not exert legal rights over that data, but it effectively does so.

I can only see three reasons why someone would support intellectual property rights with respect to file formats: they believe the manipulation of data done by software implies a transfer of ownership of the data, at least in its modified form; they are making a cynical grab for control over the data; or they are incredibly naive.

(There are border cases, such as novel compression schemes, where how the data is stored is the product. That does not really matter when someone is using a file format as a simple container for data. If a file format is truly a border case, there should also be ample forewarning to the end user.)

Re:SQLite as an Application File Format (2014) by lbutler on Saturday June 13, 2020 @12:55PM (#23510764)
I can understand why they do it, it's enterprise software so the more open you make it, then the less licences they can sell.

They do have a module you can purchase to run API calls and access their files/software but as you probably guessed that's another $40k license!

Most of my apps I build use this API, but for me to provide to other companies they need them to also buy the API extension.

I'd love to cut out the middle man and I'll do it eventually when I reverse engineer the header!

Re:SQLite as an Application File Format (2014) by xenonite on Sunday June 14, 2020 @10:06AM (#23517724)
Although the circumventing of security measures would be illegal in many countries. (Like removing the copy protection of a game)
Re:SQLite as an Application File Format (2014) by freeopinion on Saturday June 13, 2020 @01:50PM (#23511199)
At $40K/seat, that must be some special software. Simple economics invites competition. Depending on the software, it may not take too many seats to incent the funding of an alternative.

Maybe this is the kind of software that requires huge development costs. But maybe it would be worth 20 seats' worth of customers joining forces to fund a team of 5 people to build you a competing app tailored to your specific needs/wants and completely under your control.

Granted, that could bump your costs from $800K/year to $1.6M+/year. But only short-term. Once your software is production-ready, you drop the costs of your current software. So think of it more like going from $8M/10 years to $6-10M/10 years but having complete control to add the features you want. And perhaps having the opportunity to recoup $millions/year by licensing to others. Or, open source it and give others the same kind of control while benefiting from the features they add. Spread your development costs across more seats to further lower your $/seat.

Or, look at the 100 employees your vendor currently has and lose heart, then hope somebody with deep pockets funds a competitor.

Re:SQLite as an Application File Format (2014) by lbutler on Saturday June 13, 2020 @02:08PM (#23511340)
This particular software is for water utilities to model and simulate their water and waste water networks.

It's mostly used so utilites can forecast growth in their areas for the next 25+ years and see the impact on their networks and feed into their capital work projects.

A decently sized utility may spend up to $200M/yr on capital works so $40k isn't even a line item!

There is completion in the market but consultants are forced to use what their clients pick and most utilites aren't that price sensitive.

There are also open source alternatives by the EPA[1][2], and most commercial operators are just wrappers around this public domain software.

I'm trying to create FOSS to help view and run these models.

[1] https://en.m.wikipedia.org/wiki/EPANET

[2] https://en.m.wikipedia.org/wiki/Storm_Water_Management_Model

Re:SQLite as an Application File Format (2014) by divbzero on Saturday June 13, 2020 @07:03PM (#23513598)
Thanks for sharing! Your FOSS projects [1] look awesome, wish you the best in building them out. I’m fascinated by examples of niche but critical software applications. It reminds me of Patrick MacKenzie’s thesis that there are more B2B software opportunities out there than you’d expect. [2]

[1]: https://github.com/modelcreate/

[2]: https://twitter.com/patio11/status/1213188681843892224

Re:SQLite as an Application File Format (2014) by lbutler on Saturday June 13, 2020 @09:27PM (#23514375)
Thanks! If you're interested I wrote an article listing all 7 FOSS apps I created this year on LinkedIn [1], I also have a site for my main library [2].

I'll eventually convert part of one of these into a B2B app to keep what I do sustainable but I want to keep as much free and open source.

[1] https://www.linkedin.com/pulse/seven-water-modelling-apps-on...

[2] https://epanetjs.com

Re:SQLite as an Application File Format (2014) by blackrock on Monday June 15, 2020 @06:33AM (#23525857)
It sounds like they are using just a basic Linear Regression to forecast growth.

And to the uninitiated, this sounds like a very fancy word, that makes the software seem smart.

But really, it’s just drawing a straight line. And as you add more counts to your x axis, the Linear Regression “forecasts” what the value is on the y axis. This is the magical number, given by the computer, and is used to determine future load or capacity needs.

Re:SQLite as an Application File Format (2014) by spacedcowboy on Sunday June 14, 2020 @05:35PM (#23521595)
shrug I used to own a company (since sold) that sold software for $50k/seat. Most people who bought it also bought the automatic failover option, which was another $50k/seat.

The movie business doesn't even blink at that sort of cost if it there's even a small chance to prevent having to set up the remote shot again. The logistics, time, hiring, transport, accommodation, equipment, wages, etc. etc. etc. all make $50k a drop in the ocean.

We spent 2 years writing the software, developing the add-on hardware that helped, and touting it around various Post-production houses. It was used on Star Wars I, The Matrix, etc. Post houses started to take it on board as well. Then we were bought, and the product discontinued. C'est la vie.

Re:SQLite as an Application File Format (2014) by blackrock on Monday June 15, 2020 @06:45AM (#23525939)
I’ve been thinking that the next realm for movie studios is to use the random AI face generated people, that look real, but don’t exist.

These faces can be planted on other fake AI bodies, that move like real people.

Then, what you have is a background full of fake AI people. They look like real people. They move like real people. No more need to hire extras.

You can just have your primary actors act in a green screen. And virtually change the world all around them.

The first company that can commercialize this, is going to make a ton of money. And might even be able to gain first-mover advantage, as they lock in all the studios.

Re:SQLite as an Application File Format (2014) by kccqzy on Saturday June 13, 2020 @05:00PM (#23512684)
I might be wrong but I thought Dropbox used to do that as well. All the sync state is stored in a sqlite file with the header changed or removed.