I'd love to be able to make secondary applications like you've described but being enterprise software they don't want to make it too easy.
They obviously want to keep people locked in with their $40k per seat application!
I guess the first step is figuring out the page size and other bits the other meta data you set in the header [1].
I know I just have to sit down and understand the format better and I will eventually figure it out...
Might be worth actually contacting them to ask why, if you can make the case that secondary applications will increase the value of their app, not decrease it.
If you modified their app's internal state db and screwed it up because they have designed their software with certain assumptions that aren't clear from just reading their db schema, that would be a nightmare for them to support. The easiest thing for them to do is just to try to discourage tampering with their internal state.
This is especially true if there's a chance that a market for secondary apps/utils will spring up. If that's to happen and be viable, they absolutely would want to put thought into what their supported interfaces are for those apps/utils, otherwise they will end up painted into a corner and unable to change their architecture without destroying a marketplace.
Just because they don't have a leg to stand on doesn't mean this isn't potentially a huge pain in your ass that you'd probably rather avoid. Imagine if you had a LOT of customers contacting you with this type of problem to the point where you felt like you were painted into a corner and had to support some ad-hoc APIs that weren't designed to be customer-facing and which you might have been planning to remove altogether because they're part of a design that's changing.
This is exactly the situation the obfuscation is attempting to avoid. They're just doing it with a technical solution rather than a human telling another human "don't do that."
To the point re: modifying internal state and screwing-up the application - If you're writing anything out to persistent storage you should assume that it's untrusted data when you read it back in. If for no other reason than physics itself is a malicious (or, at best, ambivalent) actor.
Re: the concept of untrusted data, this is off in the weeds argument for argument's sake IMO. Do reasonable validations of the state data, sure, but picking nits about the nature of trust and internal application state is an infinite hole I'm not jumping into with you.
Philosophically, people should be able to do what they want on their machines. But expecting support (eg figuring out how their third-party software has corrupted my database) is another matter, so I can see why people would install a speed-bump or two...
[1] https://github.com/scottlamb/moonfire-nvr/issues/44#issuecom...
I mean I know we're all on board with the idea of intellectual property actually being a thing now, but surely there are limits? I've seen people take the hard-line stance that if something is your property you should be able to dictate exactly under what situation it can be used, but there have to be limits to IP holders rights on some level, and I feel like reverse engineering a file format is a pretty reasonable place to draw that line.
We most certainly are not. I personally believe that intellectual property as a whole doesn't make sense in the 21st century and should be abolished.
> there have to be limits to IP holders rights on some level
There are. The laws generally recognize fair use and reverse engineering for interoperability.
> I feel like reverse engineering a file format is a pretty reasonable place to draw that line
Absolutely. Unfortunately, in the US it seems corporations can force people to give up their rights by making them agree to it. Therefore, "you must not reverse engineer our software" is a standard clause in every contract and it's not negotiable.
Most property owned by companies is not taxed.
https://taxfoundation.org/does-your-state-tax-business-inven...
I think taxes are very difficult. Personally, I think when Google or Facebook places an ad on its own (public) platform, they should have to pay a tax similar to a sales tax as they have sold themselves. Probably shouldn’t apply to the ads on google.com homepage as they don’t sell those as far as I understand.
The article mentions neutrality in taxes which is either silly or disingenuous though. Our tax code is not neutral. We openly use tax code as a way to motivate public behavior.
Ad space as inventory would be an interesting concept. It is tricky though. What is the value of something before it is sold? What is a fair tax rate? How often does this inventory adjust as you have more or fewer users? Messy.
Admittedly I was being a bit facetious about that.
Do you believe that models should have no right to be compensated if some corporation takes a picture of them off the internet and uses it in their own ad campaign?
Do you believe that McDonalds should be free to make and sell happy-meal toys of whatever kids’ movie is hot lately, even using the logo of the movie in their advertising, with no obligation to compensate the people who made the movie?
Do you believe that if an inventor comes up with a new system for drug delivery and tries to sell it to some pharma company—but the pharma company turns around and does industrial espionage to get access to the technique themselves—then the pharma company should be able to just walk away with the new technology, with the inventor left with no legal recourse?
Those are all “intellectual property”, too. IP isn’t just software patents and overextended copyright terms. A world truly without any IP law wouldn’t be a utopia for innovation; it’d be a dystopia of every middleman having the legal right to produce and sell their own fakes of everything, to the point that brands cannot exist.
It’d be a world where every store, even the brick-and-mortar ones, even the ones selling things like drugs, work like shopping on Wish/AliExpress.
It’d be a world where e.g. the Coca-Cola company makes fake Pepsi products that looks exactly like the ones PepsiCo makes but which taste much worse, and puts them in stores beside the real PepsiCo products, to get Pepsi drinkers to buy those, taste them, think “Pepsi isn’t the same any more”, and switch.
It’d certainly be a world where every drug is just a generic, because you couldn’t maintain a drug brand in the face of identical dups—but it’d also be a world where even the generic store brands of drugs could be switched out at every step of the supply chain for cheaper/nastier alternatives, with no legal consequence (as long as the resulting drugs still met FDA standards.) Without IP law, there’d be no legal recourse to the suppliers who did that. It’d be, at best, a contractual dispute; and so would effectively always come down to the relative depth-of-pockets of the buyer vs. the elements of their supply chain.
Is that a world you want to live in?
Certainly, we need IP reform. But not even the most hardcore libertarian really wants to live in a world where every kind of IP right is unilaterally abolished. Having global capitalism entirely unfettered by IP rights, is like having a car entirely unfettered by brakes. You don’t get a faster car; you get a car that crashes into trees a lot.
The world you are describing is impossible and part of it already exists precisely because of the power of copyright, not the other way around (models are already screwed and have to give up rights to corporations, they certainly aren't powerful enough to monitor where their images are used, inventors too have to give up rights to corporations and do get their stuff stolen, remember how Google did that? And it was just one public occurrence).
Pre-FDA "patent medicines" beg to differ. Austria's wine industry (https://en.wikipedia.org/wiki/1985_diethylene_glycol_wine_sc...) begs to differ. Those were competitive markets! And they certainly increasingly optimized for something as competition increased. What they increasingly optimized for, though, was the naive experience that made people buy the product (e.g. taste; "feeling good"), at the expense of health/welfare outcomes. It turns out that some poisons taste good, and make your product more popular!
Over the long term, statistically, yes, companies that do bad things in the name of profitability probably get boycotted and die.
In the medium term, though, the people that bought the products die as well. That's a necessary step in that long-term equilibrium. The "state of nature" of a capitalist market is one where companies cut corners until the corners kill people, and then people get mad and get together to kill those particular companies.
This is a dynamic equilibrium though. It's not that you end up with corporations afraid to cut corners. You end up with a constant stream of new corporations forming, going well for a while, cutting corners, killing people, and then being taken apart. It's the reverse of P.T. Barnum's "a sucker is born every minute": an unethical entrepreneur is born every minute, to take advantage of those suckers. On average, the set of corporations in existence at any given moment would be half "one step away from killing people", and half "already killing people but nobody's noticed yet."
And indeed, this is how things were for much of the Victorian era (with cyanide-based paints, nitrocelluloid plastics, and other such already-well-known hazards continuing to be sold on the open market) up through to the 1950s.
Being humans rather than animals, we have the unique capability (if not often the motivation) to learn an object lesson without having to personally suffer its negative consequences even once. We can see someone else burn their hands on a stove, and then make a rule about not touching stoves, such that nobody ever has to actually burn their hand on a stove to personally re-derive that rule again.
I don't know that that's true at all! I'd say the pendulum is swinging in the opposite direction.
No. Intellectual property is not genuine property. It is a state granted monopoly and is antithetical to free market principles.
IPRs are (in general) things that are protected by law and open to licensing and civil suits for damages if used outside those laws and licenses:
* Trade Secrets and NDAs
* Copyrights
* Patents
* Trademarks
I can only see three reasons why someone would support intellectual property rights with respect to file formats: they believe the manipulation of data done by software implies a transfer of ownership of the data, at least in its modified form; they are making a cynical grab for control over the data; or they are incredibly naive.
(There are border cases, such as novel compression schemes, where how the data is stored is the product. That does not really matter when someone is using a file format as a simple container for data. If a file format is truly a border case, there should also be ample forewarning to the end user.)
They do have a module you can purchase to run API calls and access their files/software but as you probably guessed that's another $40k license!
Most of my apps I build use this API, but for me to provide to other companies they need them to also buy the API extension.
I'd love to cut out the middle man and I'll do it eventually when I reverse engineer the header!
Maybe this is the kind of software that requires huge development costs. But maybe it would be worth 20 seats' worth of customers joining forces to fund a team of 5 people to build you a competing app tailored to your specific needs/wants and completely under your control.
Granted, that could bump your costs from $800K/year to $1.6M+/year. But only short-term. Once your software is production-ready, you drop the costs of your current software. So think of it more like going from $8M/10 years to $6-10M/10 years but having complete control to add the features you want. And perhaps having the opportunity to recoup $millions/year by licensing to others. Or, open source it and give others the same kind of control while benefiting from the features they add. Spread your development costs across more seats to further lower your $/seat.
Or, look at the 100 employees your vendor currently has and lose heart, then hope somebody with deep pockets funds a competitor.
It's mostly used so utilites can forecast growth in their areas for the next 25+ years and see the impact on their networks and feed into their capital work projects.
A decently sized utility may spend up to $200M/yr on capital works so $40k isn't even a line item!
There is completion in the market but consultants are forced to use what their clients pick and most utilites aren't that price sensitive.
There are also open source alternatives by the EPA[1][2], and most commercial operators are just wrappers around this public domain software.
I'm trying to create FOSS to help view and run these models.
[1] https://en.m.wikipedia.org/wiki/EPANET
[2] https://en.m.wikipedia.org/wiki/Storm_Water_Management_Model
I'll eventually convert part of one of these into a B2B app to keep what I do sustainable but I want to keep as much free and open source.
[1] https://www.linkedin.com/pulse/seven-water-modelling-apps-on...
And to the uninitiated, this sounds like a very fancy word, that makes the software seem smart.
But really, it’s just drawing a straight line. And as you add more counts to your x axis, the Linear Regression “forecasts” what the value is on the y axis. This is the magical number, given by the computer, and is used to determine future load or capacity needs.
The movie business doesn't even blink at that sort of cost if it there's even a small chance to prevent having to set up the remote shot again. The logistics, time, hiring, transport, accommodation, equipment, wages, etc. etc. etc. all make $50k a drop in the ocean.
We spent 2 years writing the software, developing the add-on hardware that helped, and touting it around various Post-production houses. It was used on Star Wars I, The Matrix, etc. Post houses started to take it on board as well. Then we were bought, and the product discontinued. C'est la vie.
These faces can be planted on other fake AI bodies, that move like real people.
Then, what you have is a background full of fake AI people. They look like real people. They move like real people. No more need to hire extras.
You can just have your primary actors act in a green screen. And virtually change the world all around them.
The first company that can commercialize this, is going to make a ton of money. And might even be able to gain first-mover advantage, as they lock in all the studios.
Being able to load a read-only .sqlite database might seem cool, but I can't think of a single instance in which that's smaller and/or more efficient than calling data endpoints that use gzip/brotli for transport.
Or are you thinking "as replacement for IndexedDB"? In which case, hard agree but then it's an actual database, not used as file data container.
You just blew my mind. Thank you!
I can't believe the things I find out from random HN comments.
I don't really see any difference or advantage of using SQLite over XML, unless you need full RDBMS engine power for your configuration (highly unlikely).
1. SQLite database files are self descriptive, which JSON and XML are not.
2. SQLite is a compressed binary format. JSON and XML are plain text.
3. building on that, SQLite files were designed to contain arbitrary data, and you can store whatever you want using the BLOB datatype. JSON and XML can't, they have no notion of types, everything has to be syntax-compliant strings.
4. SQLite files can be encrypted. JSON and XML can't.
5. SQLite files were designed to be queried. JSON and XML are not.
And I already hear you try to object, so let's expand:
1. JSON and XML cannot describe their own structure, their syntax is prescribed, but any schema has to come from either external files, like XML's DTD, or from literally nothing because JSON has no official schema language. Yes, https://json-schema.org/ exists, but it's certainly not an official spec - maybe "yet", maybe just "full stop".
2. Yes, you can compress JSON/XML using zip/etc but now it's no longer JSON/XML. Now it's an archive file like any other archive file, and it's "nothing" you can work with until you unpack it, incurring delays, and then repack it once you're done, incurring even more delays (because packing is far more work than unpacking), and worse: the bigger the file gets, the longer the delay becomes.
3. Sure, you can convert binary to a text format and then put that in JSON/XML too, but there no types: you have no way of universally indicating which field is plain text, and which field is binary-as-text nor a way to universally indicate which encoding you've used.
4. Same as (2): sure, you can encrypt the data yourself, but there is no universal spec for indicating what encryption you used for which fields, or parts of the file, or the entire file, in JSON or XML.
5. JSON and XML are data serialization formats: they are perfect for transport, but they are not data repositories and even at the spec level make zero affordances towards efficient data retrieval, storage, and representation. Which for an application file format is critical.
Can you use JSON/XML as application file formats? No, not really. They're terrible for that purpose. They might work for configs (as you point out), and they are fantastic for moving small quantities of data from one system to another (for large quantities, they make no sense: even something as dumb as CSV becomes more efficient when dealing with lots of data) but they are absolutely unsuitable for storing arbitrary application data, because they are horrendously inefficient for almost everything an application needs out of a good file format, without rolling your own standards, for which there is no enforcement tooling unless you write that yourself, too. And then get others to adopt it as well. And now you're started down the path of replicating what SQLite already does, and doing so poorly.
We all win when folks decide to leave it accessible, but I'm not going to hold "encrypting a file format so that people can't easily reverse engineer it" against folks who are trying to sell software.
I think I'd mostly disagree. Selling an application is one thing, but the data itself is usually customers' and holding their data hostage is not a proper thing to do.
Listen to what you're saying: the data isn't locked, you just need the... application, to... unlock it.
That's beyond tenuous, it's invalid.
That argument that a proprietary file format "locks" data ignores the fact that it's the pair {application, file} that determines whether your data is locked away or not. For instance: Microsoft Word data? not locked. You can trivially export it in quite a lot of ways. Can you easily get it from a traditional .doc file? Hell no, but that doesn't mean the data itself is inaccessible. Instead of using sqlite, or xslt, or PERL, or whatever, you use word.
https://www.sqlite.org/see/doc/release/www/index.wiki
There are third party encryption approaches for SQLite. One of the most popular is SQLCipher:
https://github.com/sqlcipher/sqlcipher/
Those two don't have compatible file formats. eg no interoperability
Not sure if any others do, but yeah it would be handy if there are.
For example, Adobe's Lightroom uses sqlite as their application file format, which means that it's almost trivial to write an application to help it do things it's either really bad/exceedingly slow at (like removing images from a catalogue that no longer exist on disk) or literally can't do (like generating a playlist of files-on-disk for a specific collection or tag), without being locked into whatever janky scripting solution exists in-app. If there even is one to begin with.
All you need is a programming language with a sqlite connector, and you're in the driving seat. And sure, you'll need to figure out the table schemas you need to care about, but sqlite comes with sqldiff so it's really easy to figure out which operations hit which tables with only a few minutes of work.
Good luck reverse engineering proprietary file formats in the same way!