IsaiahShiner

> Which version of language? Which interpreter? Ah, very good question. Currently, it's just whatever version is in /usr/bin/$LANGUAGE. You can see in /home/$USER/.dit-languages how the language configurations are defined. You can add another configuration, just by copying the Python configuration, changing the name and path to Python2.7, and then doing this:

    validator Python2.7 {{ //Older python code }}

In the long run, this should be made a lot simpler I think, which leads me to your next question:

> How do you make sure that both sides of communication run the same logic? This will probably require some way to send language configurations along with a dit. If you send someone a dit which requires Java code, but the client doesn't even have Java, the dit should be able to recover from this somehow, by automatically installing Java, creating the configuration, etc. I'm not exactly sure how this should work, but its one of those clearly solvable problems that I'm trying to leave be until I have a good example of the real problem.

> Is the code transmitted together with data? What is benefit of running this code on receiving side instead of sending side? Yes, the code is always tied directly to the data. Currently, source code must be sent, but in the future, it could pull from a link, or even run the validation offsite, via an API (important in case you want closed source classes).

In theory, there is no reason for the receiver to run the validation, (assuming the receiver trusts the sender to validate). But one purpose of dit is to prevent a message from ever being separated from it's meaning. If a client receives some data in a dit format they have never seen, they can learn how it works by looking at the dit. The validation code can even be used as ad-hoc unit tests when trying to process the data automatically. And also remember that querying (and converting, mutating, anything else that might be added) also requires the dit code.

> Do you serialize and store validation and query code on disk? Is this good idea? Currently, the modified code is regenerated every single time. You can see the code in /tmp/dit/. This is slower, but it does mean that the code and serialized data are always as new as possible. Caching could be an option, if done carefully.

> .old data that cannot be queried as well as new data. ...old dates cannot be converted to various date formats while newer can. I'm actually not sure what you mean here, I can't picture how this could happen. Could you explain?

Hmm, I can see how I'm being confusing. This is partially trying not to give information overload with poorly written explanations, especially for features which I haven't added yet. It's also because dit is really trying to solve everything with Perfect Data all at once, and I need to go back and be clear about when I'm talking about which concepts. To solve Perfect Data, we need a single system that can do:

- Validation: This is for when you ask for a name and get "4jZw3ef\n". This already exists in database constraints, APIs, and, as you said, in things like JSON-Schema and Avro. The key difference is that the 'schema languages' are not languages. They don't have full flow control, libraries, classes, not even simple string functions. Dit does, by proxy of allowing for embedded languages, which let you do whatever you want.

- Schemas: Obviously other schema languages already do this. This is covered by dit classes. To solve the "FirstName", "LastName" issue, you just make sure you're using a Name schema like the one I demonstrated, in which Name contains a FamilyName/GivenName. But then everyone is using different classes for the same data, which you can solve by...

- Global Online Schemas: The only one really doing this is Schema.org. Everyone can agree that https://schema.org/Person is a reasonable official schema for a person. However, they have no validation, and changes are decided by a consortium, not free change. DitaBase.io would have a sort of copy of GitHub, in which each class is it's own page, and doesn't explicitly belong to anyone. Using meritocracy, democracy, and compromise, the community would agree on which exact schema of Person is best. If someone disagrees, they can make their own style and try to explain why it's better, while offering easy conversion between their style and the old style.

- Format Support/Universality: This is what I was getting at when I said "you can't write data in dit". Dit cannot afford be locked into one technology platform, the way JSON-Schema is locked to JSON, and the Semantic Web is locked to HTML. How do I write down data when you ask for it outside of a dit? In .json, .csv, .xml? Why not .yaml, .toml, .xlsx or .gsheet? There's no way to serialize data raw data as dit, only take data written in a .dit and print it some other way. The print functions and query system should allow you to request data in any format you want, including more than one. And ultimately, dit could never provide all of the features people could want. https://ipld.io/ is basically doing a JSON-Schema style of thing, specifically for the block chain. Dit isn't going to replicate their custom block chain features, so instead dit should support IPLD, by allowing data to be read and printed in their format, and converted every which other way you can think of. I mentioned DitaBase having it's own object model, but the truth is this would almost certainly reference Schema.org, at least by extension.

Perhaps the thing I was least clear on overall is that dit is still an evolving concept. I know where dit is now, and I know where dit wants to go, but I'm not really sure how to get there. Maybe assigners are a terrible way to solve that specific problem. That's fine, we'll throw them out and make something better. All I know is that Perfect Data needs to be solved, is solvable, and dit has a much better chance of solving it than any other existing solution.

Ah, this? https://news.ycombinator.com/item?id=11945722

Super interesting! It's like, data, without an interpreter, has no meaning. We rely super heavily on humans as general purpose interpreters, and when we design computers, we make sure as much of the meaning as possible is implicit.

Rich was saying that the interpreter is secondary, which is only true if data serves a single purpose. Alan is getting at the idea that data could serve every purpose if only the interpreter wasn't implicit. If the concept of "data" referred to both the value and the meaning, the language and how to interpret it.

But, from a reddit comment: An interpreter doesn't magically give meaning to data, since the interpreter has to translate to something, and that something has to be written by a program that understands both the from and to sides of the conversation.

The point of dit is to literally be that universal language. Yes, you do need both sides to explicitly lay out what they main, both to convey and understand the meaning. The difference is that the number of connections is unlimited. You need a human for that critical first step, but then the machine can speak "dit" language to any other machine that speaks dit. Rich couldn't conceive of the idea that dit could be real. I think it can. I think we can make Licklider's vision, data and programs that can move around without respect to the original developers intentions because they all speak dit.

Thank you for sharing, super interesting!

Dit is ready!

About 4 months ago, we posted just a website to HN: https://news.ycombinator.com/item?id=21263839

Now we're launching the MVP and hope you'll join us in trying to make data much better!

For a primer on what this is all really about, I wrote quite a bit on this exact topic: https://github.com/isaiahshiner/dit-cli/blob/master/docs/wha...

Hello everyone!

Its been 2 and a half months since I posted https://news.ycombinator.com/item?id=21263839

I've made a lot of progress. The most significant thing is certainly the demo! (https://www.ditabase.io/demo.html) I only knew vaguely how the actual product would look, and the demo forced me to put it to code.

This resulted in two important changes: 1. No more schemas, only objects 2. Dit is the real product

Objects take the place of schemas in every way. I can no longer picture what exactly you would want out of a schema that you can't get with an object. They cover schema relationships, and because of the way scripts work, every object can use a different language and it doesn't matter. Schemas seemed like a headache, but object orientation is very simple.

Then, the only logical delivery mechanism is a single file type like dit. Whatever you want that's related to DitaBase can be neatly packaged in a dit. That includes all the things I know about now and all the features I might want to add in the future.

The very next thing is certainly the MVP. This will consist of a CLI Linux app that validates dit files in much the same way the demo does. I am optimistic that I can release that by February 1st.

I am looking for feedback almost exclusively on the demo, and whether it makes sense to you as a first product. What projects can you picture using it for? What features seem make or break to you? Do you still think it's a terrible idea? I want to hear it all.

And to be clear, I realize that many of you are probably not my first users, but you are developers. In the future, I hope all of you will be users, so your feedback is still important.

Please let me know what you think!

Hey everybody! Looking for feedback on the essential components of the product. I would love to hear all your ideas on how dit could work differently, and how I can make it better!

For block length, if you can't use context or control character/symbols, then the label would be required to specific the length of the payload in bytes, or similar. When you reach the end of the given length, then the next token should be the closing payload tag/brace.

The important thing here is that everything gets described. The structures within a field, such as label, meta, validators, etc. always follow the specification set by the previous label. That label then sets the specifications for the next payload. And since the validators and such are Turing complete, the specifications can be anything, rebuilt within dit schemas.

Let me see if I can boil down your comments to specifc questions that I can answer.

"Is DitaBase compliant with JSON-Schema?"

The answer to this question is critical to understand about how DitaBase is different: if you want it to, then yes! The validators and converters are fully Turing complete, and not limited to Javascript either. If you can make a language interpreter in C/C++ (think Javascript V8), then you can use it with DitaBase. So whatever format you want, just rebuild it in (or port it to) dit schemas. Therefore, it isn't possible for DitaBase to not be compliant with something (at least, as far as I can think) because if someone thinks its important, they'll rebuild it in dit schemas.

"Schema.org is just documentation. Are there programatic, queryable schemas?"

Yes, but only optionally. In theory, all that is required to be a dit is a validator. You could inherit from "thing" and just directly have some text and a validator. But if you want to use Controllors (CRUD for a specific schema), Containers (abstract the shape of the data from the content) and other things like that, then yes, you need programatically defined schemas, just like in J-S. If someone has them, then you can use them. If not, in theory, if their schema is public, then you could just make a pull request with the relavent additions. DitaBase is very open compared to something like Swagger.

"Uh, anonymous functions? Duh?"

Oh, yes that makes much more sense that the ridiculous names I was giving those in the example. That actually just never occured to me. Of course these things will have to get slowly fleshed out over time, and I suppose there's no reason you couldn't have both.

"GraphQL, BSON, custom tools, cool things"

This is the great thing about DitaBase. Since you know the exact structure of any dit, and everything about dits is generic, you can just build all of this on top, without any overhead. DitaBase will almost certainly build some things like this in house, especially something like GraphQL, and some data analysis and statistical plotting utilites. Or perhaps, similar to how VScode absorbs open source projects into itself when they get really popular, DitaBase could probably do the same.

You are correct, DitaBase is in some ways extremely similar to Linked Data/Semantic Web/RDF etc. In fact, I would say that DitaBase is, in some ways, just another attempt at making the Semantic Web deliver on its original promise. DitaBase obviously couldn't exist without HTTP.

"Does dit use RDF?"

The answer to this question is critical to understand about how DitaBase is different: if you want it to, then yes! The validators and converters are fully Turing complete, and not limited to Javascript either. If you can make a language interpreter in C/C++ (think Javascript V8), then you can use it with DitaBase. So whatever format you want, just rebuild it in (or port it to) dit schemas. Therefore, it isn't possible for DitaBase to not be compliant with something (at least, as far as I can think) because if someone thinks its important, they'll rebuild it in dit schemas.

The reason DitaBase will succeed where Linked Data failed is by making DitaBase something that people in their day jobs want to use too, because the tools are just so good. Right now, a company with a data problem is extremely incentivized to seek a solution that solves only their specific problem. Obviously, because it's cheaper. But end-to-end solutions usually are not compatible with things like Linked Data, which needs solutions to be more general.

If a company sees that the best solution to that problem is becoming compliant with DitaBase, then obviously they're going to do it. But surprise surprise, now they're compliant with Linked Data as well.

Hey! Well, I can tell you that as of right now, it's hovering around 0.8% conversion, which isn't terrible. We'll see if I can improve that before activity on the page dies down.

You're right, what I have now is relatively messy. I was balancing brevity and specificity. If I was properly technical in my descriptions, it may have read like documentation. And then the other issue is that I have both business people and programmers on the page, and each gets kinda confused because I'm not sure who to cater too. Same problem with the case studies. I had them more specific before, but I cut out what I thought was unnecesary and now they appear rather simple.

I think the solution is to split the whole thing up into several dedicated pages, which I resisted because it would take longer, and this is all probably less important than building product right?

Regardless, I do now intend to add an FAQ, and write some sort of a technical paper/RFC, (which I should probably do anyway before I run off and start writing serious code). I'm not totally sure what to do about the case study, maybe just replace the entire thing with a video?

I would love to hear more about the project you tried! Naturally, I haven't been able to Google examples of people who tried this idea before and failed. It does sound different in that, I really do intend dits to be fully, totally and completely generic. Nothing other than the idea of labels with scripts which describe payloads is sacred. Any language, any style, anything. This means that it can't possibly miss a certain use case (you can just build it) but it does mean that the whole thing is rather complicated, and as others have said, maybe too complicated, like a pipe dream. I am certainly willing to accept that, but no way to find out the result of that without building it.

I really question whether a more restricted system like JSON-Schema could ever become dominant because it's much more prescriptive, especially when you have a limited number of possible data types. In my mind, data can truly look like anything, so coming up with a rigourous standard doesn't seem realistic: there can always be a more broad standard, ala https://xkcd.com/927/ as others have mentioned. I intend to beat 927 by taking the high ground. Its like Lisp: you can't make a more generic Lisp, because anything that satifisfies the components of Lisp is just a dialect. You can't have incompatible data or an incompatible competitor to DitaBase, its totally generic. I also don't see why you couldn't do UI generation with DitaBase using the controllers (I concept I have not explained yet, they allow CRUD for a specific schema).

I am not very knowledgable on Twitter, but now seems like a great time to start! I appreciate the invitiation!

An example dit is now live at https://www.ditabase.io/example.html

The first thing I'll say is that if it would be helpful, I can have an example of a real dit (showing real syntax, the way labels change on the fly, etc.) on https://www.ditabase.io/example in a few hours. I only didn't do that to save some time. Thank you for expressing interest!

So these are all questions I can answer:

1. How do I pull this into a project?

Assuming you have a dit that you can use (full of scripts and such that you need), all you have to do to use it in a project is use existing infrastructure to tell the dit to validate itself, convert to something, query the data inside, etc. The plan is that this would all work with a linux app. I suppose a repl code interpreter would also be convenient?

  dit validate workspace/assets/example.dit
  dit convert workspace/assets/example.dit "sy7QvPp9" //UUID of the schema you want to convert to.
  dit query workspace/assets/example.dit "LengthEnglish" //Lots of options, etc.

2. How do I access new dits?

All public dits would be available for free on the website. Anyone can view, reproduce, fork, pull request, etc. The starting dits would almost certainly be based on schema.org's object model. If you want something specifc, you could start at say, the DitaBase fork of https://schema.org/Organization and then look at the most popular child schemas until you find what you're looking for. The whole concept is to function the way open source software already does. If someone wants to announce they've made something new, or ask for help finding something, they use exising channels.

Naturally, some people will want to keep their dits closed. In this case, they fork public dits and keep their own closed, using whatever project managment stuff they prefer. They never have to inform anyone how they are using dits, the same way you can start a new git project without ever informing GitHub or BitBucket or something.

3. How do I write a dit and publish it?

The same way you would start a new GitHub project, but on DitaBase instead. Create an account, start a new dit, and beginning adding whatever you need for your project. Every dit gets its own project page, and dits probably will not really belong exclusively to the original owner of the project, because for really popular open dits, they will get used by lots of people.

Of course, dits are not really like regular code projects, since having a public dit that isn't on the DitaBase system isn't all that useful. You can also imagine that you might want to use tools available on the website specific to dits, like pasting a dit and asking the site to check it for you, or attempt to convert it. It might show you a graphical map of the many different components of your dit and how those relate to where you want to convert to, and if a converter is missing or bugged, you can jump right to that page and start adding one, right on the website (many converters will be extremely simple).

4. Where is the source code? Is it going to be open source and free?

At the moment, there is no product. I'm only about a month into this project and I want to get feedback before I start going nuts making a product without hearing from anyone.

I intend for the overwhelming majority of vital DitaBase features to be open source. The Linux app for example, obviously open source. Any dits someone makes, if they want them to be, totally free and open source. DitaBase can make its money in other ways, such as providing all of this stuff as a service, and capitalizing on the new market advantages of DitaBase. In the long run, I envision a search engine that queries public dit repos, rather than just webpages, and can return data directly instead of just saying "Well, here are some webpages which probably contain what you want." So, yes, I want to challenge Google search.

Hey guys, looking for feedback on the essential concept of the idea. I'd like to hear from real programmers whether they understand what DitaBase is and whether they think they would use, either now or in the future.

It's been difficult to explain DitaBase, and I still don't know if that's because its actually a terrible idea, and much too complicated. If it is a good idea, then I think sometimes good ideas look complicated like this until people figure out how best to explain them. You wouldn't have expected the internet, or say, object orientation, to be instantly and clearly obvious what it is when they were first being talked about, and that is how I see DitaBase.

Its basically that DitaBase is 4 essential concepts: schemas, validation scripts, conversion scripts, and public version control. If you take any single one out, then the really useful part, the dit file type, doesn't work. But it's also true that these concepts have never been applied in this way (that I have found), and so there isn't any default, easy language to call upon to explain DitaBase to people readily and easily.

The answer to the question "What is DitaBase?" feels like it has to be "1, 2, 3, 4" where I list out the essential components. Perhaps, its a "common language for data", or a "general data accuracy system", but those concepts sound totally meaningless by themselves.

I really would love to hear some feedback on how you understand the idea and whether my thinking on this is sound, or if I am totally off base, being caught up too close to the idea.