Seems ineffective with abstract concepts.

"In mathematics, a real number is a number that can be used to measure a continuous one-dimensional quantity such as a distance, duration or temperature. Here, continuous means that pairs of values can have arbitrarily small differences. Every real number can be almost uniquely represented by an infinite decimal expansion."

->

"toki pi nanpa, nanpa lon li nanpa li ken lukin e suli tawa wan. lon ni la, nanpa tu li ken ala li poka li ken lili. nanpa lon ali li ken lukin kepeken nimi nanpa mute tawa pini ala"

->

    "Talking about numbers, when a number is present, it can be seen to be important or significant to an individual."

    "In this context, the number two might not be significant or might be seen as small or lesser."

    "All numbers can be seen using many number names endlessly or without an end."

> Seems ineffective with abstract concepts

I’ve only heard of Toki Pona for about 20 minutes, but everything I’m reading says that’s just a normal property of the language, not a limitation of ChatGPT.

As a human, which of the 120 Toki Pona words would you confidently and correctly translate as “continuous” or “real” or any of these other linguistically uncommon concepts?

Based on my own chat with ChatGPT, I feel almost certain that it warned you about this exact limitation of the language, but you ignored that?

"continuous" (in this actually incorrect usage, since the correct term is "dense")

->

said of an algebraic structure such that for every x != y there exists z such that x < z and z < y

->

characteristic ("nimi"?) of a set ("poki") where for ("tawa?") all ("ale") thing ("ijo") different ("ante" + "lon"?) than other ("ante" + "nimi"?) thing ("ito") there exists ("lon") a new ("sin") thing ("ijo") such that the first ("wan" + "nimi" + "ito"?) thing is less ("lili") than the new thing ("sin" + "ito"?) and the new thing ("sin" + "ito"?) is less ("lili") that the second thing ("tu" + "nimi" + "ito"?)

Would need some way to join into a grammatical structure and probably a preamble that establishes that the context is mathematics.

I would definitely not attempt to translate something like that in one shot - I'd read it and digest it for a bit and come up with a way to say the same thing without being unnecessarily obtuse, Thing Explainer style. mi nanpa e mute ijo la kipisi ijo li ken ala lon. ijo wan sin li kama la mi suli wan e nanpa. taso mi ken ala suli kipisi e nanpa. nanpa pi mute ijo la ni li ken ala: ijo tu en kipisi pi ijo wan li lon. taso mi nanpa e suli ijo la kipisi nanpa ale li ken. ijo li kama suli sin pi kipisi wan la mi ken suli pi kipisi wan e nanpa. kipisi li ken lili ale. nasin nanpa ni li ken nanpa e suli ijo e suli tenpo e suli seli. nanpa pi suli ijo la ni li ken: mi sitelen e nanpa la sitelen ni li ken nanpa ni taso li sama ala sitelen pi nanpa ante. taso nanpa mute la sitelen ona li suli ale li ken ala pini. GPT3 doesn't translate this very well, it seems to miss the fine points of grammar, but maybe GPT4 would do better

This is what GPT4 came up with for the translation:

> The text you provided is in Toki Pona, a constructed language with a minimalistic vocabulary. The translation to English is challenging due to the highly context-dependent and simplified nature of the language. Without further context, here's a rough translation:

> "When I count many things, cutting things is not possible. When one new thing comes, I increase the count by one. But I can't increase the count by cutting it. In the case of counting many things, this is not possible: two things and a cut of one thing exist. But when I count large things, cutting all numbers is possible. When something becomes large by cutting one, I can increase the count by cutting one. Cutting can make everything small. This counting method can count large things, long times, and hot things. In the case of counting large things, it's possible: when I write a number, this writing can count it but is not the same as writing a different number. But with many numbers, its writing is always large and cannot end."

> Please note that the translation might not capture all the nuances, and the interpretation may vary based on the context in which the text is used.

Much closer

This gives a 404 for me

Sorry, I just deleted it from my history without realizing that it would break the share link. Updated the link with a new session demonstrating the same thing now.

This is precisely the case where ChatGPT (or any LLM) will have trouble that a layperson would have trouble validating: Toki Pona has a limited document corpus and requires contextual interpretation obvious to experts.

In short, expect hallucinations and other lies.

In this case, I validated against the hand-translated example in the main link. It did fine.

This is probably easy mode for it, given the tiny vocabulary at hand, even though it has surely seen very little Toki Pona in the wild. Large language models are extremely good at translation. I’ve personally tested ChatGPT-4 against Google Translate and other translators on some hard translations (languages that are not similar to English, such as Chinese), and it has consistently provided better translations than Google Translate in my opinion. For validation purposes, news stories that have been reported worldwide make things easier, since you can establish the facts of the translation by hand, and then compare what comes out.

What does your comment actually add to the discussion? Yes, ChatGPT can be wrong. So can Google Translate!

If you’d like to provide a clear example of ChatGPT-4 (not ChatGPT-3.5) making up a Toki Pona translation that is completely disconnected from reality, please do. That would be legitimately interesting conversational material. General cynicism isn’t.

ChatGPT absolutely has limitations, and translation is a hard problem, even for the industry-standard tools. Crossvalidation with multiple translation tools and ideally humans is important for any translations that actually matter.

Since LLMs' only valid uses involve entertainment and creativity, and Toki Pona is more or less a recreational language without serious application, I think they go together like two peas in a pod.

>only valid use...

Well. That's an opinion I suppose.

-edit- After posting, I decided this was too snarky/low value on it's own. So to add:

I personally get quite a bit of value out of it as an ad-hoc German tutor and as one-among-many tools for research and learning on new topics for which I am not already an expert.

I've also used it to help me make and troubleshoot very simple code in languages that I am not an expert in.

I personally have trivially found lots of uses that probably all cluster around "educational" to one degree or another.

Is it perfect? Absolutely not. Does it hallucinate/get things wrong? Absolutely. Is it useful despite those limitations? Emphatically yes.

This idea that its "only valid use" is entertainment doesn't even make sense to me. "Entertainment" is perhaps where I find it the most deficient relative to other current options.

  > In this case, I validated against the hand-translated example in the main link. It did fine.

Presumably that hand-translated example was also part of the training data.

Not likely. The article says it was posted 1 year ago. ChatGPT’s training data cutoff is September 2021, so no, that would not make sense.

I've done a fair amount of translating Toki Pona based on my own sentences, and based on recent online posts. None of that would have been in the training set. It does a good job. Since Toki Pona translation is "lossy", you can even prompt it to give you multiple interpretations and discussion.

Another challenge, is that toki pona requires a lot of context, not only of previous sentences, but visual and communal. For example, I can say 'soweli lili' and point to a cat, then in all further conversations 'soweli lili' will mean exactly a 'cat' until specified.

I could use sentence that could mean literally hundred thousand of different things, but if I explain it properly once, you are expected to keep it as context.

ChatGPT4 will struggle to keep all context as it will surely accumulate.

Lies are typically done on purpose and in this example used to indicate intent, when there is none.

Nope:

https://news.ycombinator.com/item?id=35483823

What about going the other way? I'm on chat gpt3.5 FWIW, but when I ask it to translate simple phrases from English to Toki Pona (electric car, barking dog, etc) it hallucinates words that aren't in the ~120 word dictionary I found.

As you noticed, ChatGPT3.5 is extremely prone to hallucination. It can still happen on ChatGPT4, but it is much less frequent, in my experience.

Here is what I got on the first try: https://chat.openai.com/share/88b0358e-c96d-4880-b939-7c62cf...

The answers seem plausible, and the words are actual Toki Pona words.

My special instructions have nothing to do with language or translation, but for the sake of transparency I did it a second time with custom instructions disabled: https://chat.openai.com/share/b85a59d9-2678-45c8-8c2e-8b8a80...

Regardless, there is a very limited dictionary to work with in Toki Pona, so communicating extremely specific ideas seems like it is always going to be problematic. The language doesn’t even have a specific word for “dog”.