Yes, you can fine-tune using plain text completions. You don't need structured conversations unless you want conversational abilities. Plain text works great if you want the model to generate text in a specific style or domain. It all depends on what you're trying to achieve.

Nice.

And about the cost of finetuning: is there a difference in price when only training the model on completions?

The cost depends on the number of tokens processed, so fine-tuning on completions costs the same per token as any other data.