A 4bit LoRA fine tune of this project would cost less than $5 to train even up to 30B/33B.

Hopefully similar work can be done with LoRA so the fine-tuning is not as expensive

I wonder how much it was total in $ for the fine-tuning.

Also, does anyone have some sort of table/formula that relates MB/GB of training data to $ for fine-tuning?

The repo above can be replicated for similar costs. Easily less than $10 for up to 30B using LoRA (which requires only 24GB of VRAM for 30B/33B and smaller).

Any chance you’re up to sharing the training parameters?

I feel like the whole Open Source ML scene is slowed down by a strong chilling effect. Everyone seems to be afraid to release models.

Meanwhile, other models are freely available up to alpaca 30b:

https://github.com/underlines/awesome-marketing-datascience/...

(But you're right, those who send DMCAs are likely to just a send a link to the original repository :)

Sending out DCMAs is a different process done, likely done by the legal team.

Makes sense if they originally licensed the model weights from Meta. Fortunately you can get the weights via torrent without agreeing to the license by visiting facebook's repository and getting the magnet link yourself: https://github.com/facebookresearch/llama/pull/73/files

Great time to start an awesome AI coding list!

https://techcrunch.com/2022/12/20/petals-is-creating-a-free-...

could be modified to create a system that's trained.

    def quicksort(arr):
        if len(arr) < 2:
            return arr
        else:
            pivot = arr[len(arr) / 2]
            left_index = 0
            right_index = len(arr) - 1
            while left_index <= right_index:
                while arr[left_index] < pivot:
                    left_index += 1
                while arr[right_index] > pivot:
                    right_index -= 1
                if left_index <= right_index:
                    temp = arr[left_index]
                    arr[left_index] = arr[right_index]
                    arr[right_index] = temp
                    left_index += 1
                    right_index -= 1
            return quicksort(arr[:left_index]) + [arr[left_index]] + quicksort(arr[right_index + 1:])

    def quicksort(arr):
        if len(arr) <= 1:
            return arr
        pivot = arr[0]
        left = [arr[i] for i in range(1, len(arr)) if arr[i] < pivot]
        right = [arr[i] for i in range(1, len(arr)) if arr[i] > pivot]
        return quicksort(left) + [pivot] + quicksort(right)

Ok I'll be careful, but if it says a bad word, I'm sending you my therapy bill.