Re:
by aditiyadav on Thursday February 02, 2023 @12:55PM
(#34630430)
I am planning to increase my catalogue to 1Mn. books as I get more users on the website.
Running a language model is challenging as it took ~22 hours for training on 30k books and I expect it to take at least a week on M1 GPU.
On Size, I expect 10Mn books to be taking ~16GB storage in my database.
Thanks for your comments and good luck to your data pipeline.
I'm working on different product that involves processing book content, and trying to plan for the computational requirements of millions of books and it's a lot, but I'm guessing running an language model over them would be several magnitudes more demanding.
Good luck, seems like a really promising use of AI!