Chunking 2M files a day for code search using syntax trees

Re:Chunking 2M files a day for code search using syntax trees by yding on Monday July 31, 2023 @05:21PM (#36949022)

This is really cool and a much needed contribution to helping LLMs run better on large code bases.

[ Parent | on HN ]

Re:Chunking 2M files a day for code search using syntax trees by kevinlu1248 on Monday July 31, 2023 @05:25PM (#36949084)

Thanks! Would love to see this algorithm in LlamaIndex.

[ Parent | on HN ]

Re:Chunking 2M files a day for code search using syntax trees by thomasfromcdnjs on Monday July 31, 2023 @07:04PM (#36950078)

Also thank you! I had an idea of how this all worked but this definitely concreted my thoughts.

[ Parent | on HN ]

Re:Chunking 2M files a day for code search using syntax trees by kevinlu1248 on Monday July 31, 2023 @07:17PM (#36950196)

I'm glad to hear! Let me know if you need more questions or are interested in implementing this chunking algo.

[ Parent | on HN ]

Re:Chunking 2M files a day for code search using syntax trees by sandGorgon on Tuesday August 01, 2023 @12:44AM (#36952423)

oh really ? Thats awfully kind. I'll take that in for EdgeChains as well.

[ Parent | on HN ]

Re:Chunking 2M files a day for code search using syntax trees by kevinlu1248 on Tuesday August 01, 2023 @03:26AM (#36953340)

Feel free to use the algo! Happy to help and reach out if you need help with implementation!

[ Parent | on HN ]

Re:Chunking 2M files a day for code search using syntax trees by kevinlu1248 on Tuesday August 01, 2023 @03:24AM (#36953327)

For others in this thread, this algo is now publicly accessible in LlamaIndex at https://github.com/jerryjliu/llama_index/blob/e567e6a20cf89b...

[ Parent | on HN ]