Re:LLMs work best when the user defines their acceptance cri...
by raw_anon_1111 on Saturday March 07, 2026 @09:21AM
(#47287886)
Also with Redshift - split the file up before ingestion to equal the number of nodes or combine a lot of small files into larger files before putting them into S3 and/or use an Athena CTAS command to combine a lot of small files into one big file.
So in my other case, the whole thing was
Web crawler (internal customer website) using Playwrite -> S3 -> SNS -> SQS -> Lambda (embed with Bedrock) -> S3 Vector Store.
Similar to what you said, I ran into Bedrock embedding service limits. Then once I told it that, it knew how to adjust the lambda concurrency limits. Of course I had to tell it to also adjust the sqs poller so messages wouldn’t be backed up in flight, then go to the DLQ without ever being processed.
Re:LLMs work best when the user defines their acceptance cri...
by Mooshux on Saturday March 07, 2026 @11:12AM
(#47288883)
The file splitting tip for Redshift is solid. One thing that caught us in a similar SNS/SQS/Lambda/Bedrock setup was not having a DLQ on the Lambda event source. When Bedrock started throttling hard, messages dropped silently and our vector store ended up with gaps we didn't notice for almost a week. Worth adding if you haven't ... it's the kind of thing you only miss once.
The deeper issue is that "efficient ingest" depends heavily on context that's implicit in your setup: file sizes, partitioning, schema evolution expectations, downstream consumers. A Lambda doing direct S3-to-Postgres import is fine for small/occasional files, but if you're dealing with high-volume event-driven ingestion you'll hit connection pool pressure fast on RDS. At that point the conversation shifts to something like a queue buffer or moving toward a proper staging layer (S3 → Redshift/Snowflake/Databricks with native COPY or autoloader). The LLM won't surface that tradeoff unless you explicitly bring it up. It optimizes for the stated task, not for the unstated architectural constraints.