brunoscheufler

Bruno from Inngest here, thanks for asking!

In general, we use OpenTelemetry[1] for instrumenting our services in production, collecting metrics and logs for important events. Specifically, we have set up

- multiple dashboards informing us about current system usage (events received, processed) including e2e latency distributions, compute resource usage for different deployments, and top operations

- metrics on critical systems (data stores including Redis, messaging infrastructure, connection poolers for Postgres, etc.) to gauge current resource utilization and typical load patterns

- alerting on unexpected deviations in KPIs (a subset of the metrics above) to help us spot and react to issues quickly

- forecasting on product usage and compute resource utilization patterns for planning medium to long-term infrastructure work

Hope this helps!

[1]: https://opentelemetry.io/

Hey, Bruno from Inngest here, great question! Due to how Redis Clustering works, each individual slot (or keyspace) is assigned to a single shard (primary/read replica group), so data remains available.

More specifically, in production we're running multiple cluster shards in a primary/read replica configuration with support for automatic failover. Individual nodes are distributed across multiple availability zones (AZs) to prevent issues in one data center from impacting the entire cluster.

In case of downtime or maintenance of the primary node, the respective read replica is automatically promoted to primary, and writes can be continued within a few seconds.

Hope this helps!

Hey HN!

I'd like to share a product I've been working on called PromptCanvas, a spatial prompt interface built to help you ideate, visualize, and test the most feasible prompts for Large Language Models.

The idea came from observing how rigid current interfaces are for exploring LLMs. Many seemed to stifle the creative ideation process and made reusing content harder than it should be - a real roadblock when you're brainstorming different prompt ideas.

PromptCanvas tries to solve this by drawing inspiration from canvas and whiteboard software. You can explore and optimize prompts visually, thanks to a familiar and intuitive user interface. PromptCanvas aims to aid marketing teams, writers, developers, and anyone who requires to create a deliverable leveraging LLMs.

Tech-wise, it's built with a stack consisting of Next.js, Vercel Edge Functions, TLDraw, TailwindCSS, Framer Motion, Radix UI, Neon, Upstash, and NewRelic. OpenAI's GPT-3.5-Turbo and GPT-4 are the underlying language models supported as of now.

Production readiness was a major focus with attention given to elements like malicious prompts, budget overruns, service outages, and system observability. It's been an interesting challenge getting familiar with TLDraw canvas and its APIs for building the extension, among other things.

You can try out PromptCanvas at https://promptcanvas.gradientsandgrit.com and learn more about the development process, usability tests conducted, and upcoming features at https://gradientsandgrit.com/promptcanvas.

Feedback and criticism are always welcome! Happy to answer any questions here.