vikasnair

Openlayer (YC S21) | Remote or onsite (US, CA, NYC) | Full-Time | Customer Engineer

Openlayer is solving the AI reliability problem. We are looking for stellar customer engineers to join the team and help make customers happy. Think air-traffic control for our customers: you will be in charge of building requested features, integrations, and fixing bugs.

We are hiring in SF, NYC, and open to remote. Our team is currently 6 people, but we are very actively growing and adding more headcount. Those with experience in Python, Next.js, Docker, and cloud compute systems are encouraged to apply!

https://www.ycombinator.com/companies/openlayer/jobs/yIE9WI3...

Openlayer (YC S21) | Remote or onsite (US, CA, NYC) | Full-Time | Design engineer

Openlayer is solving the AI reliability problem. We are looking for stellar design engineers to join the team and help build the platform that next-gen teams use to build the most powerful AI systems.

We are hiring in SF, NYC, and open to remote. Our team is currently 6 people, but we are very actively growing and adding more headcount. Those with experience in Next.js, React, and TypeScript are encouraged to apply!

https://www.ycombinator.com/companies/openlayer/jobs/ZEEO8UB...

Hi HN!

Most of us get how crucial AI evals are now. The thing is, almost all the eval platforms we've seen are clunky and don't have any product cohesion. There's too much manual setup and adaptation needed, which breaks developers' workflows.

That's why we're releasing a simpler workflow.

If you're using GitHub, you only need to add two files to the repo (one config + one script). Then, connect your repo to Openlayer and define must-pass tests for your AI system. Once integrated, every commit triggers these tests automatically on Openlayer, ensuring continuous evaluation without extra effort.

We offer 100+ tests (and are always adding more), including custom tests. We're language-agnostic, and you can customize the workflow using our CLI and REST API.

As a final note, you can leverage the same setup to monitor your live AI systems after you deploy them. It's just a matter of setting some env vars in your staging/prod environments, and your Openlayer tests will run on top of your live data and send alerts if they start failing.

Let us know what you think!

Compared to Galileo, we offer a more comprehensive suite of evals that support more tasks than LLMs and NLP.

We offer more features around error and subpopulation analysis, versioning, running evals during development, and collaboration. Through what (I believe) is a more clean and simple DevEx and UI!

re: Lilac, there’s some intersect w/r/t dataset exploration, but we have more evals than the ones they offer. More than data quality, we give insights into data drift and model performance and let you set up expectations and get alerts on whether they fail during development and production. + distinct in some of the ways described above

We’re really happy to see more tools and platforms in this space. Definitely a big uptick since we started 3 years ago, w the advent of gen ai this is all top of mind (and deservedly so).

You can upload just the predictions of the model (and whatever metadata you want to track), so in that sense any format is supported.

If you want to unlock explainability for your tabular classification or regression, or text classification models, you can upload the actual model binary. We support a bunch of frameworks out-of-the-box, but you can use any architecture through our custom upload.

More info:

https://docs.openlayer.com/documentation/how-to-guides/uploa...

https://docs.openlayer.com/documentation/how-to-guides/write...

Will fix, thanks for catching!

Comparison table is a great idea - we will add.

Some tools you mentioned here are monitoring solutions. Our value over them is that we catch issues in your models before those models ship. Monitoring is definitely essential, but to purely rely on it is to rely on your users to catch your bugs for you. Moreover, if you can find more issues ahead of time, you can circle back to retraining / recollecting data more immediately (and can dramatically bridge the dev cycle and iterate faster).

Beyond this, we also offer an extensive set of insights and tools for debugging that these other tools do not offer - including insights that use your model rather than the data (or just the predictions, as a proxy). Specifically, by uploading your model artifacts to Openlayer, you can see explainability (LIME and SHAP values) for your predictions and generate synthetic data to run counterfactual analysis on your model (to try and probe for edge cases not already represented in your data).

And finally, this is all baked onto a clean UX for tracking the versions of your models as you work towards improving the issues you find on Openlayer (almost like git for ML).

We work with both startups and enterprises across a range of task types!

Some common ones are fraud and churn detection for financial institutions or e-commerce sites (both tabular classification examples). It's very important for these types of tasks in particular to guard against biases and false negatives, so they use us to set up wide test nets that help give them assurance that their models are working properly before they hit production (and to monitor them post-deployment).

Another example is Zuma (https://www.getzuma.com), a startup building an AI-driven chatbot that uses us to track their experiments and improve the accuracy of their NLP intent classification model.

Of course, we're also building out support for evaluating LLMs. Because this is an open problem, we've been spending a lot of time interviewing people in the space who are building these models (please reach out if this is you!).

Ha, yeah it's definitely not the most ideal!

It’s important to note a significant percentage of these deaths are cases in which other substances with trace amounts of fentanyl are consumed unbeknownst to the user. Dealers often use the same scales, which is one risk factor for cross-contaminating supply.

My mother is a physician, she just last night told me about a case she saw over the weekend in which a young 20-something nearly OD’d on fentanyl from taking ecstasy. She survived, but with life-altering brain trauma rendering her unable to remember who she is. She needs tubes for her food supply, and a ventilator to breathe.

Here you can see a few images of what is a lethal dose of fentanyl: https://www.dea.gov/galleries/drug-images/fentanyl

Scary stuff, and maybe there’s an argument there to be made in favor of legalizing current black-market drugs. Definitely a PSA to test your drugs.

CDC on cross-contamination: https://www.cdc.gov/drugoverdose/deaths/other-drugs.html

We have a Next.js website with a blog section that's powered by Sanity. Problem is, it's pretty finicky to install basic components that we like to use (which looks like Payload comes with out of the box) + communicating with Sanity via groq is clunky with Typescript (and possibly causing some SEO issues).

How simple is it to install Payload in an existing Next.js app to power basically just the /blog/* subdirectory?

Great idea! Definitely something I could see being useful in industry.

Have you thought about making this work for more things than ArXiv? Could be interesting to prepend Smort to any URL and create “rooms” for annotating the web (other static content like Wikipedia, or even temporal content like Youtube videos). But maybe that’s too wide of a scope, just got me thinking :)

This is exactly what we’re trying to solve at Unbox (S21).

Tons of potholes like this exist in AI we use everyday. We used to be ML engineers at Siri and had to invest millions into monitoring tools to stay on top. This is fine and all, but what’s better is to catch them before you ship and before your users suffer (sometimes literally, as in this case).

We think that better tools for QA-ing models, which allow more people (not just ML engineers) to get eyes on the model, might help catch mistakes proactively rather than retroactively.

Haseeb Q notably donates 30% of their income, citing 80000 as inspiration. Really interesting life story, as well:

https://haseebq.com/about/

If that’s true, as it is true for planets in the TRAPPIST system, then at least there is a potential for life to develop around the thin longitudinal slices located more centrally, where climate may be more temperate.

Also see this comment below, something I wasn’t aware of: https://news.ycombinator.com/reply?id=20212953&goto=item%3Fi...

I’m curious- at the end of the article, one of the scientists remarks that the stars might be “zipping around” their host star faster than measurements predict, which could rule out potential for life. Is this referring to your suspicion re: proximity => tidal locking => extreme temperatures? Or is there some way that speed of orbit can affect potential for life?

Perhaps FB is the only company that extended an offer with reasonable pay, or perhaps the only company willing to sponsor a Visa.

The list of possible extenuating circumstances extends to infinity when you think about it more deliberately. If you don’t, you’ve successfully bucketed someone based on quick, irrational judgment. That is poor behavior.

New-grad out of NYU CS looking for entry-level opportunities in software engineering (backend/front-end/mobile).

Location: USA

Remote: No

Willing to relocate: Yea

Technologies: Python/Django, Swift, Java, Javascript/Node/ Express/React

Resume/CV: https://www.linkedin.com/in/vikasnair/

Email: [email protected]

I had a very similar problem with an iOS app [1] I published which aggregated 3rd-party news content via RSS.

It took 3 months of back-and-forth with Apple Review to bypass those legality and content clauses in addition to a “limited functionality” clause. The key features introduced that finally put me past the review board were:

* Embedding a link to Safari on each news article cell

* Only allow content to display in SFSafariViewController (this was desired anyways)

* Allowing user to pick and choose categories of news to be shown

* A progress tracking feature which very simply measures and displays users’ time spent reading

The former two got me past the clauses identified in this article, re: 3rd party content. The latter two helped me to prove my app did something other than display 3rd party content (the limited functionality clause).

Was frustrating to find these workarounds, especially when the App Review board is not very responsive. Thankfully, some kind soul at Apple called me and helped me resolve everything within a day after my 5th consecutive reject.

[1] http://appstore.com/dossierallyouneedtoknow