aviaviavi

> We're a Haskell shop (and have been for over 10 years now) and are finding agentic development with Haskell to work pretty damn well.

It not that it didn't work at all, so much as that it worked much better in other languages when we compared side-to-side. It's possible we had dev practices that could have been modified further to suit Haskell better (we had been doing that for years already), but increasingly I want the toolchain to adapt to us, not the other way around.

Hey Jason! :)

Main factors were (roughly in order):

- None of us are experts in Rust, and we're all solid at Python.

- Rust felt like an under-correction for what we wanted (get all friction in front of the LLM out of the way).

- Our high-performance stuff is not being migrated at this time (Scarf Gateway), so we're just talking about basic CRUD backends here. Basically any language will work.

I'd be very curious to hear your take if you gave another language a proper try for comparison with the same tools. I think you'll be as surprised as we were.

The number of compiler runs doesn't matter as much as the total elapsed time it takes to finish the task. In just about every test we ran, LLMs are faster at building in Python than Haskell.

I would love to see that happen!

Scarf Gateway is a core service of Scarf (https://scarf.sh) that we've been running in production since 2020. You can think of it as a powerful link shortener that can also sit in front of Docker containers, Python packages, tarballs, etc - any other artifact you distribute can sit behind it. It also emits logs that can be used for robust analytics that you probably aren't getting from your package registry provider today.

It was originally a nginx+lua service that we migrated to Haskell as the requirements became more complex. Now that the code has settled, we've open-sourced it, so you can self host Scarf Gateway or get involved with development!

Scarf | Sales Engineer, Marketing Manager | Remote | Full time

Scarf builds advanced, maintainer-friendly tools for open-source adoption and download metrics.

- Sales Engineer: https://about.scarf.sh/jobs/sales-engineer

- Marketing Manager: https://about.scarf.sh/jobs/marketing-manager

Scarf | Director of Engineering | Remote (US timezones only)

Scarf builds builds maintainer-friendly tools for sustainable distribution of open-source software. Scarf identifies and connects open source projects with the companies that rely on their software.

If you love open source, love startups, and excel at leading great engineering teams, we'd love to hear from you.

Full job description: https://about.scarf.sh/jobs/director-of-engineering

Scarf | Director of Engineering | Remote (US timezones only)

Scarf builds builds maintainer-friendly tools for sustainable distribution of open-source software. Scarf identifies and connects open source projects with the companies that rely on their software.

If you love open source, love startups, and excel at leading great engineering teams, we'd love to hear from you.

Full job description: https://about.scarf.sh/jobs/director-of-engineering

Scarf | Remote (US-Timezones preferred)

- Senior Software Engineer (Frontend) - Typescript (https://about.scarf.sh/jobs/software-engineer)

- Senior Software Engineer (Backend) - Haskell, Nix, K8S, Postgres, Kafka, AWS (https://about.scarf.sh/jobs/software-engineer-frontend)

Scarf is a startup that builds maintainer-friendly tools for sustainable distribution of open-source software. Scarf identifies and connects you with the companies that rely on your OSS.

We're looking for talented engineers to join our small remote team. If interested, please send a resume to [email protected]

Scarf | Remote (US-Timezones preferred)

- Senior Software Engineer (Frontend) - Typescript, Vue (https://about.scarf.sh/jobs/software-engineer)

- Senior Software Engineer (Backend) - Haskell, Nix, K8S, Postgres, Kafka, AWS (https://about.scarf.sh/jobs/software-engineer-frontend)

Scarf is a startup that builds maintainer-friendly tools for sustainable distribution of open-source software. Scarf identifies and connects you with the companies that rely on your OSS.

We're looking for talented engineers to join our small remote team. If interested, please send a resume to [email protected]

Absolutely agree. And that's why we've put so much effort into making sure the system handles all PII as correctly and securely as possible.

End-user privacy does not need to be compromised in order to give OSS maintainers a basic quantitative understanding of how their software is used. This is our best attempt at a solution. We will be continually improving it better however we can.

Great suggestion, very appreciated. Global privacy control wasn't on my radar but this looks like what we should do. DNT is considered deprecated, at least according to MDN docs.

Information derived from IP can be much more granular than just ISP level

> The problem here is that i DON'T tolerate this from large companies either. I find the pixel tracking thing outrageous and disable images by default in my email client to avoid it.

If you are already using OSS today and grabbing that software over the internet, you are tolerating it even if you claim otherwise. If you pull something down from GitHub, Microsoft has all the data that we're talking about here.

> I understand your argument, I just find it personally strongly disagreeable

Fair! And I understand yours too. I also think your argument is more idealistic than practical for the current state of the ecosystem, especially considering how many parties already have access to this web traffic data. Maintainers having this data too is a very benign additional party to have access to it. Furthermore, it's a concrete way we can all chip in to help OSS maintainers and make their jobs a little bit easier, short of reaching for your credit card (which we should all be doing too).

> I like this approach slightly better than what scarfjs was doing. I first ran into that with react query a few years ago. It was a chilling effect for me at least and I was glad it changed.

Glad to hear you like this better, we do too. We built Scarf Gateway in a large part due to the response of the react-query community (and a handful of other projects) to scarf-js. Lots of discussion on GitHub and the Reactiflux discord provided good learnings for us: mainly that mechanisms that phone home, especially at unexpected times, were particularly unpopular. We also heard more acceptance of the idea that the registry/host platforms who already have this information could be sharing it with maintainers.

We want to support maintainers with better data in a way that best suits the OSS community and respects privacy. And so we went back to the drawing board, and Scarf Gateway is the result!

Still, I understand you may still have remaining hesitations here anyway with Scarf-powered download links. Are there any specific privacy concerns we can mitigate?

This argument conflates licensing of a piece software with the the distribution channel that distributes artifacts of that software. The service being discussed here is purely part of the distribution layer and has no footprint on the artifacts themselves. It's merely a passthrough layer sitting in front of the current stack.

If you are using open source today, you're already hitting servers that have access to all of the same information Scarf sees. Visiting a URL is by definition asking a server on the other side to process your request. That data can be very helpful to all of the great open source maintainers out there, but has historically been difficult or impossible to access. The result will be better informed maintainers, and better OSS for everyone.

User agent and other headers can be used to provide more differentiation, but you're correct to point out that limitation (assuming you meant IP not ISP).

Glad to hear and thanks for the kind words!

Fully complying with GDPR is a requirement as we build this out. Our data policies and practices have been thoroughly reviewed by our legal team. If we are doing anything incorrectly with respect to GDPR, it will be promptly addressed.

It turns out that the data we are actually storing about end-user traffic do not meet the criteria that trigger requirements for explicit consent. Scarf also operates a data processor with respect to GDPR, rather than a controller.

Well, Scarf offers free pixel tracking too so you definitely have the correct model for what we do, though sorry to hear you dislike the approach.

Our goal is to help enable OSS developers to financially support their work. Do you think it's still wrong when it's OSS developers trying to sell their services or premium offerings to the companies that already rely on their work? If so - companies are tracking people all the time at a very granular, personally identifiable level. Why should we hold OSS developers to an even higher standard than what we tolerate from large companies?

This still needs to be added to our docs. A `dnt=1` query param in a download URL is interpreted as an end-user opt-out. We plan to add more forms of opting out based on user feedback. We want to ensure it's low-friction to opt out of tracking.

Hi HN, a comment to give a little more backstory here:

At Scarf, we aim to give open source developers more visibility into how their software is being used. As people with experience distributing binaries and artifacts hosted on platforms like GitHub Releases and S3, a repeated struggle was not having any visibility into downloads. Which versions of the software were being downloaded the most? On which platforms? Where in the world? Which companies were downloading?

This year we built Scarf Gateway, which acts as a redirect/analytics layer for any container registry. Supporting other kinds of artifacts was a natural extension, and arbitrary file downloads is perhaps the most general extension we could build!

Curious to hear what people think.