avdempsey

Internet Archive | https://archive.org | Full-Time | Remote PT-ET Hours | Non-Profit

Help build web crawlers, preservation, and public access services for over one thousand partner libraries and other cultural heritage institutions.

https://app.trinethire.com/companies/32967-internet-archive/...

Our department is still in a world of ansible and VMs, so we can't yet take advantage of some of the work that's gone into making it easy to run in k8s. We're using Postgres for Temporal's persistence because we're already familiar with operating it (and it's pretty cool already you can have some choice in DB). We've hit a bug in the newest version of Temporal server that doesn't play nicely with pgbouncer in transaction pooling mode, but they've responded to our ticket and seem to have a solution. We're running on the previous version for now. Other than that, it's just been the up-front cost of building the ansible playbooks. We haven't pushed it to any kind of load limits yet, or built-out a real high availability deploy of it yet. Day to day operation at our utilization level has been no drama so far.

We’ve been using temporal (the Python SDK) for some new projects at Internet Archive. It’s early days but we’re very excited. We run our own infrastructure, and we get more power-loss events or other intermittent issues than most. The durable execution of workflows that temporal promises seems like it was made for us. And once code is “temporalized” we get to eject a bunch of ad-hoc resiliency stuff into the sun, and what’s left is a lot clearer.

There’s a lot to learn with it. I’ve seen ramp-up take a few months per engineer, though we’re also making it a little harder on ourselves by self-hosting, being the early adopters on the Python SDK (Go, Java, and TypeScript are the most mature I think), and dealing with a mix of Python async and multiprocessing (a bunch of CPU bound activities in the mix). The docs are solid, and the team is responsive to community users.

In my career I've seen resources pulled away from QA teams (sometimes completely disbanding). It would be interesting to see things swing the other way.

I'm more or less waiting on the same thing, but wonder how many it will be able to pick up. The maintainer states somewhere that many of Pylint's remaining checks involve type-checking and inferring types for less than fully typed code. I've also seen him state somewhere that writing a type-checker is probably out of scope since these are massive projects in their own right. I've not seen any statement about the expected future coverage of Pylint.

Perhaps running Ruff plus a type checker gives us close to what Pylint does today? Pylint is pretty comprehensive, and awesome for that, but I'd love to lint at the speed of Ruff.

Internet Archive | Data Engineer | Remote (US, CA) | Full-Time | archive.org

Internet Archive is a non-profit building a free library of all of the published works of humanity to share with the world. We're not there yet, but we've managed to accumulate some data along the way. Can you help us engineer it?

The Archiving and Data Services department provides services to mission-aligned organizations (primarily other libraries and cultural heritage institutions). These services include: web crawling SaaS, managed large-scale crawls, long-term digital preservation, and particularly relevant for this role: making use of these web archives and digital collections.

We're looking for a Data Engineer to help us with some of the following: - Turn researcher Jupyter notebooks into robust systems (these notebooks are mostly in Scala) - Develop data munging/wrangling/deriving workflows (we use Spark and Temporal.io) - Help administrate a 7.5 Petabyte Hadoop cluster - Potentially write jobs for our main, in-house long term storage cluster - There's always APIs that need work (these are mostly in Python) - ML experience is an interesting bonus

We're fully remote, employees can be based anywhere in US or Canada.

This is a new opening as of Dec 1, so new we're still working on getting it posted. If interested, please reach out to Alex at avdempsey [at] archive [dot] org.

Does ActivityPub support hierarchical federation? Or could it be grafted on?

The meat and meatspace “replacements” are bridge technologies that reduce the suffering of animals and people respectively. Sometimes a vegan just wants a burger. Sometimes an exec just needs a little help to let go.

Sure these bridges have plenty of problems today, and maybe they’ll go to nowhere. But maybe in the far future they’ll lead us to cruelty free, healthy designer tissues that taste better than anything an animal makes, and Bret Victor’s dynamic medium.

You can get pretty far if you don't have much in the way of concurrent writes. Concurrent writes can be safe mind you, but that safety comes from locks that are polled against at increasingly long intervals (at least in my experience backing Django with SQLite). If you have one thread making many short writes over a long period, another thread will check to see if it can acquire a lock at 1 second, 2 seconds, 5 seconds, etc and if it doesn't time that check exactly perfectly you'll keep polling until your timeout. That is, out of the box you don't have a fair queue for the write lock. That kind of access pattern doesn't describe all apps of course! And in WAL mode your reads won't be blocked by these long sequences of writes.

I don’t think there is any scheduling. Each connection polls to see if the write lock is available up to the max busy timeout setting.

The connection polls at these intervals: static const u8 delays[] = { 1, 2, 5, 10, 15, 20, 25, 25, 25, 50, 50, 100 };

So, if you are using the default 5 second timeout, and you are trying to acquire a lock while an exclusive lock is held, you will wait 1 second, then 2 seconds, then 5 seconds, and timeout. I’m not sure if you timeout after 3 total seconds have elapsed, or sometimes after the 2 and sometimes after the 5.

If you have a thread running many fast queries in a loop you can deny access to another thread that needs a lock. The other thread may get lucky and poll for the lock at the exact moment in between locks from the other thread, but it might not.

Internet Archive | Software Engineer | Remote | Full-Time | archive.org

How do you build a machine that preserves the ever-changing cultural expression of humanity and make it free for all?

https://archive.org/about/jobs.php

Internet Archive (archive.org) | Multiple Positions | Remote (SF HQ) | Full Time

Interested in a mission-driven job ensuring perpetual open access to information for a global audience? Enjoy helping scale the use of services and products critical to hundreds of national and international non-profits, libraries, universities, cultural heritage institutions, and mission-aligned organizations? If so, the Internet Archive is seeking Senior and non-Senior Software Engineers for its Archiving & Data Services team.

https://archive.org/about/jobs.php

Internet Archive (archive.org) | Multiple Positions | Remote (SF HQ) | Full Time

Interested in a mission-driven job ensuring perpetual open access to information for a global audience? Enjoy helping scale the use of services and products critical to hundreds of national and international non-profits, libraries, universities, cultural heritage institutions, and mission-aligned organizations? If so, the Internet Archive is seeking Senior and non-Senior Software Engineers for its Archiving & Data Services team.

https://archive.org/about/jobs.php

Internet Archive (archive.org) | Sr Software Engineer | Remote (SF HQ)

Interested in a mission-driven job ensuring perpetual open access to information for a global audience? Enjoy helping scale the use of services and products critical to hundreds of national and international non-profits, libraries, universities, cultural heritage institutions, and mission-aligned organizations? If so, the Internet Archive is seeking a Senior Software Engineer for its Archiving & Data Services team.

https://app.trinethire.com/companies/32967-internet-archive/...

That claim struck me as extraordinary, but sure enough: https://en.m.wikipedia.org/wiki/Past_sea_level

Check out the, serious, Pleistocene Park project: https://pleistocenepark.ru/

The basic argument runs like this: reintroduce still living species (and then megafauna), megafauna turn forests into grasslands (see what modern elephants do to trees), grasslands keep snow on the ground longer and contribute to a cooler climate.

Internet Archive - Web and Data Services | Software Engineer | San Francisco & Remote

https://internetarchive.applytojob.com/apply/6tYI5QGrTE/Seni...

The Internet Archive is a non-profit working to provide universal, free, access to all knowledge. The Web and Data Services department provides archiving services to other aligned institutions. We develop a web archiving SaaS, run custom massive web crawls for partners, help researchers mine web archives for social good, and more.

Want to help build a machine (and a team) that keeps up with the whole internet?

The position is listed for senior engineers, but we can also scale the role for non-senior engineers as well.

Contact: [email protected]

Do you have some recommended reading? I enjoyed reading Neil Postman’s Technopoly, but any contemporaries you find particularly insightful?

The temporal dimension is just as important as space. Adding a year between eating meat and a plant-based diet also makes the alternatives more appealing.

After being meat and dairy free for awhile my cravings changed. I guess our brains are plastic? I ate (and loved!) meat for 28 years, but today I don't even crave it, and can report I enjoy food just as much as I used to. If being plant-based required will-power, I don't think I'd be able to do it.

I think it's actually easier to go all the way, and avoid meat completely, than it is to cut back. If meat is a regular, if limited, part of your diet the craving never goes away.

Your phrase "each layer of indirection" made the following analogy pop into my head: eating meat is like using an electron app running in a VM. I suppose "bare metal" would be if we could ourselves photosynthesize, alas.

While this link is very interesting, its advice doesn’t seem to completely pertain to the kind of “single deployment” APIs most of us are probably making.

From the draft: “This document specifies best practices for writing specifications that use HTTP to define new application protocols, especially when they are defined for diverse implementation and broad deployment (e.g., in standards efforts).”

That’s not to say there aren’t useful ideas here (I found it very interesting in its own right), but the provisions against fixed URL schemes is followed by no commercial HTTP API I’ve ever seen.

Sure enough, the rich text formatting is copied. The workaround I have for this is the Get Plain Text app. It strips all formatting from copied text (either automatically or on a keyboard shortcut). Sad it's required for this use case. But at least this works.

I think you’re right that they need to be more affordable, and I think they will be as the market grows.

It can be surprising how little code you're actually using from the Go-Kit project, and how it's really more of a structure for your app.

It will not save you time typing: each route can have a transport request decoder and response encoder (encoder's are often shared), a request and response type, the actual business logic, a function that takes the request struct, applies it to the business logic function and returns a response object, a function that takes the previous function and connects it with the transport machinery (building the actual endpoint), and possibly middleware implementations.

However, by teasing all of these things apart from the very beginning it's very easy to come back later and add a circuit-breaker, or (as the article says) switch your transport from HTTP to grpc (or support both).

Fascinating! I don't think the article went into what physically encoded numbers into the probability streams to begin with, only mentioning that this is simple. Does anyone know about that? Is this happening on the analog to digital conversion?

If you can make a transition to a high protein, mostly plant-based diet work in the U.S., then that gives you an existential proof that it would someday be possible for countries further back in the consumption curve.

At the same time, we also have evidence that “leapfrog” development can happen, e.g. France’s Minitel phase is something the U.S. skipped.

Figuring out a desirable reduced meat lifestyle (people have to want it for themselves) then appears to be highly relevant to combat global warming.

By the same token, a gradual increase in the legal and customary respect shown to animals could inculcate a growing respect for our fellow humans.

This looks pretty sweet so far. I love having multiple panes, multiple tabs, and good shortcuts. When you click on a row it also appears in a vertical orientation in the sidebar on the right, that's a nice touch.

Today, I use mysql in the terminal most of the time. The main reason I've been using the terminal instead of SequelPro is my terminal has the scrollback available from my previous queries. This is great for comparing results, printing out a few table schemas to help write a query, and acting as kind of a log of my thought process. Terminal has a lot of other disadvantages, but I've accumulated a lot of tricks to mitigate them.

A nice implementation of tabs and panes could chip-away at that advantage, definitely, so this app is exciting. One limitation in the current implementation is it looks like table views need to take up the whole tab. It would be nice to have a table view in one pane, and a query editor in the next.

The app has a nice query history sidebar. In addition to the query, it has the timestamp; my terminal doesn't have that. Could this concept be expanded to include query results? My first instinct was also to try dragging and dropping one of the queries from my history into the desired pane. That didn't work, but its behavior of adding it to the pane where the cursor was last works well.

All in all, I think this will definitely find a regular place in my tool set; thank you for making something cool!

There are some short term wins of course, especially as they can use this to paint their competitors’ business model and strengths as a negative. But I think this approach will pay additional dividends when they move deeper into healthcare and more computers make the move onto and then into our bodies.

My understanding is they try to mitigate this by having a minimum account balance to participate.