petrosagg

Materialize | Staff Software Engineer (Sources/Sinks) | NYC or Remote (US/Canada/EMEA) | https://job-boards.greenhouse.io/materialize/jobs/5357900004

Materialize is a cloud operational data store (ODS) enabling fast, incremental, and consistent transformations of live data using SQL.

We’re hiring Staff Software Engineers to build high-performance, distributed systems in Rust, focusing on efficient data ingestion and delivery. You’ll work on data integration pipelines, leveraging cutting-edge concepts like differential and timely dataflow to ensure correctness and efficiency.

Note: the linked position is focused on the data integration aspect of Materialize but we're also looking for wider database roles so feel free to apply if the general area of Materialize seems interesting to you!

What we’re looking for:

- 5+ years of systems-level engineering (Rust, Go, C/C++, or similar; Rust experience a plus).

- Strong CS fundamentals (B.S./M.S. equivalent).

- Bonus: distributed systems, SQL databases, CDC, or stream processing experience.

Compensation: $200K–$225K + equity.

Backed by Kleiner Perkins and Lightspeed ($100M+ raised), we’re remote-friendly and committed to building a diverse, world-class team.

Materialize | Staff Software Engineer (Sources/Sinks) | NYC or Remote (US/Canada/EMEA) | https://job-boards.greenhouse.io/materialize/jobs/5357900004

Materialize is a cloud operational data store (ODS) enabling fast, incremental, and consistent transformations of live data using SQL.

We’re hiring Staff Software Engineers to build high-performance, distributed systems in Rust, focusing on efficient data ingestion and delivery. You’ll work on data integration pipelines, leveraging cutting-edge concepts like differential and timely dataflow to ensure correctness and efficiency.

What we’re looking for:

- 5+ years of systems-level engineering (Rust, Go, C/C++, or similar; Rust experience a plus).

- Strong CS fundamentals (B.S./M.S. equivalent).

- Bonus: distributed systems, SQL databases, CDC, or stream processing experience.

Compensation: $200K–$225K + equity.

Backed by Kleiner Perkins and Lightspeed ($100M+ raised), we’re remote-friendly and committed to building a diverse, world-class team.

Materialize | Staff Software Engineer (Sources/Sinks) | NYC or Remote (US/Canada/EMEA) | https://job-boards.greenhouse.io/materialize/jobs/5357900004

Materialize is a cloud operational data store (ODS) enabling fast, incremental, and consistent transformations of live data using SQL.

We’re hiring Staff Software Engineers to build high-performance, distributed systems in Rust, focusing on efficient data ingestion and delivery. You’ll work on data integration pipelines, leveraging cutting-edge concepts like differential and timely dataflow to ensure correctness and efficiency.

What we’re looking for:

- 5+ years of systems-level engineering (Rust, Go, C/C++, or similar; Rust experience a plus).

- Strong CS fundamentals (B.S./M.S. equivalent).

- Bonus: distributed systems, SQL databases, CDC, or stream processing experience.

Compensation: $200K–$225K + equity.

Backed by Kleiner Perkins and Lightspeed ($100M+ raised), we’re remote-friendly and committed to building a diverse, world-class team.

Thank you for the detailed response!

> However, while checksums can be used under the “Crash Consistency Model” to solve consistency through power loss, PAR showed that checksums are not sufficient to be able to distinguish between a torn write at the end of the (uncommitted) WAL caused by power loss, and a torn write in the middle of the (committed) WAL caused by bitrot.

The PAR paper states that "although Crash preserves safety, it suffers from severe unavailability". I assume that when TigerBeetle loads state from RAM into a CPU cache/register it operates under the NoDetection consistency model or the Crash consistency model if ECC RAM automatically resets the CPU on read errors. At the same time it doesn't suffer from severe unavailability so what gives?

The answer is probably that ECC RAM is just reliable enough that the NoDetection/Crash models are fine in practice.

I can believe that off-the-shelf checksum and redundancy options offered by filesystems like ext4 and ZFS or systems like RAID don't hit the required error probabilities but why does the argument stop there? Couldn't a distributed database generate error correcting data on every write in the application layer so that the probability becomes low enough such that NoDetection/Crash become a non-issue for storage, just like RAM? Is there some other fundamental difference between reading and write data from RAM versus a disk?

> Sure, we’re not yet injecting storage faults, but then formal proofs for protocols like Raft and Paxos assume that disks are perfect, and depend on this for correctness? After all, you can always run your database over RAID, right? Right? > If your distributed database was designed before 2018, you probably couldn’t have done much. The research didn’t exist.

I'm trying to understand this part but something seems off. It seems to imply that those proofs do not apply to the real world because disks are not perfect, but neither is RAM. The hardware for both RAM and disks has an inherent error rate which can be brought down to arbitrary levels by using error correction codes (e.g ECC RAM).

I'm assuming that the TigerBeetle correctness proofs are predicated on perfect RAM even though in reality there is a small probability of errors. This tells me that there is an error rate which they consider negligible. If that's the case what is the difference between:

* TigerBeetle's storage approach

* Paxos or Raft with enough error correction on disk writes that the probability of errors equals that of ECC RAM which is considered negligible

I've probably made a logical error in my reasoning but I can't see it. Can someone enlighten me?

The feature you're referring to is const generics which is different than generic associated types (GATs) which is the feature being stabilized.

min_const_generics have been on stable for while

I must be missing something. The paper describes the algorithm of the CRDT and mentions that "timestamps ’t need to be globally unique and totally ordered".

Then it mentions multiple times that Lamport clocks/timestamp can be used as the timestamp in their system but as far as I know these only give a partial order of events. How is this reconciled in their system?

Here you go https://www.youtube.com/watch?v=WJj_NMhYwf0

The OP link is over https, which appears to work fine but only for redirecting you back to http

transcript: https://www.infoq.com/presentations/netflix-five-level-owner...

Thanks for explaining this

Thanks for explaining this, I have a much better idea about those systems now

Yes, and this problem is exacerbated by the raspberrypi's microUSB power input. We've seen countless number of cases where a Pi exhibiting SD card corruption was also experiencing undervoltage. If you're using raspberrypis in production it's paramount to have a good microUSB cable and power supply

I'd be interested to hear about the failure modes of I2C that you've observed. Unfortunately I'm not familiar with CAN or SPI. Do these alternatives provide error correction as part of the protocol? Do they offer superior reliability in some other way?

balena founder here. balenaOS does use the hardware watchdog available on the raspberry pi to detect CPU lock ups and automatically reset the board. On top of that we're also running software watchdogs that check the health of key system components and restart them if they become unhealthy.

It's true that SD cards are known for getting corrupt. balenaOS separates the partitions of the device and keeps the userspace in a readonly one while keeping mutable OS state in a separate partition. We are very conscious of writing as infrequently as possible to the SD card for this reason. The partition that accepts the most writes is the one holding the user container, which will get written to during an update, and also in case the user container stores any data on the device.

I'm aware that SD cards will internally swap blocks and don't really care about partition boundaries but assuming you're using an SD card with a well designed firmware it shouldn't lose a block during wear leveling.

That said, the SD card problem is one reasons we designed balenaFin :)

balena founder here. balenaOS comes with all the infrastructure needed for robust host OS updates. We expose this functionality to our users via a button in the web dashboard. We don't yet have an automated, rolling upgrade style mechanism.

The main consideration for a feature like this is that sometimes containers have dependencies to interfaces exposed by the operating system which are not always stable. This is especially true for IoT usecases because containers will typically interface with some device connected to the system.

Tangential to this, we're working on an extended support release schedule (a la firefox) for balenaOS. I could see us building an automated OS update mechanism on top of that. We'll definitely think about it, thanks a lot for your feedback :)

Resin.io | Fullstack, frontend, security, golang engineer, Etcher maintainer, software engineer, technical sales lead | remote | full-time | salary + equity

Resin.io provides a software platform that helps developers build, deploy and manage code on connected devices. We also maintain a variety of successful open source projects including etcher.io, balena.io, and resinOS, and make contributions to high-exposure projects such as Docker, Electron, and AppImage.

Fullstack Engineer - https://resin.workable.com/jobs/209405

  You will work on core resin.io features throughout the whole stack on a
  service infrastructure for IoT devices, and solve complex technical
  challenges that have an impact on a worldwide network of GNU/Linux embedded
  devices

Frontend Engineer - https://resin.workable.com/jobs/640840

  You will develop web and command-line frontends using cutting edge
  technologies (React, Redux, TypeScript), and contribute back to our open
  source UI toolkit

Golang Engineer - https://resin.workable.com/jobs/759862

  You will shape the future of container technologies in embedded Linux and
  IoT. You will work in Balena, our open source Moby-based container engine.

Software Engineer - https://resin.workable.com/jobs/759871

  We take process seriously. You will have a crucial role in defining how
  resin.io writes, tests, and deploys code. 

Open Source Maintainer for etcher.io - https://resin.workable.com/jobs/357361

  You will take the lead of the Etcher image flasher, one of the most popular
  open source Electron applications, and make it a core part of the IoT
  provisioning and development workflow. You will also work on Etcher Pro, a
  standalone drive duplicator product based on the Etcher software.

Technical Sales Lead / Sales Engineer - https://resin.workable.com/jobs/619036

  We run sales like we run engineering. While closing deals is important, we’re
  more interested in helping customers succeed with resin.io and solving their
  technical challenges.

The problem is that when a user refreshes the page using the reload button the browser will revalidate even cached resources. This results in a lot of requests that get carry the `If-Modified-Since` header and 304 responses. Cache-Control: immutable avoids this revalidation altogether since it's guaranteed that the source won't ever change.

>No more jQuery from Google CDN, no more analytics/exception tracking/fonts from your favoured provider and no more advertising conversion tracking

You don't have to trust the CDN for any static resource you load as long as you use the subresource integrity feature of modern browsers[1]. You basically include the hash of the content on your main domain and the browser will validate it when it loads it from CDN. So you only need to trust your main site.

[1] https://developer.mozilla.org/en-US/docs/Web/Security/Subres...

Edit: It was already mentioned by michaelmior https://news.ycombinator.com/item?id=12138277