benwilson-512

We have been doing exactly this at CargoSense! We have gotten very far on just Postgres but as our BI and other analytics queries have gotten more complex and our scale of “business tables” grown 20x over the last 18 months it just wasn’t cutting it anymore.

The CDC to Clickhouse flow has been very fast and reliable, and then for us at least setting up some incremental materialized views that set up what we need for reporting has been perfect. Clickhouse’s storage efficiency has been mind boggling.

My wife works as an acoustical consultant at a global construction firm. The things you hear about factories, offices, and even hospitals is wild. Don’t get me wrong the construction world works very hard to avoid issues but I think we in software tend to hold other engineering disciplines up on a pedestal that doesn’t quite match the messiness of reality.

The processing unit in my espresso machine died and was back ordered for 4 months, so raspberry pi to the rescue!

Is that much processing power necessary? Not even a little, but hey now it hosts its own web UI (local network only).

Firmware (Elixir, nerves): https://github.com/benwilson512/coffee_time/blob/a49814e5d8a...

Hey folks! Author of Absinthe here, feel free to ask me any questions.

Hey vryotek, short version: stickiness is not necessary or important. It does use a websocket.

To elaborate a bit, while each LiveView client is indeed associated with a long lived process in the LiveView server, the Phoenix framework provides hooks and other lifecycle management tools to minimize the impact of disconnects from eg blue green server deploys. Most interestingly for example with forms, the "automatic form recovery" does a process where when the reconnect happens it delays re-rendering until the client side has had a chance to send the current form state back to the server and allow it to synchronize.

That's very cool. Presumably as well this could be opt out if you had specific operations (perhaps from a write-only API) that don't need to wait to do further writes?

In field metering situations you can have the timeline of each meter, when it records data points, and then you have the different timeline of the backend service, when it actually _receives_ data points.

Bonus points if the meters can send data out of order.

ARM support for the JIT is a great addition for any of us on an M1! I believe on Intel the JIT was improving compile times by 30-50%.

In practice we've seen it actually improve performance, because when fetching a data range for a device fewer actual rows have to be fetched from the disk. You pick certain columns (like device ID) that remain uncompressed and indexed for rapid querying, and then the actual value columns are compressed for a range of time.

We've got a few billion rows in TSDB, pretty happy with it so far. Our workload fits the OLTP workflow more than OLAP though, we're processing / analyzing individual data points from IoT devices as they come in, and then providing various visualizations. This tends to mean that we're doing lots of fetches to relatively small subsets of the data at a time, vs trying to compute summaries of large subsets.

Compression is seriously impressive, we see ~90% compression rate on our real world datasets. Having that data right next to our regular postgres tables and being able to operate on it all transactionally definitely simplifies our application logic.

Where I see a lot of folks run into issues with TimescaleDB is that it does require that your related data models hold on to relevant timestamps. If you want to query a hypertable efficiently, you always want to be able to specify the relevant time range so that it can ignore irrelevant chunks. This may mean that you need to put data_starts_at, data_ends_at columns on various other tables in your database to make sure you always know where you find your data. This is actually just fine though, because it also means you have an easy record of those min / max values on hand and don't need to hit the hypertable at all just to go "When did I last get data for this device".

I recently installed two MrCool DIY minisplits which worked out great. They come with pre-charged quick connect coolant lines so you can install them without the need for flushing coolant or soldering the lines.

If you're comfortable putting a hole in your house for the line and running power, it can really save a lot of money.

> Specially now we are using all the hotwire stuff.

Maybe I'm missing something, but doesn't this have precisely the same latency characteristics as live view?

I do agree that some of the "you don't need redis" and similar are overblown (particularly claims about tasks replacing background jobs), but I think there is truth to these claims in the following sense: Particularly in Rails, Redis gets used not just as a tool, but also to manage shortcomings in the language / runtime. Eg: Hotwire. Ruby doesn't have a good way to concurrently manage stateful connections, so it has to fallback to Redis. Ruby doesn't have a good way to manage inter-process or inter-node communication, so once again Redis steps in to fill that gap.

There's nothing wrong with using Redis, it's a fantastic tool. Using it as a state holder for an event loop through ends up feeling extremely convoluted after you've written a few GenServers (Phoenix channels / liveview).

The most immediate impact I'm noticing locally is a ~25-30% improvement in compile times from the JIT, which is always a delightful find in any release.

Synthetic benchmarks of the Elixir GraphQL implementation show a 64% improvement! Very excited to see how this plays out with real load, but this is very promising.

And even if you're young, if you find yourself waking up at night with your heart racing, excessive snoring, etc, get it checked out. I got a CPAP machine at 20, and it's hard to overstate the impact. You can even just straight up buy them these days and adjust the settings yourself, sometimes the insurance premium on the sleep study is as high as the cost of the machine.

This looks very cool. Is there a story for managing users associated with AWS IAM roles or users?

Hey! We're looking to evaluate TimescaleDB for a logistics IoT scenario. Some of the data that enters our system comes from connected devices where recorded_at and inserted_at columns are basically the same. Some data however is sourced from dataloggers that may record for months before the data arrives at our system.

With TimescaleDB, would I use the recorded_at or inserted_at column for the hypertable?

Does this change if data for an individual sensor can sometimes arrive out of order? If the sensor malfunctions and the data contains timestamps in the far past or the far future does this cause issues with TimescaleDB?

What we've done in postgres so far is have the tables with data generally structured around the recorded_at column because most analysis wants to look at the data "in order" . to generate reports, graphs, etc. Each data row also contains a "payload_id" relating it to a "payloads" table which helps group data by when it actually hit the system. Data processing has generally been built around the payloads and then query any additional data in recorded_at order on the main data tables if we need to look back or forward in time.

Hey Erik, thanks for the post. In this vision, would this cluster of servers be reserved exclusively for timeseries data, or do you imagine it containing other ordinary tables as well?

We're using postgres presently for some IoT, B2B applications, and the timeseries tables are a half dozen orders of magnitude larger than the other tables in our application. Certain database operations, like updates, take a very long time because of this. I've wondered if by splitting the timeseries tables onto their own server I could handle updates independently, with the main app gracefully handling the timeseries DB being offline for some period of time.

It's more than just about downtime though. If through poor querying or other issues the timeseries db is overloaded the customer impact of the slow down would be limited.

We've also felt the complexity of React and Apollo. It works best when, as others mentioned, you've got distinct teams that can focus on each part. In situations where that isn't the case the same decouplings that make it easier for teams to operate independently just add overhead and complexity.

We're in a similar boat these days so in fact our latest projects are back to simple server side rendering, but we're still making the data retrieval calls with GraphQL. It ensures that that the mobile app and reporting tools we're also developing will have parity, and we don't need to write ad hoc query logic for each use case. The built in docs and validations have simply proven too useful to pass up, and you really don't need a heavy weight client to make requests.

> "I've only done something as a hobby so I'm not qualified to do it professionally"

Definitely don't let this hold you back. As long as you're up front about where you are, and are willing to learn, lack of professional experience with a particular language is by no means a deal breaker.

Hey folks! Co-Author of Absinthe here, happy to answer any questions about it. The post here is good, although these days we recommend using Dataloader vs Absinthe.Ecto. Dataloader really extends the idea behind Absinthe.Ecto while providing in request caching, pluggable backends, and easier query manipulation.

A lot of folks are failing to read the article. They're intentionally holding each connection open for 1 whole second. This is a whole different ballgame than benchmarks where each connection is allowed to terminate as rapidly as it can send back a plain text response.

In Erlang (and presumably Go) green threads end up in a waiting queue when waiting on further work to do, and as such don't need to be polled or consume any CPU at all while idle. It's only when they receive a message indicating more work can be done that they move into the work queue.

<citation needed>

These notes definitely seem like a useful reference, although I wish the author spoke more about what it was they were trying to accomplish. As someone with only a cursory familiarity with Lasp I've been hoping to see some applications that put its characteristics to use.

Hey! Good questions.

A) Absinthe 1.4 provides first class support for GraphQL Subscriptions, which give you the ability to push data to clients based on subscription documents they submit, triggered by events within the system. GraphQL Mutations are the way you push changes at the server, and Absinthe provides tools when building your schema to indicate a mapping of mutation fields to interested subscription fields.

B) This significantly depends on your underlying datastore. GraphQL (and consequently Absinthe) is entirely agnostic about how, where, or in what you're storing and managing your data. This may seem like a non answer but the point really is that you're entirely free to pick whatever approach works best for your data and your means of storage.

C) A significant portion of this answer is the same as B, although there are some conventions within the GraphQL community that help here. The Relay connection pattern uses opaque cursor values alongside each item returned in a page, and you can encode in these cursors whatever information you need to provide a coherent pagination experience.

D) I'm a big fan. Phoenix has actually done some really amazing stuff with an ORSWOT CRDT in order to track channel presence within a cluster. We're in the middle of looking at how to integrate that with subscriptions. More generally, I think when you can work your problem into the feature set of a CRDT they can be immensely powerful.

All in all though I think you may be ascribing to GraphQL or Absinthe a stronger role in the management of the data than it has. If you want to use CRDTs then you just articulate one in your schema and push it to clients and let them push diffs at you. Same could be said for much fo B, C, D. Absinthe doesn't manage the _state_ of your application your clients. Rather, it provides a way for your to communicate that state in whatever way you choose.

The Github page may not be of much interest unless you're trying to get started, so let me say a couple words about our subscription implementation that's in the 1.4 release candidate.

The idea with GraphQL subscriptions is that a client can submit a document that is run in response to an event on the server, which pushes the result of that document to the client. So if you had a UI showing new orders, you might have a subscription that looked like:

  subscription {
    orderPlaced {
      id
      customer { id tableNumber }
    }
  }

When an order is placed you can publish an event to this subscription field either manually, or setup pubsub relationships in the schema itself.

Here's the challenge: Suppose loading the customer of an order requires a DB lookup. If you have 100 people who all submit that subscription, and you run each of those subscriptions independently, you're going to end up with 100 individual database lookups for the same customer.

De-duping documents can help; if the documents are identical and the context you're executing them in is identical, you can just execute it once and push the values out. The problem is if you have authentication or authorization rules in the document then you can't really do that, each document needs to be executed individually within the context of whoever submitted it.

One of the main features of our subscription implementation however is that the batch dataloading mechanisms we have to avoid N+1 queries within a single document can also work with sets of documents. When an order is placed we take the full set of documents triggered by that event and run them as a batch, so any redundant DB lookups can get coalesced into a single query.

Elixir itself has been a huge help here. We can achieve this batched document storage and execution without any external dependencies like Redis and, because we integrate easily with Phoenix PubSub (other backends possible) we get cluster support for free also without any external dependencies. Add in linear scaling with the number of CPUs and it's really been a fantastic platform to build this on.

Sorry to hear it's a bother to you. The Scala implementation of GraphQL is called Sangria, so we thought we'd continue the theme. It also has the perk of providing some fun design opportunities.

Thanks! Stay tuned for for a deluge of new guide content: https://github.com/absinthe-graphql/absinthe/pull/416

We've been super busy getting the book finished but we want to make sure that we've also got everything someone needs to get going available in the docs.

Hey folks! I’m one of the Absinthe co authors, happy to answer any questions!

We’re hoping to get the 1.4 release out soon, it’s was delayed a bit to make sure some features were in place for the book. For now though the 1.4.0–rc is available and we’ve been using it in production.