daddykotex

  Location: Montréal, Québec, Canada
  Remote: Yes, (hybrid in Montréal)
  Willing to relocate: No
  Technologies: Scala / Java, Typescript / Javascript, Python, Rust, PostgreSQL / MySQL / SQLite, AWS / GCP, Terraform
  Résumé/CV: https://davidfrancoeur.com/resume.pdf
  Email: [email protected]

Senior Software Developer with extensive experience with Scala (and Java), Typescript (and Javascript) and Python. Very interested in functional programming languages like Haskell and Nix, but also low-level system programming like Rust.

Not an expert, so I could be wrong, but my understanding is that you could just copy those bytes directly from the wire to your memory and treat those as the Arrow payload you're expecting it to be.

You still have to transfer the data, but you remove the need for a transformation before writing to the wire, and a transformation when reading from the wire.

Thanks for the tip, that's very cool. I did not know about `markitdown` and `streamdown`.

Recently, another alternative was teased out: https://www.golem.cloud/

From the folks at Ziverge, who've worked on ZIO in Scala.

They use a similar approach I believe. It's discussed in this podcast: https://podcasters.spotify.com/pod/show/happypathprogramming...

What are the biggest difference with something like materialize.com?

I love the way you explain what's ReadySet. Congrats on the launch.

I don't write much on Hacker Nees, I'm much of a reader. But...

Your product looks great. I watched the demo on the home page and I'm impressed.

I'm definitely going to give a try. The developer integration seems awesome.

Congratulations on the release.

Thanks for Travis.

It's great and it's is really useful, at least for our open sources project.

I'm working in a small/medium business in Montreal, Canada. I've been hear for almost 3 years and AFAIK no body here takes stimulant.

Of course I might be wrong, but I know a lot of these people well enough to call them my friends and I have not see anyone taking stimulant nor overheard anyone talking about their consumption.

You can contact me too, no problem.

As you know, there are many plugins to do authentication in Kong. We started with jwt, then a coworker decided that we needed a more flexible approach so he basically forked the jwt plugin to add stuff for our needs. It quickly became confusing and hard to maintain.

When we tried to introduce a new feature (tokens similar to what Github offers with personal token, that is a revocable token with a given set of permissions), we had a rough time.

In the end, the decision to fork the plugin was maybe not the good one and the decision to bring the token feature into this plugin were maybe not the right one. But still, working in the plugin code was unpleasant in my opinion.

By the way, keep up the good work, it's a solid piece of software!

So far we're still using Kong. Planning an upgrade soon. But we try to avoid shifting to much domain logic in the form of Kong plugins.

This video was very interesting, thanks for sharing!

Disregarding the article completely, I'll share my opinion on Kong because we use quite a bit at my workplace. We use an old version (0.9.x as of right now). The things I shared below might not be true anymore in new versions but I can't tell and are regarding the plugin system.

IMO, the idea of taking things like authentication, rate limiting, etc in a proxy is a wonderful idea (https://2tjosk2rxzc21medji3nfn1g-wpengine.netdna-ssl.com/wp-...). So in theory, the approach Kong takes is wonderful, but I think the implementation is not so much.

Kong is layer over nginx and uses lua as a scripting language to add all sorts of stuff. Quickly, you reach the limit of the plugins capability so you think, well, I will write my own plugins. Well, to me it was a very unpleasing experience. I found that the thing was hard to test and hard to maintain. Maybe my criticism is more lua than Kong but since Kong relies heavily on lua, there is not much I can do.

There is also magic happening. You declare a schema.lua files to configure the store to hold your data. Then automatically you've got a DAO interface available with a bunch of method on it to work with the store. You don't know what methods are available or what arguments should be passed into these functions.

Anyway, this is my take after spending quite a few hours working on home made plugins in lua for Kong.

In the end, I'm glad Kong is open source and it's a great piece of software. It helped us reduce our applications complexity but make sure you don't start to shift to much logic into it because the plugin system can be hard to work with.

Here is an interview at Software Engineering Daily with Mike Jumper: https://softwareengineeringdaily.com/2016/06/27/apache-guaca...

It seems like it is quite different, you actually change your code to listen with NGINX unit.

https://unit.nginx.org/docs-installation.html

I came to ask the same question. The landing page is terrible it says pretty much nothing.

Thanks for sharing, it makes me realize that our messages are not independent.

Suppose you build a STREAM or TABLE from a topic and assume a field in the event is `id`. Later on, you introduce an update to this event where where your replace `id` by `user_id`, how is KSQL reacting?

How do you do a replay if you don't keep the event somewhere? I did not mean that messages were to be stored in the bus?

thank you

Ok this makes sense.

It matches what some others have been saying as well. Thanks

Ok, thanks for the pointer

I'm also wondering: how one deal with changes to events when using KSQL?

I've had the question for a while so I'll ask it here, maybe someone can help me.

Suppose you modeled your domain with events and your stack is build on top of it. As stuff happens in your application, events are generated and appended to the stream. The stream is consumed by any number of consumers and awesome stuff is produced with it. The stream is persisted and you have all events starting from day 1.

Over time, things have changed and you have evolved your events to include some fields and deprecate others. You could do this without any downtime whatsoever by changing your events in a way that is backward compatible way.

What is the good approach to what I'd call a `replay`?

When you want to replay all events, the version of your apps that will consume the events may not know about the fields that were in the event for day one.

In your model example, you forgot to override hashCode as one should: https://stackoverflow.com/questions/2265503/why-do-i-need-to...

EDIT: nvm, I completely missed it

This was not my experience. Of course you need a team that has the drive to learn the language, whatever it is.

If you go out on your own and decide to build your project with Scala and a big focus on using a functional style but no one else is following, you are going to have a bad time.

I'd like to know what are your issues with Scala and versions because we don't have these. IMHO, Scala is years ahead of Java and it's possibly so different that comparing them does not even make sense.

The transition from Java to Scala is awkward because you can do most of the things you do with Java in Scala. With that said, I think it's counter productive to do it. Scala is very different and approaching it with an imperative Java (even an OO) style is the worst thing to do.

One thing that I realized is that some people can go off and write cryptic code in Scala, very easily. This is true of other languages as well but in Scala, you can do a lot of things that will make you regret it the next time you try to read the code.

At work, we greatly benefited from a transition from Spring with Java to Play with Scala.

This mostly due to the inherent complexity in Spring and the fact that Spring developers are also Spring experts. When the main developer behind the application left, we struggled to add new features or even fix bugs because the team lacked the Spring expertise.

The rest of the business mostly dealt with Scala, so it was almost a no-brainer to go with Play.

The outcome has been very surprising. The application has better performance overall, is better suited for streaming and we have much more expertise in-house to add features and fix bugs.

The re-write was not without pain though. Spring is a well-supported and very rich framework. It probably does a bunch of things that the casual web developer will likely forget.

I had the exact same feeling.

Well written OP.