wonton53

I work on a payment system that invoices consumers on behalf of other companies and pays out the money to these companies by splitting the payout on multiple partners (luckily I do not handle the support and follow-up). The system have ability to refund invoices and customers some times end up paying more than or just parts of the invoice. On top og all this it integrates with several old 90s systems with poor datetime handling and poor uptime (some times a windows xp box in the cudtomers offices). It also handles card payments and over all, by far the hardest thing to get right is the invoicing. Its just so extremely fuzzy and time sensitive.

Interesting approach. I have some coleagues that actually remote desktops to their powerful desktop computer. I never did that because I fear all the tiny issues I have to deal with, but they tell me it works quite well. Also just offloading the heavy lifting through ssh or other types of build servers seems super interesting (I think some build systems have built in support for a common server, not sure if it is true, but it could be very helpful if it works well)

I think it is quite common. I develop financial systems, and I compile apps regularly for running tests locally and large parts of the system is running locally using docker compose. This has the benefit of fast iteration for unit and system tests. Of course the tests and deployment is also run by a CI server but that is just for QA purposes so people dont have to remember to run their tests locally and to keep a clean main branch. I cannot imagine how long our feedback loop would be if we had to wait for the CI server for every change, or if we had a common dev server that everyone interacted with through a «dirty» development branch. Basically I think it is just different use cases. My experience with jupyter is that you use it for data analysis which is really more IO bound than cpu bound, also big data is often not possible to keep locally (because of the size and also in europe GDPR)

I might have misunderstood the scope of the idempotency layer, I thought it was this thin layer close to the web server that just replays the last answer from the services that are lower down. So that means stripe actually goes down to some store or external system to check the transaction status for subsequent calls with the same key?

I know how complicated integrating with stuff like the bank, 3Dsecure, AML and fraud prevention is. One of the systems I integrated with had no refund functionality, they expected a manually initiated bank transfer from our customer bank to the end user! So I certainly understand some states are irrecoverable in a web request. It is important to do though, because it saves all clients work for each automated recovery that can be done.

> Practically though, it doesn't happen very often

It probably happens much more often than you think if you say this as someone with an insider perspective. I have had to integrate with banking systems and payment systems that use this approach, and it is extremely frustrating and comes of as a way of offloading work to the client. If a payment capture succeedes but subsequently always returns 500 an api client has to first query the status and then execute if-then logic for something that could easily just have been a retry of the request (3x the logic). This is acceptable since there are a million other ways to mess up idem potency so integrations kind of end up this way anyway. But the worst part of such an approach is debugging the issues as a client. A client cannot see the internal logs and therefore have to call support which will ALWAYS answer «the transaction seem fine with us» and basically just close the case to protect their KPIs. Im pretty sure this type of issue has a name but cannot find it (state client see dont match the real state). I dont mean to offend the work people put into these APIs, but I cannot see the good qualities of this approach other than saving development hours (and possibly saving one db query, but as an effect you get a status request plus another capture request as a workaround from your clients).

I can absolutely see that there are many cases where the code is not run many times or the dataset is dirty and that types actually get in the way. As a example, I cannot stand using stuff like VSCode with lsp because it is unbearably slow for both linting and code completion. I feel pycharm is faster (actual typing is slow, but code completion makes me rarely type more than 3 characters before completing which makes it feel faster). I’m mentioning this because I would never dream of typing out a data structure, which is probably because my work is so different from an analysis job. For me code for data structures should be either generated or be simple enough to type out in a few minutes and there should be as few as possible of them. In sharp contrast to data engineering type of work where data just is dirty because it is so much of it from everywhere. Anyway, in both cases I dont want to type a full variable name without some completion, ever, much less 100 times which it seems some people are completely fine with. I’m showing my inexperience with python here, but I would love if there existed something like the F# type providers so I do not have to type so damn much:D

That makes a lot of sense. And I think the lack of teaching technique has to do with the broad variety of programming tasks. Techiques are way too specific to the domain and we won’t see more of it in school until «computer science» is split into more and more fine grained studies or get a bigger part of existing studies like math is now. e.g now I feel embedded, backend and frontend are distinct «directions». In the future I expect it to be further branched into for example finance, healthcare, chemical processing, public infrastructure. This is gradually happening all the time though and «true» computer engineering will always exist, but I think it will be more and more niche just like embedded and OS level programming is more niche now but was the only option before. Anyway I think types are key to making programming safe enough to bring it to this kind of broader workforce (in some distant future, with good IDE support).

Very true, but there is a gradient there. I have an example of the opposite extreme. A few years ago I worked with a person who would rather continue working in php 5.3 than start using Scala, which was hot at the time. He literally copy-pasted an old project instead of starting a new one in Scala (or ANY other language, this was a payment system and php does not even have strong enough crypto support in old versions). And I can tell you for sure he had at least 1 year exposure to Scala and doing at least 1 full project. His reasoning for copy-pasting old php: He was «just too stupid to understand Scala». It is an extreme example, but most people I met who dont like strong typing fall into that category (lazy? Certainly not stupid) and I just dont know what to say to them or what to think myself to accept them or even how to use whatever hidden powers these people have other than creating circular dependencies for the company by making obvoius mistakes they themselves spend most of their time fixing. Im not actually bitter, just frustrated that so many seem to just not care.

That makes perfect sense actually, thank you for taking the time to answer. And I’m indeed a SW eng with a degree, so spot on. I do something similar to data analysis «for fun»/non professinally but my day job is payment/banking systems. Live payment systems and settlement systems are both very different from analysis tasks. They have all the focus on «correctness» and integration interfaces with other organizations (which are ALWAYS wrong because «they» don’t understand types/strict interfaces). While analysis in my experience is dealing with tons of dirty data sources not in real time or the code will even just run once (not millions of times). I always go to cleaning the input and validating the data, but I suspect that is because of my day job and i feel it is less valuable to put in this effort than it is for my day job. So I have myself also skipped using types many times to learn whatever library I need or to iterate quicly (minutes compared to days). For example, it is very comfortable to just let pandas give you 0 for NaN when you just want a plot. So I understand it takes more time because the value of types is probably much lower to you in absolute terms.

Great! I am so glad more people see types as a useful tool. Personally I see it as essential, so I struggle to understand the mind set of people who do not want to use them. I understand that people think it feels like a lot of work, but that is like nothing compared to the 90% of time people who dont use types spend on reading logs and fixng the same issues every day. I assume people have some sort of amnesia and thinking «oh the value was undefined, I have never seen THAT error before», but that is probably just me being bitter. So I was wondering, since you seem like a super-fresh converted;did you have any specific reservations originally, and whas there any specific type of solution to a issue that finally changed your mind?