siddontang

I like the idea of supporting multi cloud, not cloud vendor lock-in, but I still doubt providing a tool can solve this.

We have been building a DBaaS(database as a service) for many years. Until now, we have only supported AWS and GCP, have not supported Azure. It is really hard to let your product run stably on multi clouds.

For example, not all the cloud vendors provide the same infrastructure at the same time. We wanted our database run on K8s, but several years before, only GCP provided its managed K8s - GKS, so we had to provide our DBaaS on GCP at first. We also wanted to let our customer use PrivateLink to access our service, but GCP had supported too late than AWS, so we had to provide the VPC mechanism for a long time.

Even all the cloud vendors provide the same service, there are still some differences in the details above. E.g, one customer complained us that they met a backup problem on Google Cloud Storage, they found that the speed of our backup was not the same as our product spec statement. Then we found that we only fully tested this feature on AWS S3.

So Now even our DBaaS runs well both on AWS and GCP, we still don’t have the confidence to deploy the service on Azure. We plan to devote a quarter to testing to run DBaaS on Azure in the end of this year.

IMO, I still recommend handling the interfacing with the cloud directly to let us know the clouds better. Of course there are some awesome tools helping us to do some functional encapsulation, hiding the underlying implementation, but we must ensure these tools are easy enough to maintain.

IMO, Using AI for image generation is the trend, whether we like it or not. But for me, as a software engineer, I still insist on creating pictures for my own blogs or presentations, etc.

I'm not a painter, of course I paint badly. But I find that it is not so difficult to create a picture that expresses my ideas. Mostly I try to use https://excalidraw.com/ for sketching hand-drawn, and find the free stock pictures from https://www.freepik.com/.

I think this is another joy of creation, like programming :-)

Using map or array instead of switch-case is a common optimization way, but please do a benchmark if you have no confidence whether you need do or not.

E.g, it doesn't already work in Go, https://stackoverflow.com/questions/46789259/map-vs-switch-p...

But if there are too many switch-cases, the codes may be ugly to read, so maybe using a map or array is a better choice here.

I highly recommend "A Philosophy of Software Design", the book teaches me a lot about how to manage complexity in programming.

Disclosure: The maintainer of TiDB here.

When we started to build the distributed database TiDB, we had the similar discussion - whether we need to provide a so strong consistence for the customers? Inspired by Google Spanner, we also wanted to let the application programmers to take easy and not struggle with the consistent problems in applications layer by themselves.

So we decided to provide the strong consistency. But unlike Google Spanner, we couldn’t use TureTime API, so we introduced a standalone timestamp service named TSO to allocate timestamp. Using TSO is simple to guarantee all transactions’ order, but this introduces another problem - if the TiDB is deployed crossing different regions, the transaction latency is high.

Another example for consistency is from our realtime analytical processing design. TiDB is a HTAP(hybrid transaction/analytical processing) database. It is normal to keep consistency in the TP part, but for the AP part, is it necessary to keep consistency? If there is an update in the table, does the customer really need to get the latest data from the AP part? Here we still choose “YES”, we want our customers to focus on their business and not worry whether the query data result is consistent of not.

Of course, keeping consistency in the database comes at a cost. Now we have been trying our best to optimize the latency and performance problems :sob:

IMO, we choose a right way. We now have supported strong consistency at first, and then we can provide a loose consistency for performance too, but on the other hand, if we only build a loose consistent database at first, it is hard to provide a strong consistency for the customers.

yes, `FROM scratch` may be better most of the time. I just use `alphine` for many years, and have not tried `scratch` before.

We build our binary first with one image as the builder image, then use `copy` to copy the binary from the builder to the final executable image like alphine.

an example Dockerfile likes:

  FROM golang:1.18.1-alpine as builder
  # RUN apk add, wget, etc, and build the binary

  FROM alpine
  # or FROM scratch
  COPY --from=builder builder/binary /binary

  ENTRYPOINT ["/binary"]

As a chief engineer, I started to work with my colleagues to build an open source distributed, relational database from scratch 7 years ago. At that time, MySQL Sharding, or NoSQL was the popular solutions for scalability. So the first challenge for us is how to design our system architecture? Followings are something we do:

- Paper. We learned a lot from the paper, like Google Spanner, Google Percolator, Raft Algorithm, etc. - Learn from the open source projects and Leverage the power of community. E.g, we learned storage engine from RocksDB, ClickHouse, etc. to build our own engine. We also let the community contribute ideas and codes to our product. - Try to link people as many as you can and learn from them. E.g, when I read a paper, mostly I will try to contract the authors and have a discussion about the paper with them, at the same time, I also ask them to introduce their friends to me for further communication. - Sharing. The more you go out and share like Meetup, Blog, Webinar, etc., the more you get.

IMO, multi cloud is the future, just as eggs can’t be in one basket. I’ve been building a distributed database for more than 7 years, and find that many of our customers had been using different databases in different cloud vendors before they migrate to our product. E.g, one customer used Azure MySQL, Cloud SQL for MySQL, Aurora for MySQL.

Another thing I find vendor lock-in is not nice, for example, one customer, which is also a big customer of one cloud vendor, has suffered a lot from the cloud vendor’s own database. The customer finds the database it is not stable when its data size grows, has to spent a long and tough time to upgrade to the new version regularly, couldn’t get a good-enough service it expected before.

IMO, not only for our database, but also for other applications on the cloud, supporting multi cloud is mostly a MUST. And we also need to build an easy-to-use product, provide a good-enough service for our customers.

I've been building a distributed database product for more than 7 years, managing a team of nearly 300 R&D staff and serving hundreds of customers.

Customer Support is hard for us. Not only the number of OnCalls, but also the complexity of the product causing that we can't solve the problem easily and fast. We are under a lot of pressure.

To solve this, I have tried following ways:

- Build an independent team (may name Escalation Engineering or L3 in other companies, here let's name L3) to focus on the customer problem solving.

- Establish the OnCall duty mechanism, some engineers (maybe 10 one time) must join the L3 for two weeks only focusing on OnCall. Why it is so important, because I believe the quick way to learning our product is from OnCall.

- Get to know our customers better. We are a database product, if we can know our customers' business workloads, we can give them more advices to let the business application run more better in our product.

- Build many Diagnostic tool to solve the OnCall easily.

Finally, I would like to say that although customer support is sometimes painful, it allows us to better understand our customers and know the strengths and weaknesses of our products.

Very sad to hear this news, but not so unexpected.

As a distributed database engineer, we have been working with some universities and exploring the optimization of database on Intel Optane for a long time, and even have published some papers.

But everything was only in the experimental stage, it was difficult for us to use the research successfully in production, because nearly all of of our customers hadn't bought Intel Optane. They thought Optane was too expensive and didn't find suitable scenarios. As far as I know, the only successful story was from one of our customer, he used Optane for Redis which had an acceptable latency and better cost-effective.

Things go worse when the cloud is raising. We find that we can't use Optane on the cloud, so our later priority is to explore the optimization on the cloud disk like EBS GP3, etc.

I also recommend A Philosophy of Software Design, the book taught me a lot about how to manage complexity in programming.

I joined a startup 7 years ago as the first employee to build an open-source distributed database and have been working as a chief engineer since then. I write a lot of code, but most of the time I'm actually doing something other than code:

I visit customers to show our product proposition and listen to their requirements, write articles online to introduce our product, participate in meetups to talk about the technical architecture, operate the community to attract contributors, talk to ISVs or other companies about product integration...

Many of them are not programming, but why I still have to do these? I think even we have a good enough product, if nobody knows about it, it is nothing. We must try our best to improve the awareness of our product.

By the way, doing these things also make me grow faster, become more focused and even better at programming. So if you are a Software Engineer, please have a try, this is another kind of fun :-)

For me, programming not only brings me pleasure, it also brings me peace of mind. Programming can quickly put me in a state of mind flow.

My most recent experience was when I was learning Next.js. As a distributed database engineer, I nearly have no knowledge of web development, but I really want to show some database demos on the Web. So one night, I began to learn Next.js from scratch - I tried to connect to the database, query the result and then render the chart. After I deployed the application on Vercel, two hours had passed.

I felt tired but very excited. I believe this is the pleasure of programming :-)

As a back-end engineer focusing on distributed database, I nearly have no knowledge of web development. I rarely write JS, can't tune CSS, etc.

But when I met Next.js and Vercel, I found that they are very friendly to beginners. I can build the demos on the web, more beautiful and intuitive (Previously, I had only to build demo on the console).

EdgeDB seems more popular in NA and Europe. See https://ossinsight.io/analyze/edgedb/edgedb