krnaveen14

Reminds me of the time where I've implemented highly optimised version of Sieve of Eratosthenes in Java, just for a Code Golf Challenge.

Memory Usage is greatly reduced by using BitSet.

https://github.com/krnaveen14/PrimeNumbers/blob/master/src/m...

https://codegolf.stackexchange.com/a/66915/41984

OMG, it's been 8 years already!

The Loneliness of a pretty good developer https://news.ycombinator.com/item?id=31438426

The Arrival of Java 18!

[1]: https://inside.java/2022/03/22/the-arrival-of-java18/

JDK 18 G1/Parallel/Serial GC changes

[1]: https://tschatzl.github.io/2022/03/14/jdk18-g1-parallel-gc-c...

Yes, definitely. It's just that we used Kafka in the start for microservices communication and later choosen Pulsar for large data streaming.

We have it in our roadmap to migrate the former to Pulsar, but not the utmost priority right now. It also simplifies Infrastructure management by keeping one event streaming stack instead of two.

We choose both Kafka and Pulsar from the start. Haven't done any migrations from one to other.

If we are gonna move from Kafka to Pulsar (operational data),

- Publish new messages only to Pulsar

- Wait for Kafka Subscriber to process all pending messages

- Start Pulsar Subscriber (step 2 and 3 can done in parallel or sequential based on message ordering needs)

We don't need to worry about pulsar subscriber offset since existing messages in kafka topic won't be copied to pulsar topic preventing duplicate messages. The whole migration would be completed in 1-2 min (depending on Step 2 wait time).

This approach is carefully taken such that there can be little lag in subscriber but the publisher should not be affected by any means.

- Kafka couldn't cope up if there are hundreds or few thousands topics. High CPU load, longer startup times...

- Even empty Kafka topic consumes 20MB of on-disk storage (that's 20GB for 1000 topics)

- Inevitable coupling of non-partitioned topic to a particular Kafka broker limiting the storage scale

- Tiered storage was not available previously in Kafka (is it now available in open source version?)

- Native Multi Tenant support with Authentication / Authorization support not available in Kafka (essential for customer based thousands of namespaces and topics)

Based on our experience with Apache Kafka and alternative streaming systems, Apache Pulsar natively addresses the Honeycomb's needs.

- Decoupling of Broker & Storage Layer

- Tierered Storage (SSD, HDD, S3,...)

We use both Kafka and Pulsar in our systems.

- Kafka is used for microservices communication and operational data sharing

- Pulsar is used for streaming large customer data in thousands of topics

A few years back we were doing heavy file processing in Java with most of the time being spend in file i/o. Initially it was designed as receive zip file, extract zip to directory, read the extracted files one by one and processing it. If the zip file is 1GB and expands to 10GB when extracted, the amount of IO being done is significantly large. 1GB Read -> 10GB Write -> 10GB Read. Suppose our AWS Instance type is capable of 50GB/s, we were spending a minimum of 420 sec in IO operation itself.

This limited the throughout capacity of number of files which can be processed within a duration where the next set of zip files would be received in fixed interval. Since we passthrough the file only once during processing, we had to eliminitate the zip extraction and read the files in zip one by one as decompressed byte stream. This was possible with zipInputStream.getNextEntry() and reading the bytes but it posed a major refactoring and inconvenience where we now have to deal with byte[] instead of File in every place.

Then comes the most advanced nio and filesystem provider features (it was already available but we came to know about the benefits of it only then). All we had to do was simply replace File with Path and new FileInputStream() with Files.newInputStream() instead. Regarding zip file decompression, we simply replaced ZipInputStream and ZipEntry with FileSystems.newFileSystem() instead. Near instantly we were able to reduce 21GB IO into just 1GB IO reducing the total processing time exponentially.

Based on this understanding, we were able to implement the similar approach in zip file creation also where files will be written directly to zip as a compressed stream and only using Path in all places.

Developers are not just required have to a mental model about the memory constraints but also about the volume of Disk IO operations where the latency would instantly kill the application performance once the page cache cannot hold the files in memory.

I suppose many of us (including myself) are waiting for Java 17 LTS release so that there's is a single migration phase (8 -> 17) rather than doing it now (8 -> 11) and also (11 -> 17) after few months.