News for hackers. Stuff that matters.
Re:Apache NiFi by mberning on Monday May 11, 2020 @02:08PM (#23145129)
We have a team using this at work. They had built a process and needed it to be put on a VM and run periodically. They said the requirements were a dual core machine and 8gb of ram. The “binary” was like 1.8gb. I’m sure it included a jre and a full nifi runtime, but god damn that is ridiculous. Had this process been built using go or crystal or something like that it probably would have been less than a megabyte and able to run with 512mb of ram.
Re:Apache NiFi by ismail on Monday May 11, 2020 @04:30PM (#23146646)
From the spec (8gb) it appears that NiFi was not needed. Were you co- locating zookeeper on the same nodes?

Also you say it needed to be run periodically? It’s supposed to be a long running service. If you need something that you can spin up and shutdown in a container or VM then it is probably not the solution.

Use it if you have high volumes of data you need to transport from A -> B. We run on a cluster with 256gb ram, 128 core per a node.

Re:Apache NiFi by lordlinus on Tuesday May 12, 2020 @01:31AM (#23150834)
The reason the binary is huge because it contains all the jar/nar for the connectors ( https://www.nifi.rocks/apache-nifi-processors/). If you only require a subset of them you can use the minifi version which is quite small ( 49 MB for java and 3.2 MB for c++ version)
Re:Apache NiFi by MasterIdiot on Monday May 11, 2020 @02:38PM (#23145460)
NiFi is intended as a serious workhorse, we run it on a cluster with 256GM of RAM and over 20 cores for each server (on-prem), the 2GB of disk storage is negligible in any sort of scale (Just like the dual core/8gb machine)
Re:Apache NiFi by RandyRanderson on Monday May 11, 2020 @02:20PM (#23145269)
Interesting - can you expand on this: "Had this process been built using go or crystal or something like that it probably would have been less than a megabyte and able to run with 512mb of ram"?
Re:Apache NiFi by harikb on Monday May 11, 2020 @02:18PM (#23145243)
Well, when one considers "no payload/data" flows through this system and it is merely managing metadata & workflow status, I wonder if it is web-scale enough :P your team will now ask for a cluster of these monstrosities.