Re:Apache NiFi
by mberning on Monday May 11, 2020 @02:08PM
(#23145129)
We have a team using this at work. They had built a process and needed it to be put on a VM and run periodically. They said the requirements were a dual core machine and 8gb of ram. The “binary” was like 1.8gb. I’m sure it included a jre and a full nifi runtime, but god damn that is ridiculous. Had this process been built using go or crystal or something like that it probably would have been less than a megabyte and able to run with 512mb of ram.
[ Parent
| on HN ]
From the spec (8gb) it appears that NiFi was not needed. Were you co- locating zookeeper on the same nodes?
Also you say it needed to be run periodically? It’s supposed to be a long running service. If you need something that you can spin up and shutdown in a container or VM then it is probably not the solution.
Use it if you have high volumes of data you need to transport from A -> B. We run on a cluster with 256gb ram, 128 core per a node.
The reason the binary is huge because it contains all the jar/nar for the connectors ( https://www.nifi.rocks/apache-nifi-processors/).
If you only require a subset of them you can use the minifi version which is quite small ( 49 MB for java and 3.2 MB for c++ version)
Re:Apache NiFi
by MasterIdiot on Monday May 11, 2020 @02:38PM
(#23145460)
NiFi is intended as a serious workhorse, we run it on a cluster with 256GM of RAM and over 20 cores for each server (on-prem), the 2GB of disk storage is negligible in any sort of scale (Just like the dual core/8gb machine)
Re:Apache NiFi
by RandyRanderson on Monday May 11, 2020 @02:20PM
(#23145269)
Interesting - can you expand on this: "Had this process been built using go or crystal or something like that it probably would have been less than a megabyte and able to run with 512mb of ram"?
Well, when one considers "no payload/data" flows through this system and it is merely managing metadata & workflow status, I wonder if it is web-scale enough :P your team will now ask for a cluster of these monstrosities.