News for hackers. Stuff that matters.
Re:Apache NiFi by cs02rm0 on Monday May 11, 2020 @02:52PM (#23145603)
I've used it a fair bit, though not for a couple of years. Few points, some of which may be out of date:

* I've seen customers fall into the trap of thinking they don't need expensive developers because you can drag and drop, just people who can use a mouse can crack on with NiFi.

* It persisted its config to an XML file, including the positions of boxes on the UI. Trying to keep this config in source control with multiple devs working on it was impossible.

* Some people take the view that you should use 'native' NiFi processors and not custom code. This results in huge graphs of processors with 1000s of boxes with lines between you have to follow. Made both better and worse by being able to descend and ascend levels in the graph. The complexity that way quickly becomes insane.

* You're essentially programming with it. I've no doubt you could use it to write, say, an XMPP server if so inclined. Which means you can do a great many things of huge complexity. Programming tools have developed models for inheritance and composition, abstraction, static analysis, etc. which NiFi just didn't have. The amount of repeated logic I've seen it's configuration accumulate is beyond anything I've seen from any novice programmer.

I ended up feeling like it could be an OK choice in a very small number of places, but I never got to work on one of those. The NSA linking together multiple systems with a light touch is possibly one such use case. For most everyone else, I couldn't recommend it.

Re:Apache NiFi by zokier on Monday May 11, 2020 @03:31PM (#23146012)
> I've seen customers fall into the trap of thinking they don't need expensive developers because you can drag and drop, just people who can use a mouse can crack on with NiFi.

That is sort of the key problem I see with NiFi (and equivalents). The heavy emphasis on graphical UI and visual paradigm sort of implies that its oriented towards non-developers, but problem is that it doesn't make non-developers suddenly expert system architects or developers even if they manage to click through the UI. And many developers probably prefer just defining stuff in code instead of having fancy UIs. So it sort of falls between these to categories.

Of course there is huge spectrum of skill in people, and there are probably plenty of "semi-technical" persons to whom this is perfect match, especially if supported by some more techy people.

Re:Apache NiFi by dmix on Monday May 11, 2020 @04:32PM (#23146691)
Doesn't Microsoft Access have a similar interface? It's been a long time since I've used it.
Re:Apache NiFi by EvanAnderson on Monday May 11, 2020 @05:24PM (#23147269)
Based on a quick look at NiFi, SQL Server Integration Services (SSIS) seems similar. The configuration management tooling for SSIS is pretty good and it's amenable to version control better than, it sounds like, NiFi is. SSIS still suffers from the potential "gotchas" the top-level poster mentions.
Re:Apache NiFi by drbw on Tuesday May 12, 2020 @01:49AM (#23150907)
I haven't used SSIS since about SQL2008 - back then it terrible to use with version control - not only was it a huge blob of XML, it had more xml escaped and shoved into attributes of the main document! Whats more it seemed to re-allocate the GUIDs of the elements every time you opened a diagram, so there were always changes...

Sounds like it's improved a bit since then.

Re:Apache NiFi by popinman322 on Monday May 11, 2020 @04:57PM (#23146998)
To me, honestly, this seems more similar to Azure Data Factory.
Re:Apache NiFi by mistrial9 on Monday May 11, 2020 @05:54PM (#23147556)
it is genuinely odd to see this prejuidice against graphical tools persist in 2020 .. text file dot-conf and graphical tools are different but.. is it controversial that a graphical tool can emit performant code ?

I have no experience with NiFi itself.. just that graphical tools are not inherently "non-technical"

Re:Apache NiFi by cameronh90 on Monday May 11, 2020 @06:16PM (#23147776)
I don't think it's about whether a graphical tool is performant. Many programming languages aren't performant either.

It's more that graphical tools don't generally serve their stated purpose of making it possible for non-programmers to get things done because it appears that the main skill behind programming is actually problem decomposition and modelling rather than syntax.

Additionally, graphical tools tend to have a bunch of problems inherent to them, such as being unable to write comments and harder to store in git and work on collaboratively and only having one single editor, which is usually much buggier than a text editor and compiler.

So: graphical "languages" don't make most things much easier, and make other things much harder.

Re:Apache NiFi by ibejoeb on Monday May 11, 2020 @09:58PM (#23149779)
> it appears that the main skill behind programming is actually problem decomposition and modelling rather than syntax

Totally. This is where declarative and intention-oriented systems shine. Take something like SQL, where, in the majority of cases, the end-user needs to know next to nothing about algorithmic complexity and can still achieve excellent performance and correct results.

It'd be neat to see a system that was actually designed toward problem decomposition. What's the state of the art around this kind of stuff?

Re:Apache NiFi by tonyarkles on Wednesday May 13, 2020 @12:01AM (#23163346)
We have very obviously worked on different types of end-user SQL. One that springs to mind had 30 or 40 tables, evolved over the years as “special cases” popped up, and by time I was asked to look at it, it was maybe too far gone. I tried. Maybe if the schema had been tended like a garden, the queries wouldn’t have been so crazy, but...

Mind, though, that this system had made a ton of money for 15 years, so in a way it was a huge win for them! They just eventually painted themselves into a corner and performance started to get worse faster than Moore’s law could save them. Last I heard, the great untangling is still in progress, and I stepped away from that project a few years ago.

They likely have still come out on top when you take the sum of revenue they made from the system, but they incurred a huge unanticipated cost and got backed into a pretty bad corner. Sales people were still selling features that didn’t exist, and the team was desperately trying to build new things while also fixing the ever degrading performance.

Re:Apache NiFi by hermitcrab on Tuesday May 12, 2020 @04:48PM (#23159568)
I think GUI tools come a cropped when they try to do too much, but can work very well in limited domains.

We have recently released a GUI tool to do data transformation/ETL ( https://www.easydatatransform.com ). It is aimed more at power Excel users, than programmers or data scientists. We have tried to overcome some of the issues mentioned here:

* It is only intended for the limited domain of data transformation, which limits the complexity.

* No need to remember any syntax/commands for the vast majority of use cases.

* The transformation template is stored in a single XML file, so it can be versioned.

* You can use regexp, Javascript and command line arguments to get extra power if you need it.

* It is written in C++ and is fast.

Obviously it can't do everything you can do with a general purpose programming language, but it hopefully do enough for many use cases and is much easier to learn.

Re:Apache NiFi by atoav on Tuesday May 12, 2020 @01:50AM (#23150910)
I am working with graphical tools and node based structures for years. Think about Blenders shader nodes or modular synthesizers. These systems are good to play around, try things out, quickly change how things are connected.

Above a certain complexity they become very hard to organize. If the complexity of your problem is below this level and the user has any self control, all is good. If the complexity of your problem is higher you'd need a programmer who knoes how to handle this anyways.

Re:Apache NiFi by lmm on Monday May 11, 2020 @11:56PM (#23150458)
> is it controversial that a graphical tool can emit performant code ?

No. But it's controversial to claim that a graphical tool can do maintainable code.

Re:Apache NiFi by JimmyRuska on Monday May 11, 2020 @07:30PM (#23148614)
I don't think it's a prejudice. I've seen pentaho introduced as a way to enable business analysts to do work otherwise expensive developers would have to do. It simply didn't work, business analysts were not able to use it, devs had to take over and when they did they did not want to use pentaho. It's an easy trap for upper management to fall into thinking the selling point of these tools is the ability to allow the non tech staff to do the programming.
Re:Apache NiFi by Mouse47 on Monday May 11, 2020 @10:31PM (#23149983)
+1 to this. I've seen similar stories play out with Microsoft's Logic Apps, Power Apps, and to a lesser extent Dynamics.
Re:Apache NiFi by skywhopper on Monday May 11, 2020 @07:34PM (#23148671)
That seems to be the thrust of the comment you’re replying to: the graph is essentially a programming language, with all the same complexity but multiplied because the same level of complexity is being managed without the same level of sophistication of tools like version control and even basic code re-use constructs typical in text-based languages.
Re:Apache NiFi by tiew9Vii on Tuesday May 12, 2020 @06:06AM (#23152329)
How do you do what has become basic best practices in a gui?

Everything under source control, defined as code.

* This gives you the ability to trace who did what

* Good code-commit hygiene with prs

* The ability to revert changes

* Ability to have an exact replication and promote across environments

Pragmatically test changes, unit test your flows

* Tie all the above up with a deployment pipeline

NiFi is being used in the critical parts of business data pipelines and all the devops rules for reliable development and deployment go out the window.

If NiFi is used by one person for a minor role the GUI is fine. But if it’s the core data pipeline which is a critical service used by multiple teams it’s not the smartest way to do things.

Re:Apache NiFi by chii on Monday May 11, 2020 @09:19PM (#23149546)
There's a general level of elitism in the programming world. GUI tools are seen by some programmers as a "worse" tool, but without any real evidence, and it's likely due to the fact that GUI tools enable non-programmers to reach some level of output that used to be the sole domain of programmers.

It's a similar train of thought that leads to a lot of game developers to look down on games made with GUI tools like gamemaker or such.

Re:Apache NiFi by RhodesianHunter on Monday May 11, 2020 @11:07PM (#23150184)
Can we claim that it's elitism when it's born from experience?

Programmers have seen these sorts of UIs fail to deliver countless times as described in the root of this thread. Where is the counter example?

Re:Apache NiFi by pjmlp on Tuesday May 12, 2020 @01:28AM (#23150823)
Unreal, Unity, Substance, Octane Render, and Scratch.
Re:Apache NiFi by wpietri on Tuesday May 12, 2020 @12:36AM (#23150624)
GUI tools have a very long history of being about to revolutionize the industry but ending up as dead ends. If you want evidence, just look to the past.

I'm honestly all for tools that enable non-programmers to get things done. E.g., Excel, Google Forms, Zapier, and a lot of Salesforce tooling all deliver to some extent on that promise. But there's only so much they can do.

You're right that there's also an elitism problem. But if we're going to get past that, we have to start by understanding what GUI tools are good for. And being honest about their limits.

Re:Apache NiFi by pjmlp on Tuesday May 12, 2020 @01:28AM (#23150824)
Not all game developers, AAA game engines have lots of visual languages specially for what the game designers and artists end up using.
Re:Apache NiFi by ekianjo on Monday May 11, 2020 @08:22PM (#23149143)
Its indeed drag and drop but as soon as you want to do something that's slightly a little more complex than the regular processors in place, you need to use regex filters, branching, AVRO converters, and non-tech users will be lost very quickly.

I see it very useful to automate certain operations (watch S3 storage, take action as soon as object comes in and store in into a DB), as for such use cases it's pretty much drag and drop.

Re:Apache NiFi by tiew9Vii on Monday May 11, 2020 @05:42PM (#23147441)
Another NiFi article was in Hacker News the other week with a fair few positive comments.

I left a comment with my thoughts and we pretty much agree with the exact same issues. Good to see I’m not the only one as seeing some organisations use it more and more to the extent insisting any data ingested in to systems need to go via nifi. Granted some of these are extremely large and disfunctional companies.

Re:Apache NiFi by closeparen on Monday May 11, 2020 @03:32PM (#23146023)
I have not found that argument persuasive to the managers who believe that coding is inherently wasteful if what you’re trying to do is technically possible in a workflow builder GUI.
Re:Apache NiFi by marcinzm on Monday May 11, 2020 @03:39PM (#23146090)
That's true but, personally, I just avoid working for managers like that.
Re:Apache NiFi by closeparen on Monday May 11, 2020 @03:48PM (#23146186)
I can avoid this at the line manager level, but Directors love these things.
Re:Apache NiFi by tatersolid on Tuesday May 12, 2020 @05:49AM (#23152215)
No, we don’t.
Re:Apache NiFi by monstrado on Monday May 11, 2020 @05:31PM (#23147337)
NiFi is also very useful for visually chaining together scripts (e.g. foo | bar | baz). In NiFi you can use ExecuteProcess and ExecuteStreamCommand to basically do what pipes (|) does. Further, the intermediate data between each process is backed by a WAL. You can also mix and match your pipeline, for example you can mix-in some of your custom scripts, and periodically use built-in processors that are provided to do additional stuff (e.g. PutKafka).
Re:Apache NiFi by oneplane on Monday May 11, 2020 @04:40PM (#23146784)
Compare coding to managing and suddenly they're on board. In practical terms: a checklist for work can achieve the same thing as a manager verbalising and visualising that checklist. Every argument the manager has against that is the same as 'gui workflow' vs code.
Re:Apache NiFi by ashtonkem on Tuesday May 12, 2020 @01:59AM (#23150952)
The idea that some sufficiently smart compiler/language/tool will replace expensive developers is a siren song that’s as old as COBOL. Most of these tools have historically failed to deliver much (if any) value. In the few cases where they genuinely made programming easier, the bar for what computers were expected to accomplish was raised to the point where specialized labor was once again required.
Re:Apache NiFi by leoh on Monday May 11, 2020 @10:38PM (#23150019)
Curious if anyone from the NiFi team cares to comment / has thoughts on how to work around these issues (e.g. source control)
Re:Apache NiFi by yenwel on Tuesday May 12, 2020 @01:14AM (#23150778)
They already have a solution for that https://nifi.apache.org/registry.html