News for hackers. Stuff that matters.
Re:The OpenTF Manifesto by dbingham on Tuesday August 15, 2023 @11:17AM (#37135029)
I appreciate the letter and trying to work with Hashicorp -- I used to have a ton of respect for Hashicorp. But honestly... at this point...

...just fork it into a foundation. Don't wait for Hashicorp's response. I get wanting to have the appearance of working with Hashicorp, but we've been shown again, and again, and again, and a-fucking-gain that private corporations cannot be trusted to maintain public goods. Only community governed non-profit foundations can do that.

Private corporations will put the bottom line first every single time. And in the case of investor funded enterprises, the bottom line is never ending exponential growth or bust.

Re:The OpenTF Manifesto by brikis98 on Tuesday August 15, 2023 @11:28AM (#37135180)
The truth is that a fork hurts everyone.

Imagine a future CTO trying to pick the IaC tools for their company. They see Terraform as an option, but then learn there are multiple forks, licensing questions, and a big battle happening in the community. What do they do? They are now way more likely to pick a different tool that is genuinely open source. The same is true of every dev considering where to build their career, every hobbyist, every open source enthusiast, every vendor, etc. In the end, no matter which fork wins, everyone will be worse off: the community will be smaller and more splintered.

So we opted to ask HashiCorp do the right thing first. If they choose to do the right thing, we can avoid a fork, and avoid splintering the community. We still think that's the best option. But if that doesn't work, then a foundation + fork it is.

Re:The OpenTF Manifesto by linuxdude314 on Tuesday August 15, 2023 @01:01PM (#37136355)
I doubt that’s what would happen if they could afford a license from Hashicorp.

Avoiding proprietary licenses has its place but if you aren’t using terraform to build a product this really shouldn’t impact you much.

Re:The OpenTF Manifesto by aldarisbm on Tuesday August 15, 2023 @03:26PM (#37138203)
Shouldnt impact you much. _yet_
Re:The OpenTF Manifesto by i_am_jl on Tuesday August 15, 2023 @01:21PM (#37136666)

  Imagine a future CTO trying to pick the IaC tools for their company. They see Terraform as an option, but then learn there are multiple forks, licensing questions, and a big battle happening in the community. What do they do?
I truly believe that a CTO who sees Terraform as an option and who isn't scared off by the BSL, but then has all of these other concerns, exists only in fantasy.
Re:The OpenTF Manifesto by verdverm on Tuesday August 15, 2023 @01:34PM (#37136862)
Lots of people still using elastic, mongo, and redis.

What's different about this one?

Re:The OpenTF Manifesto by vosper on Tuesday August 15, 2023 @02:28PM (#37137507)
Just went with Elastic cloud after evaluating both Elasticsearch and OpenSearch. It was an easy choice to stick with the incumbent/creator that I was familiar with. No complaints so far.
Re:The OpenTF Manifesto by verdverm on Tuesday August 15, 2023 @02:44PM (#37137666)
We just went back to TF after giving Pulumi a try. Prefer declarative syntax for infra and more abuse of Yaml ("fn::..." here) is not what I'm after.

We are working on wrapping TF in CUE since you can CUE->JSON->TF

https://github.com/hofstadter-io/cuelm

Many more CUE experiments are going on in the devops space

Re:The OpenTF Manifesto by AaronFriel on Tuesday August 15, 2023 @05:38PM (#37139603)
Pulumi has a few languages other than YAML and Pulumi is declarative[1], and the programs you write are only as complex as you want them to be. This python program declares an S3 bucket and declares ten objects to exist in it.

    from pulumi_aws import s3

    bucket = s3.Bucket('bucket')

    for i in range(10):
        s3.BucketObject(
            f'object-{i}',
            s3.BucketObjectArgs(
                bucket=bucket.id,
                key=str(i),
            )
        )

Even so, Pulumi YAML has a "compiler" option, so if you want to write CUE or jsonnet[1], or other[2] languages, it definitely supports that.

Disclaimer: I led the YAML project and added the compiler feature at the request of some folks internally looking for CUE support :)

[1] https://www.pulumi.com/blog/pulumi-is-imperative-declarative...

[2] https://www.pulumi.com/blog/extending-pulumi-languages-with-...

[3] https://leebriggs.co.uk/blog/2022/05/04/deploying-kubernetes...

Re:The OpenTF Manifesto by verdverm on Tuesday August 15, 2023 @06:11PM (#37139938)
I'm aware of the SDKs, but we don't want them because they are an imperative interface, no matter how you want to spin it as "declarative". I have access to all the imperative constructs in the underlying language and can create conditional execution without restriction.

Even if I use the Yaml compiler for CUE (which we did) I still have to write `fn::` strings as keys, which is ugly and not the direction our industry should go. Let's stop putting imperative constructs into string, let's use a better language for configuration, something purpose built, not an SDK in an imperative language. These "fn::" strings are just bringing imperative constructs back into what could have been an actual declarative interface. Note, Pulumi is not alone here, there are lots of people hacking Yaml because they don't know what else there is to do. CEL making it's way to k8s is another specific example.

This cannot be the state-of-art in ops, we can do much better, but I get that Pulumi is trying to reach a different set of users than devops and will end up with different choices and tradeoffs

(I maintain https://cuetorials.com and am very active in the CUE community)

Re:The OpenTF Manifesto by lmm on Tuesday August 15, 2023 @08:49PM (#37141372)
An imperative for loop is somehow declarative now? Lol.
Re:The OpenTF Manifesto by jjnoakes on Tuesday August 15, 2023 @09:15PM (#37141582)
This seems extremely dismissive and shallow.

The imperative part of that code appears to be analogous to templating. The actual work done under the covers is not imperative, but is based on the difference between the result of the template execution and the current state of the system. That's what makes it declarative.

Re:The OpenTF Manifesto by verdverm on Tuesday August 15, 2023 @09:43PM (#37141782)
It really depends on the interaction between the user's Pulumi script and the Pulumi engine.

If there is more than one back and forth, you become declarative, even if you imperatively generate a "declarative" intermediate representation (not really sure what state file at a point in time could ever be imperative), you then would get back some data from the engine, then make choices about what to send off to the engine in the next request.

It's important to understand that with Pulumi, you can end up in either situation. You have to be careful to not become imperative overall is probably the better way to consider this.

https://www.pulumi.com/docs/languages-sdks/javascript/#entry...

Another way this can break down is if the user writes code to call the same APIs in the middle of a Pulumi script. I meant to try this myself to verify it works, but I would assume that Pulumi is not stopping me from doing something like this.

Re:The OpenTF Manifesto by jjnoakes on Tuesday August 15, 2023 @09:52PM (#37141853)
In general maybe, but in the specific context above, I think calling that loop declarative is accurate, and laughing at that classification is a poor response rooted in a deep misunderstanding.
Re:The OpenTF Manifesto by verdverm on Tuesday August 15, 2023 @10:22PM (#37142137)

    import pulumi
    from pulumi_gcp import storage

    bucket = "hof-io--develop-internal"
    name = "pulumi/hack/condition.txt"

    cond = False
    msg = "running"
    cnt = 0
    while not cond:
        cnt += 1
        key = storage.get_bucket_object_content(name=name, bucket=bucket)
        print(cnt, key.content)
        if key.content == "exit":
            msg = "hallo!"
            break

    pulumi.export('msg', msg)
    pulumi.export('cnt', cnt)
---

        769 exit
        770 exit
        771 exit
        772 exit
        773 exit
        774 exit
        775 exit

    Outputs:
        cnt: 775
        msg: "hallo!"

    Resources:
        + 1 to create

    info: There are no resources in your stack (other than the stack resource).

    Do you want to perform this update?  [Use arrows to move, type to filter]
      yes
    > no
      details
----

Of note, all but the last exit had a newline, until I `echo -n` the file I copied up

---

ooo...

        348 what?!?!
        349 what?!?!
        350 what?!?!
        351 what?!?!
        352 what?!?!
        353 what?!?!
        354 what?!?!
        355 what?!?!
        356 what?!?!
        357 what?!?!
        358 what?!?!
        359 exit

    Outputs:
        cnt: 359
        msg: "hallo!"

    Resources:
        + 1 created

    Duration: 27s
---

I uploaded a different file while waiting to be asked to continue, and then proceeded to get different outputs

Note, while I can get the contents of a bucket in TF, I cannot build a loop around it as I have above

https://registry.terraform.io/providers/hashicorp/aws/latest...

TF might be susceptible to the same file contents manipulation between plan & apply as well, but then again, you can save a plan to a file and then run it later, so maybe not? Another experiment seems to be in order

Re:The OpenTF Manifesto by AaronFriel on Wednesday August 16, 2023 @01:09AM (#37143063)
I think this is an advantage of Pulumi, here are two use cases:

1. Creating a resource where created is not the same as ready. This is extraordinarily common with compute resources (a virtual machine, a container, an HTTP server, a process) where attempting to create follow-up resources can result in costly retry-back-off loops. Even when creating Kubernetes resources, Pulumi will stand up an internet-connected deployment more quickly than many other tools because you can ensure the image is published before a pod references it, the pod is up before a service references it, and so on. (The Kubernetes provider bakes some of these awaits in by default.)

2. Resources graphs that are dynamic, reflecting external data sources at the moment of creation. Whether you want to write a Kubernetes operator, synchronize an LDAP directory to a SaaS product, or one of my favorite examples. When I set up demos, I often configure the authorized public IPs dynamically:

    import * as publicIp from 'public-ip';

    new someProvider.Kubernetes.Cluster('cluster',
      {
        apiServerAccessProfile: {
          authorizedIPRanges: [await publicIp.v4()],
          enablePrivateCluster: false,
        },
      }
Re:The OpenTF Manifesto by verdverm on Wednesday August 16, 2023 @01:34PM (#37150580)
Of course you think it is an advantage, you work for Pulumi

I'm telling you this is not how a potential user sees the same situation, that it is a disadvantage and was one of the reasons we are not making the switch.

This example above is exactly the kind of code we don't want in ops, it depends on the user environment and physical location at the time they run the command, bad practice. Thanks for an extra talking point though

Re:The OpenTF Manifesto by jjnoakes on Wednesday August 16, 2023 @12:38PM (#37149674)
The claim above isn't "imperative is impossible".
Re:The OpenTF Manifesto by verdverm on Wednesday August 16, 2023 @01:29PM (#37150502)
The claim above is that Pulumi uses an imperative interface and that it is quite easy to slip past the declarative guardrails, so in most cases Pulumi is imperative, not declarative

The claim I keep seeing from Pulumi folks is that Pulumi is declarative, which is is not, as shown in multiple posts by many people. Please stop calling it such, it demonstrates dishonesty towards users

Re:The OpenTF Manifesto by jjnoakes on Wednesday August 16, 2023 @04:13PM (#37153067)
The claim above was that a for loop implied that the code couldn't be declarative.

> Please stop calling it such

I'm not claiming it is always declarative, I'm only claiming that a declarative example above can contain a for loop, and that laughing at that is the wrong response. That's it.

Re:The OpenTF Manifesto by verdverm on Wednesday August 16, 2023 @09:44PM (#37156149)
> Please stop calling it such

That was more me yelling into the void or larger thread than at anything specific you said, sorry :]

Re:The OpenTF Manifesto by verdverm on Tuesday August 15, 2023 @10:07PM (#37141955)
I was just wondering what stops me from reading and writing to a cloud bucket like an infinite tape?

https://www.pulumi.com/registry/packages/gcp/api-docs/storag...

Re:The OpenTF Manifesto by lmm on Tuesday August 15, 2023 @10:00PM (#37141908)
> This seems extremely dismissive and shallow.

When someone tries to make a sophisticated argument that up is down and white is black, dismissive and shallow is the right response.

> The actual work done under the covers is not imperative

Having a declarative layer somewhere in the stack doesn't make something declarative, if that's not the layer you actually use to work on and reason about the system. See the famous "the C language is purely functional" post.

Re:The OpenTF Manifesto by Pet_Ant on Wednesday August 16, 2023 @03:04AM (#37143679)
Re:The OpenTF Manifesto by jjnoakes on Wednesday August 16, 2023 @12:39PM (#37149683)
> When someone tries to make a sophisticated argument that up is down and white is black

This is where the deep misunderstanding is coming from.

Re:The OpenTF Manifesto by verdverm on Tuesday August 15, 2023 @09:02PM (#37141471)
you can have loops and still be declarative, CUE has loops, though they are considered comprehensions more technically, but there is no assignment or stack in CUE

One of the interesting aspects of CUE is that it gives us many of the programming constructs we are used to, but remains Turing incomplete, so no general recursion or user defined functions. There is a scripting layer where you can get more real world stuff done too

The CUE language is super interesting, has a very unique take on things and comes from the same heritage as Go, containers, and Kubernetes

https://cuelang.org | https://cuetorials.com

Re:The OpenTF Manifesto by i_am_jl on Tuesday August 15, 2023 @03:38PM (#37138337)

  You may make production use of the Licensed Work, provided such use does not include offering the Licensed Work to third parties on a hosted or embedded basis which is competitive with HashiCorp's products.
If I build a product with HashiCorp-owned BSL'd code, then HashiCorp releases a product that competes with mine, then my license is void.
Re:The OpenTF Manifesto by verdverm on Tuesday August 15, 2023 @04:16PM (#37138824)
My understanding is that the aforementioned companies' licenses are to the same effect, so what is the difference?
Re:The OpenTF Manifesto by i_am_jl on Tuesday August 15, 2023 @05:10PM (#37139314)
Redis is 3-clause BSD, BSD does not have a "your license is void if you sell a product that competes with us" clause. Redis does have enterprise products that are licensed in a manner similar to BSL, but Redis itself is not.

MongoDB and Elastic are SSPL. SSPL approaches the problem like the AGPL; it compels licensees who sell a service derived from the software to make available under the SSPL the source of all supporting tooling and software so that a user could spin up their own version of the service.

There's an argument to be made that SSPL is de facto "you can't compete with us" since it would be more challenging to make a competitive SaaS offering if your whole stack is source available. I don't disagree. However, as distasteful as SSPL is, at least it doesn't grant licensing to a product conditionally on the unknowable future product offerings of HashiCorp.

Re:The OpenTF Manifesto by verdverm on Tuesday August 15, 2023 @05:14PM (#37139353)
thanks for the explanation, my understanding is that they are all after limiting competition in various ways, while still trying to maintain the mantle of open source

We are certainly in interesting times around the monetization / financial sustainability of open source

Re:The OpenTF Manifesto by bit_flipper on Tuesday August 15, 2023 @05:16PM (#37139379)
SSPL has no provision even close to the reach of the "anti-competition" clause Hashicorp is using. While SSPL is not considered open source, it isn't that far off from the AGPL. The difference between SSPL and AGPL is that SSPL (1) is in effect regardless of modification of the service and (2) extends copy left virality to all programs which support running the service, including those that interact with the software over a network.

MongoDB, Elastic, etc. cannot stop you from running a competitor based on the terms of their licenses, they just ask that you publish the source code for whatever service you're running in its entirety (I acknowledge there are disagreements about how far "entirety" extends). The clause in Hashicorp's license actually revokes the right to use their software at all if you're a direct competitor.

OK, no one is going to build an open source competitor to Elastic or MongoDB because then you have no moat and your business will probably fail, I get it, but it's still possible to do without repercussion. It's not like the AGPL is that far off in terms of limitation, either, which is why you don't see many copyleft services run by large corporations unless they've been dual-licensed.

Re:The OpenTF Manifesto by Spivak on Tuesday August 15, 2023 @03:56PM (#37138570)
I don't get this one, you pick OpenTerraform and get on with your life. It's the same with picking OpenSearch over Elastic. I can use the proprietary version that locks me into a single profit-seeking vendor and doesn't have community backing or the one run by a foundation made up of companies that use and are heavily invested in Terraform.
Re:The OpenTF Manifesto by eraser215 on Thursday August 17, 2023 @10:19AM (#37162007)
How dare a vendor come up with an idea, pay people to execute on that idea, and the gasp try to make money from it? Outrageous!
Re:The OpenTF Manifesto by Spivak on Monday August 21, 2023 @10:22AM (#37209939)
How dare a vendor come up with an idea, pay people to execute on it, give it away for free to the world, acquire users and soak in all the community contributions from people who thought they were using and contributing to a public good, try and fail to indirectly monetize a hosted version because other people were better at it than them, then rug-pull out from under everyone and use copyright/government-stick to kill their competition because they can't compete on even terms.

Then a group of people who are users of idea and actually making money off it with value-adds step up to maintain it as a community project ensuring that it stays open for everyone -- yeah those guys are the assholes. Terraform would have went nowhere if it wasn't OSS and Terraform would be nothing without its outside contributions that make up far more than the code of Terraform core itself. There's a trail of bodies to prove it.

And you should love this, projects that are stewarded by its own users are incentivized to make it the best it can be instead of rejecting contributions because it competes with their cloud offering [1]

[1] https://github.com/hashicorp/terraform/issues/9556

Re:The OpenTF Manifesto by hdaz0017 on Tuesday August 15, 2023 @04:21PM (#37138862)
Hopefully it's not down to CTOs to be picking tools for their company but a process within DevOps/Engineering teams etc.

Does anyone else see this as the Nagios Effect all over again, there must be lots to learn from history?

Re:The OpenTF Manifesto by Atotalnoob on Tuesday August 15, 2023 @06:11PM (#37139945)
What is the nagios effect?
Re:The OpenTF Manifesto by pxc on Tuesday August 15, 2023 @10:44PM (#37142292)
I didn't know either, so I did some Googling and found an old announcement[1] from 2009:

> A group of leading Nagios protagonists including members of the Nagios Community Advisory board and creators of multiple Nagios Addons have launched Icinga – a fork of Nagios, the prevalent open source monitoring system. This independent project [is based upon a] broader developer community. [...] Icinga takes all the great features of Nagios and combines it with the feature requests and patches of the user community.

It also looks like in 2014, Nagios centralized and appropriated a domain name and website used for hosting Nagios plugins, away from the community (its plugin developers)[2]:

> In the past, the domain "nagios-plugins.org" pointed to a server maintained by us, the Nagios Plugins Development Team. The domain itself had been transferred to Nagios Enterprises a few years ago, but we had an agreement that the project would continue to be independently run by the actual plugin maintainers.¹ Yesterday, the DNS records were modified to point to web space controlled by Nagios Enterprises instead. This change was done without prior notice.

> To make things worse, large parts of our web site were copied and are now served (with slight modifications²) by <http://nagios-plugins.org/>. Again, this was done without contacting us, and without our permission.

> This means we cannot use the name "Nagios Plugins" any longer.

There's some previous discussion of those controversies on HN here: https://news.ycombinator.com/item?id=9452013

From that article[3]:

> [Icinga developer]: "Six months before the fork, there was a bit of unrest among Nagios' extension developers [...] Community patches went unapplied for a long time[.]"

> [...]

> Two years ago, more or less when the split happened, [Nagios author] was having problems resolving [trademark] issues with a company called "Netways".

I'm still not sure what the effect is supposed to be tbh.

--

1: https://icinga.com/blog/2009/05/06/announcing-icinga/

2: https://www.monitoring-plugins.org/archive/devel/2014-Januar...

3: https://web.archive.org/web/20160314090137/http://www.freeso...

Re:The OpenTF Manifesto by hdaz0017 on Wednesday August 16, 2023 @01:49PM (#37150854)
Nagios used to be only Open Source then they created the Enterprise version and left the open source core version lagging behind, it was forked a billion times or more :) creating the Nagios Effect. A lot of monitoring software / companies then removed / replaced the core of Nagios from their products.
Re:The OpenTF Manifesto by denvrede on Wednesday August 16, 2023 @10:27AM (#37147559)
The guys at Pulumi must be having a field day right now. It's exactly how you describe it for us. We're long overdue with an upgrade of our Terraform config from pre v1.0. We have to most likely re-write a big part of our HCL code, so why not try a competitor?

With Vault however that's another story, I've yet to find another secrets management system that has a tight integration with Kubernetes, AWS and supports providers for things like Postgresql to have ephemeral database credentials.

Re:The OpenTF Manifesto by yellowapple on Wednesday August 16, 2023 @11:28PM (#37156802)
Someone else posted a list of Vault alternatives, multiple of which (AFAICT) check your boxes: https://news.ycombinator.com/item?id=37151218
Re:The OpenTF Manifesto by denvrede on Thursday August 17, 2023 @01:44AM (#37157696)
Most of them do the first two: integration with Kubernetes and AWS. Unfortunately short lived DB creds is not in any of those listed.
Re:The OpenTF Manifesto by fishnchips on Tuesday August 15, 2023 @11:51AM (#37135452)
Marcin here, co-founder of Spacelift.

Even though I strongly believe the OpenTF fork could open up incredible possibilities for the community (I could go on and on about it), it is an equivalent of a civil war. It doesn't serve the community and our only interest is in the continued strength of the community that we continue to build for.

Based on my immense respect for what's been built under Hashi's umbrella I'd rather see a change of mind, and an opportunity to honor our pledge of resources (5 FTEs for 5 years) to the common rather than partisan cause.

Re:The OpenTF Manifesto by busterarm on Tuesday August 15, 2023 @12:40PM (#37136092)
Hashicorp isn't going to budge here. The same argument that you've made about Terraform being the underpinnings and needing to be open-sourced can be applied to their other important products like Vault, Consul and Nomad as well. The ecosystem of those three is plainly a direct competitor to Kubernetes which is open-source.

There's really no move for them to make here. It's unfortunate.

Re:The OpenTF Manifesto by fishpen0 on Tuesday August 15, 2023 @02:29PM (#37137521)
Tons of organizations run vault and consul as part of their k8s ecosystem so they don't directly compete. The vault CSI driver might be the single most installed CSI driver across all the orgs I've worked for
Re:The OpenTF Manifesto by busterarm on Tuesday August 15, 2023 @02:54PM (#37137784)
You're completely missing it.

If you are running Nomad as your orchestrator, because of the tight integrations you are almost certainly running vault for secrets and consul for service discovery/service mesh. The ecosystem of the three is the competitor to K8s.

s/ someone who runs both ecosystems at scale.

Re:The OpenTF Manifesto by twunde on Tuesday August 15, 2023 @03:16PM (#37138061)
While the Nomad stack is a direct competitor to k8s, Consul and Vault are both heavily used alongside k8s. In fact, Consul had features that were only for k8s the last time I checked
Re:The OpenTF Manifesto by busterarm on Tuesday August 15, 2023 @03:17PM (#37138074)
While these facts are true, that's totally not the fucking point and has nothing to do with the argument. I say again.

You can have software that both supports and competes with the k8s ecosystem. That's even the same type of problem all of these companies have with Hashicorp software now under the BSL.

Gruntwork builds tools that everyone using Terraform uses but they also offer services that Hashicorp would prefer that you pay them for instead.

You telling me "but people running K8s also use vault/consul" is like telling me that Gruntwork makes terragrunt which terraform users use. It doesn't mean that Hashicorp doesn't view them as a threat.

Re:The OpenTF Manifesto by hdaz0017 on Tuesday August 15, 2023 @04:08PM (#37138724)
There is a big difference though Terraform is the out and out winner in its market.

All their other products are at best small x% share of a crowded market or dominated by another product.

Re:The OpenTF Manifesto by ghshephard on Tuesday August 15, 2023 @05:32PM (#37139538)
Genuinely curious - other than Vault - what other product is there for secret management in the cloud infrastructure space. I get that CyberArk Conjur is big in the enterprise space, but I thought cloud users, even with k8s, mostly went with vault.
Re:The OpenTF Manifesto by candiddevmike on Tuesday August 15, 2023 @06:17PM (#37140000)
Most companies are probably using the secret manager provided by their cloud platform instead of Vault, especially after they tried to buy Vault.
Re:The OpenTF Manifesto by Alupis on Tuesday August 15, 2023 @07:23PM (#37140655)
Which is more than likely Vault + new carpets and paint.
Re:The OpenTF Manifesto by xyzzy_plugh on Tuesday August 15, 2023 @10:50PM (#37142325)
No, it's astronomically unlikely to be Vault at all.
Re:The OpenTF Manifesto by Alupis on Wednesday August 16, 2023 @03:52PM (#37152737)
It's far more likely to be Vault as a base, actually. The MPL would allow someone like Amazon to use Vault's source as a base, and so long as the core source wasn't modified Amazon would be under no obligation to make their modifications public.

The MPL is a lot more "business" friendly than the GPL.

The very fact that HashiCorp is changing their license and restricting use clearly indicates they see this threat as reality. Amazon/Microsoft/Google/Whoever using HashiCorp's work/effort but keeping all the money to themselves.

Re:The OpenTF Manifesto by candiddevmike on Wednesday August 16, 2023 @07:29PM (#37155253)
I don't know what you've heard but none of the cloud providers use HashiCorp Vault as the foundation for their secrets manager.
Re:The OpenTF Manifesto by xyzzy_plugh on Wednesday August 16, 2023 @10:10PM (#37156312)
Yes, they could, but no major cloud provider does. They're all bespoke. Read their white papers and it'll make more sense.
Re:The OpenTF Manifesto by oneplane on Tuesday August 15, 2023 @06:27PM (#37140097)
You generally only run Vault if you don't run on a public cloud.
Re:The OpenTF Manifesto by Alupis on Tuesday August 15, 2023 @08:06PM (#37141025)
There's a lot more reasons to run Vault than those.

Having a standardized way to "do secrets" for any team, any service, any app within the organization is very nice. Becoming cloud-agnostic for your secrets (connecting your local Vault with the cloud provider's vault) is another great benefit. Automatic secret rotation is also another great benefit. Secret versioning and auditing... etc.

It's not just "can't have this secret in VCS or viewable via kubectl".

Re:The OpenTF Manifesto by oneplane on Wednesday August 16, 2023 @08:46AM (#37146238)
> It's not just "can't have this secret in VCS or viewable via kubectl".

That is exactly what it is.

You seem to misunderstand (and thus downvote?) the statement I made. I'm not saying "haha vault bad", I'm answering "what other product" (from ghshephard) with the reality of today.

This has nothing to do with what Vault is or isn't, but just with the concept of storing secrets in a uniform way in clouds for use with cloud workloads what is being used right now.

Re:The OpenTF Manifesto by Alupis on Wednesday August 16, 2023 @03:20PM (#37152241)
I did not downvote you, no. Downvoting because we disagree isn't how that's supposed to work, even though some use it that way.

Regardless, the use of Vault is not exclusive to cloud environments.

All of the listed features of Vault have benefits within larger organizations even if they don't use the "cloud" and deploy monolithic applications.

Most frameworks have built in ways to fetch secrets/config from Vault, making it an easy standardized way to do things across all of your applications/teams.

It doesn't mean you need to use it, of course, but it has a lot of perks for many different situations.

Re:The OpenTF Manifesto by oneplane on Wednesday August 16, 2023 @03:56PM (#37152805)
I totally agree that Vault is more than a glorified password manager (because that's what most Clouds have in their implementation of a secrets store), but the thing is, everywhere I go, I don't see people use Vault, I see them use whatever AWS/Google/Azure happens to have (and often badly).

I'm not sure if that's even what ghshephard meant when he was curious for 'products', since technically all those cloud-integrated services aren't really stand-alone products for that matter.

In AWS for example, with or without EKS (and then something like External Secrets Operator in the EKS case), it's all just AWS Secrets Manager and sometimes Parameter Store. In a few cases people do manual encryption (using KMS), but in no case was HashiCorp Vault used.

Often, it's even worse: no secrets management at all. Stuff just gets pumped into environment variables (more often than not they get committed as .env files to Git), and there's just no drive to change that, even when a business policy is in place. Some even 'work around' this by storing secrets in password managers like 1Password and LastPass so they can check the compliance box without actually protecting the secrets (since they also live in plain text in VCS and at runtime in the environment).

In terms of 'products', I'd say Vault and the cloud ones don't really compare, but reality is depressing and secrets are often not as secret as the name implies. From a developer perspective, they might compare them because they desire the secrets to be injected into the environment either way, and as such the source doesn't matter much. I'm not sure if we should see that as a feature or a bug.

Re:The OpenTF Manifesto by ghshephard on Thursday August 17, 2023 @03:54PM (#37167067)
We use vault as a framework that associates authentication with secret engines via a policy framework. The Secret Engines could be AWS, PostGres Database, PKI, SSH Certificate Signer, Key value stores, etc... and the Authentication Framework might be LDAP, OKTA, or plain tokens. The Policy framework is pretty dynamic and has many thousands of possible policies mapping various authenticated entities to various authority (read, list, write, etc...) to various secret engines. Combine that with the syntactic niceties of template-rendering integration with the chosen secret-store, and maybe some clever stuff around single-use token wrapping - and I think of all of those features as belonging to a single product.

I'm relatively new to this field - and see tons of Vault at colleagues companies, and have friends who run/support Conjur (Enterprise more than cloud). Those are the only two secret-management framework/products I'd hear of - so was interesting in knowing what else had mindshare.

Re:The OpenTF Manifesto by oneplane on Saturday August 19, 2023 @03:42PM (#37192350)
I wonder if some of this also depends on how the secrets are consumed (and created), I'd imagine that if you store things like an API key and secret for a third party API, someone needs to 'enter' that data at some point in time and then set an ACL to allow a person or system to then consume it.

But if you have two programs what exchange secrets between multiple instances of each other, (one can do CRUD, the other only Read), you'd have much more interaction. Same as with a system creating secrets and a human reading it.

As for where it would make no sense at all: automated workload identities where you get time-limited temporary credentials that represent a role; most public clouds have some sort of link-local API, an injection method or mount method to provide ever-rotating secrets which gets picked up by the client SDK automatically. If you are using something like AWS, you'd be able to consume hundreds of services without ever persisting a secret anywhere.

This is also where my 'cloud' (and K8s) remarks are based on; when your workload and your resources speak the same authn/authz with a centrally coordinated policy system, there really isn't much value in adding something in the middle of that, and as such you don't see a lot of Vault and Vault-like implementations.

That said, as soon as you add something disconnected like local virtual machines, on-prem stuff etc. where authentication has historically been extremely bad and unless you brought a proper Kerberos setup you're screwed beyond mitigation. That's where Vault (when it came out) delivered a lot of value. It's probably also why we see AWS, IBM, GCP, Azure, in the same list with Vault and CyberArk. I'm surprised VMware doesn't have anything yet, but perhaps they recognise they lost this one already.

Re:The OpenTF Manifesto by hdaz0017 on Wednesday August 16, 2023 @02:11PM (#37151218)
HashiCorp Vault - is big in Secrets management but its not the Terraform of Secrets management it's just too easy to make another product.

AWS Secrets Manager

AWS Systems Manager Parameter Store

AWS KMS

Google Cloud KMS - Cloud Key Management System

Azure Key Vault

confidant - https://lyft.github.io/confidant

keywhiz - https://github.com/square/keywhiz

knox - https://github.com/pinterest/knox

strongboxsafe - https://strongboxsafe.com

conjur - https://www.conjur.org

+ many more

Ansible Vault - probably not competing directly

I am pretty sure companies like netflix, facebook, uber, tesla and others are probably using their own in house creations.

Re:The OpenTF Manifesto by tedivm on Tuesday August 15, 2023 @11:58AM (#37135553)
I really appreciate that, and I do think it's right of you to at least make the attempt.

That being said, I don't expect this attempt to work and I fully believe that a fork is going to be inevitable. I also think a fork is an amazing opportunity to standardize the language and prioritize the features developers want.

It isn't just about the license, but the way that Hashicorp has maintained the Terraform project. The github insights show that they don't have nearly as many people working on it as I would expect, and most of them are split into also working on Terraform Cloud. At the same time they don't work with the community that well- there are open issues and pull requests that just get ignored as Hashicorp clearly doesn't see value in open source contributors. This isn't just a Terraform issue either- my company had to move off of nomad due to the lack of development and support (as well as broken features).

I have strong concerns about the future of these projects in general beyond just the licensing. An open foundation that had multiple companies involved would by definition need to find a way for those people to collaborate together, and once they do that it makes it easier for them to invite community collaboration. So while I do appreciate that it is a drastic step, I think it's one that would also be far better for the ecosystem and project as a whole.

That said, maybe this is the wake up call hashicorp needs to fix these problems. If you provide five FTEs that basically doubles the size of their Terraform development team (they have more people working on it than five, but those people are split into other projects), and once they start working with other groups maybe they'll work with the community more as well. I'm not holding my breath though.

Re:The OpenTF Manifesto by lamontcg on Tuesday August 15, 2023 @12:58PM (#37136295)
Smells like the end of Chef. Management doesn't understand how much it takes to maintain the open source project and is just pouring resources into sales and marketing and products that they can charge for, and don't see how that erodes goodwill and the technological foundation of the company.
Re:The OpenTF Manifesto by miah_ on Tuesday August 15, 2023 @02:25PM (#37137467)
I also saw that parallel with Chef. I think its the story of all VC funded software that attempts to be "Open Source". For them, Open Source means "You can read the source code, and potentially fix a bug", for us, it means community, transparency, and fixing bugs beyond those your paying customer has.

I looked at github /chef/chef and github /inspec/inspec and its the same as it was shortly after I left. The only changes are from the one person who carried over after the sale to Progress, and the contracting team out of India, with dozens are unanswered queries and pull requests from the community.

What really ruffles my feathers was when they had us define oss-practices (https://github.com/chef/chef-oss-practices), clearly nobody outside our small team read (or understood) those words and goals. It feels like it was work to make us look better in OSS in order to bolster the company sale.

Re:The OpenTF Manifesto by lamontcg on Tuesday August 15, 2023 @03:14PM (#37138031)
There was a whole lot of community window dressing going on. I still wonder if they weren't trying to ship maintenance of the open source code off onto the community thinking that if all that worked appeared (or thinking that it was actually going on--believing their own bullshit about how involved the community was) that they could just leach that work.

There's probably some manager at Hashi right now trying to argue that they should offload TF maintenance entirely onto the community and they should pivot to hosting services and consulting and making money off of all that free work.

Re:The OpenTF Manifesto by hdaz0017 on Tuesday August 15, 2023 @03:55PM (#37138547)
chef said, did and tried some really dumb stuff and lots of it failed for obvious reasons, it's like docker took a chunk of their playbook and their business went the same way.

It's all good and "fun" on the IPO up.

(in x months we will rewrite the whole universe).

Re:The OpenTF Manifesto by FrenchTouch42 on Tuesday August 15, 2023 @04:31PM (#37138977)
Thank you Miah for all your contributions (beyond InSpec)
Re:The OpenTF Manifesto by rattus on Monday August 21, 2023 @05:05PM (#37215359)
Cygnus made it work. Long way from those origins to the redhat or hashicorp of today though.
Re:The OpenTF Manifesto by PeterZaitsev on Tuesday August 15, 2023 @01:37PM (#37136897)
Hi Marcin,

I recognize you're in interesting position in Spacelift. Per your recent analyses you may not be impacted and in this case you probably do not want to pissoff Hashicorp folks :)

In reality though force better be responded with force and showing Hashcorp what what was Terraform will be successful as Open Source project with or without them is best way to get them to reconsider.

Re:The OpenTF Manifesto by RcouF1uZ4gsC on Tuesday August 15, 2023 @04:46PM (#37139089)
> it is an equivalent of a civil war.

Why. It is open source. A fork should be no big deal, and definitely not a “civil war”. I think the community should be quicker to fork open source projects that are not serving the needs of the community.

The corporations are trying to have the benefits of open source without the responsibility. Forking is a normal, acceptable part of open source and we should normalize it.

Re:The OpenTF Manifesto by enneff on Tuesday August 15, 2023 @05:39PM (#37139606)
What would it mean to “normalise” forking? The costs of maintaining a fork are significant, and if one group of programmers are being funded to work on the project then it can be very difficult to fork a project in any meaningful way without significant resources behind it.

Also IIUC most of the parties in this conversation are corporations. They’re all trying to enjoy the benefits of open source development for a variety of reasons.

Re:The OpenTF Manifesto by lmm on Tuesday August 15, 2023 @08:41PM (#37141302)
> The costs of maintaining a fork are significant

Why? If we have tooling or workflows that assume forks don't happen, maybe we can fix those. Forking should be cheap, easy and frequent.

Re:The OpenTF Manifesto by waynesonfire on Wednesday August 16, 2023 @12:57AM (#37143019)
Right below your comment was another comment that started with,

"The truth is that a fork hurts everyone."

So, which is it?

Re:The OpenTF Manifesto by lmm on Wednesday August 16, 2023 @02:13AM (#37143419)
Currently forks are painful, because they aren't normalized i.e. our tools and workflows don't expect them. I'm saying rather than discouraging forks we should adapt our tools and workflows to expect them.
Re:The OpenTF Manifesto by ericb on Wednesday August 16, 2023 @05:19AM (#37144567)
Sure, tooling could be better.

But the real work is all the hard work that goes into a fork. I've watched open forks die all the time--all it takes is no one to step up and do/pay for the work, which is basically the default, because it is in everyone's interest if someone else is the one to do that.

I think that's really the crux of the problem--there are plenty of folks willing to maintain software for money, and a whole lot of people who'd rather it not cost money and if it does, not their money.

If the tooling is better, who is going to maintain this?

Re:The OpenTF Manifesto by throwaway2037 on Wednesday August 16, 2023 @05:27AM (#37144631)
I assume OP meant cost == community. That is the hard part of a fork.
Re:The OpenTF Manifesto by kragen on Tuesday August 15, 2023 @05:45PM (#37139670)
it's not war

hashicorp has decided they don't want to contribute to open source any more

they're totally within their rights to do so, and it doesn't harm anybody; it's not the equivalent of going around blowing up buildings, raping women, and napalming children. at most we can wish they had continued doing the beneficial things they were previously doing

maybe they'll change their minds, as you say, but that's no reason for the community to sit around twiddling its thumbs hoping for such a change. what's important now is that the people who are still willing to cooperate can do so successfully, and that's what opentf is about

that's even more obviously not the equivalent of going around blowing up buildings, raping women, and napalming children. it's very much the opposite, in fact

it's unclear to me which of the parties you intend to accuse of doing the moral equivalent of burning innocent people alive en masse, but either way, maybe you should think about walking back that rhetoric a bit

Re:The OpenTF Manifesto by fishnchips on Tuesday August 15, 2023 @06:10PM (#37139929)
I apologize if I appeared to downplay the horror of actual war. I'm lucky to never have experienced it first-hand.

I mainly wanted to focus on the "civil" than "war" aspect of the possible schism.

Re:The OpenTF Manifesto by anyoneamous on Tuesday August 15, 2023 @06:31PM (#37140129)
FWIW there are still people (dozens of us!) who don't feel the need to be artificially offended by a bit of hyperbole on an internet forum.
Re:The OpenTF Manifesto by bifrost on Tuesday August 15, 2023 @08:01PM (#37140996)
I concurr.
Re:The OpenTF Manifesto by kragen on Wednesday August 16, 2023 @12:28AM (#37142889)
my objection is not that mass graves, piles of mangled bodies, your close friends unexpectedly disappearing into pink mist, and terrible stenches are too sacred to be used as a metaphor for something else

my objection is that warfare involves people intentionally harming each other, and that doesn't seem to be what's going on here. it's not that war is a more extreme version of the situation; it's that it's directionally different

rather, hashicorp is struggling to not go bankrupt, so they've decided to switch to making a proprietary product instead of an open-source products; and terraform users, naturally enough, are reluctant to make their infrastructure vulnerable to a proprietary software license. hashicorp is not intentionally harming terraform users, and terraform users are not intentionally harming hashicorp

they're just not continuing their previous cooperation

Re:The OpenTF Manifesto by 0ct4via on Wednesday August 16, 2023 @12:22PM (#37149419)
Not sacred enough for you to invoke the Holocaust in a discussion about data protection legislation and the "right to be forgotten" within mere hours of this statement, however...

Such hypocrisy, attacking others for "rhetoric" that needs to be "dialled back a bit" when you're every bit as guilty of the exact. same. thing.

Re:The OpenTF Manifesto by kragen on Wednesday August 16, 2023 @01:12PM (#37150201)
please note that the comment you are replying to says the opposite of what you are implying it does

the comment you are referring to, for anyone who is interested, concerns the question of whether or not there is a higher standard of morality to which legislation can be held, or whether legislation itself is the ultimate moral authority, or whether there is in fact no objective standard of morality at all. anyone who is interested in that kind of thing can read it at https://news.ycombinator.com/item?id=37147305

Re:The OpenTF Manifesto by kragen on Tuesday August 15, 2023 @06:30PM (#37140123)
it's not a possible schism. hashicorp has clearly and unmistakably abandoned the open source community. conceivably they'll change their minds, but their communication doesn't have any ambiguity in it

i have no idea what you could possibly mean by 'focus on the "civil"'. it's good that people are being civil to one another, isn't it?

Re:The OpenTF Manifesto by tedivm on Tuesday August 15, 2023 @07:11PM (#37140515)
You're missing the point- the OpenTF group wants to mend the schism if possible, by getting Hashicorp to change their licensing back. If they immediately fork then that's not likely to happen, so they're attempting this first.

I don't think it will work, but I think it's good of them to try.

Re:The OpenTF Manifesto by kragen on Tuesday August 15, 2023 @08:33PM (#37141244)
mending the schism would be great, but we probably can't do that by pretending it doesn't exist, analogizing it to blowing children's extremities off, or analogizing acknowledging its existence to blowing children's extremities off; I'm still not sure which of the latter two was the intent of the comment I was responding to
Re:The OpenTF Manifesto by 0ct4via on Wednesday August 16, 2023 @12:22PM (#37149424)
Don't you think you should check yourself before attacking others?

> "it's unclear to me which of the parties you intend to accuse of doing the moral equivalent of burning innocent people alive en masse, but either way, maybe you should think about walking back that rhetoric a bit"

... but only earlier today were you equating a data protection law with the Holocaust... seems you could do with a bit less projection and rhetoric yourself, Kraggy.

Re:The OpenTF Manifesto by kragen on Wednesday August 16, 2023 @01:18PM (#37150291)
this is careless reasoning; rather than equating a so-called 'data protection law' with the holocaust, i said the justification others were using for that law was incorrect, because if it were correct, it would also justify the holocaust

this is not an extremely advanced form of logic, but i understand that it is not within everyone's grasp

the comment in question is https://news.ycombinator.com/item?id=37147305

octavia's other, rather astounding comment in https://news.ycombinator.com/item?id=37149528 is probably also useful context

my objection to the war rhetoric in this case is that it casts people as opponents who are not, in fact, opponents, just different parties pursuing largely independent interests. in that context playing 'let's you and him fight' seems unlikely to improve the situation

Re:The OpenTF Manifesto by 0ct4via on Monday August 21, 2023 @04:15AM (#37206712)
Why use so-called? It's literally a law concerning data protection. Unless you feel that the empirical truth somehow shouldn't be used to describe the GDPR and colloquial "right to be forgotten" aspects it entails, you're just trolling for trolling's sake -- either that, or you don't even know that you're using "so-called" improperly.

Once again you jump to mansplaining and condescension, followed by failing to even get a username correct when it's literally on your screen.

Your repeated attempts to tell people what they think, what they do or do not know, and where they live, show that every observation of you being arrogant, condescending, and disingenuous, is patently correct.

Unsure if you need a shovel to get out of that hole you've put yourself in, but you're sure backpedaling quickly, yikes.

Re:The OpenTF Manifesto by jeffchao on Tuesday August 15, 2023 @05:56PM (#37139797)
As a user and collaborator of TACOS, agreed that it could open up opportunities. Though I echo trade-offs (as in other replies) that it could start a civil war that makes it difficult for end users and collaborators -- reminds me of Python2 -> 3, Presto/Trino, and many other stories. Pledging resources is a great approach.
Re:The OpenTF Manifesto by candiddevmike on Tuesday August 15, 2023 @12:03PM (#37135613)
Speaking of nuclear options, need to get the providers to pledge/follow the fork, maybe via some kind of API incompatibility. Terraform is useless if the providers don't work with it and only the fork. Focus on disrupting the ecosystem.
Re:The OpenTF Manifesto by tedivm on Tuesday August 15, 2023 @12:06PM (#37135645)
Hashicorp left the provider frameworks under the original licenses, probably because they don't want to scare provider developers off. So for now both Terraform and a potential fork can continue sharing the same providers without issue.
Re:The OpenTF Manifesto by dividedbyzero on Tuesday August 15, 2023 @12:50PM (#37136204)
Are they even able to unilaterally relicense third party providers?
Re:The OpenTF Manifesto by throwaway1101z on Tuesday August 15, 2023 @01:06PM (#37136444)
Indirectly, by changing the license for the SDK.
Re:The OpenTF Manifesto by bickfordb on Tuesday August 15, 2023 @03:40PM (#37138356)
This is the interesting part of all of this. The meat of Terraform is in its provider ecosystem. Anyone can make a new frontend (or even fork the existing one?), get rid of all the warts, add the missing encryption features gated under enterprise and have a much better tool.
Re:The OpenTF Manifesto by PeterZaitsev on Tuesday August 15, 2023 @01:07PM (#37136447)
I totally agree. I do not think pleading with Hashicorp to reconsider will result in changing back the license

Doing the Fork and showing it IS sustainable and has broad community support can encourage Hashicorp to make concessions.

After taking this unilateral hostile step I do not think Hashicorp deserves the community trust and what industry needs is "Foundation Governed" Terraform like solution, whatever name this solution will have.

You can see example in Confluenct which builds proprietary solutions around Kafka, where Kafka itself is Apache project.

Re:The OpenTF Manifesto by nprateem on Tuesday August 15, 2023 @02:27PM (#37137485)
It was never a public good.
Re:The OpenTF Manifesto by dbingham on Tuesday August 15, 2023 @02:43PM (#37137650)
> In economics, a public good (also referred to as a social good or collective good)[1] is a good that is both non-excludable and non-rivalrous. For such goods, users cannot be barred from accessing or using them for failing to pay for them. Also, use by one person neither prevents access of other people nor does it reduce availability to others.

Any free open source product qualifies as a public good. It is free for all to use, and one person using it does not exclude anyone else from using.

Re:The OpenTF Manifesto by VectorLock on Tuesday August 15, 2023 @07:56PM (#37140952)
I don't think you can just "fork it." Hashicorp already owns all that code because they make contributors sign it away to them.
Re:The OpenTF Manifesto by pxc on Tuesday August 15, 2023 @10:54PM (#37142344)
They can't retroactively take source code away from people who they already granted access to it under the MPL. The old code is still available under the MPL forever- even if they take down all of their own public copies of it, anyone with the old Terraform code is still free to upload their copy for the creation of a new fork. That's kinda the whole idea with these open-source licenses :)
Re:The OpenTF Manifesto by VectorLock on Wednesday August 16, 2023 @01:09AM (#37143067)
I've heard some people discuss that the contribution agreement that Hashicorp makes people sign gives them the right to change the license for existing contributions, but I'm not a lawyer so I really couldn't say for certain either way.
Re:The OpenTF Manifesto by linuxandrew on Wednesday August 16, 2023 @04:24AM (#37144187)
HashiCorp makes its external contributors sign a CLA to basically hand over the copyright.

However the MPL and licensing in general is irrevocable. They have irrevocably licensed Terraform 1.5.5 under the MPL and an enterprise license (dual license). Anyone can use, modify and distribute version 1.5.5 under the terms of the MPL.

Since HashiCorp retains full copyright they can release the next version under the BSL.

Note that many free software projects (like Linux) don't have a CLA which makes relicensing impractical since every contributor would have to agree to it.

Re:The OpenTF Manifesto by xinayder on Wednesday August 16, 2023 @02:47AM (#37143590)
Why not get it under the umbrella of either the Linux Foundation or CNCF? Things like this and Ansible should be really kept under neutral companies and not companies like Red Hat and HashiCorp that have shown that all they care about open source is the free work they get from contributors.