armon

Lock contention is a real issue for any multi-threaded system, and while a RW mutex is useful when you have a longer executing critical section, for something very short lived there is still a cache coordination cost. In many of the HashiCorp applications, we work around this by using an immutable radix tree design instead [1].

Instead of a RW mutex, you have a single writer lock. Any writer acquires the lock, makes changes, and generates a new root pointer to the tree (any update operation generates a new root, because the tree is immutable). Then we do an atomic swap from the old root to the new root. Any readers do an atomic read of the current point in time root, and perform their read operations lock free. This is safe because the tree is immutable, so readers don't need to be concerned with another thread modifying the tree concurrently, any modifications will create a new tree. This is a pattern we've standardized with a library we call MemDB [2].

This has the advantage of making reads multi-core scalable with much lower lock contention. Given we typically use Raft for distributed consensus, you only have a single writer anyways (e.g. the FSM commit thread is the only writer).

We apply this pattern to Vault, Consul, and Nomad all of which are able to scale to many dozens of cores, with largely a linear speedup in read performance.

[1] https://github.com/hashicorp/go-immutable-radix [2] https://github.com/hashicorp/go-memdb

We use HCL broadly across the HashiCorp portfolio. It was popularized by Terraform, but also used in Vault, Consul, Nomad, and Packer.

Important to note, that HCL is designed to be completely interoperable with JSON. In practice, we find that HCL tends to be more human readable and writable and is significantly less verbose.

However, if you are doing any sort of machine generation, it can be convenient to generate JSON and feed that in.

We've continued to grow the Nomad team, and are working towards the big 1.0 release milestone later this month. The OSS usage continues to grow double digit every quarter and our commercial offering generates millions in revenue. Nomad is also the backbone of both HashiCorp Cloud Platform and Terraform Cloud. Suffice to say, we continue to support Nomad and depend on it!

You would point the application at the local port. It operates very similarly to SSH port forwarding. No fancy magic to intercept all traffic.

We don't have a white paper on this yet, but we have a white board video that explains both how it works conceptually as well as at a more technical level of deployment architecture and data flow. https://www.youtube.com/watch?v=tUMe7EsXYBQ&feature=emb_titl...

That is correct! The local proxy has a listening socket and handles all the authentication, encapsulation, and forwarding transparently.

Disclosure: I'm one of the founders of HashiCorp.

Glad you found Consul easy to get going with. We definitely focus on enabling K8S as a first class platform. There is documentation on setting up Consul and operating with Kubernetes here: https://www.consul.io/docs/platform/k8s/index.html

We also have a bunch of learning content and guides on using Consul and K8S together to get started: https://learn.hashicorp.com/consul?track=kubernetes#kubernet...

HashiCorp (https://www.hashicorp.com) | SRE, Software Engineers, Engineering Managers | REMOTE | Full-time

HashiCorp builds DevOps tools for modern applications. We build open source tools including Vagrant, Packer, Terraform, Consul, Vault, and Nomad. We also have a set of cloud services (Vagrant Cloud, Terraform Registry, Terraform Cloud, etc) that are high scale SaaS services. The HashiCorp products are downloaded tens of millions of times per year and power tens of thousands of organizations. Join our growing team and help organizations deliver modern applications using DevOps practices and enable dynamic multi-cloud infrastructure.

For a complete list of openings, see https://www.hashicorp.com/jobs

Some of our openings include:

* Infrastructure Engineer (multiple openings): https://www.hashicorp.com/jobs/1807978

* Software Engineer, Vault/Consul/Nomad- https://www.hashicorp.com/jobs/1844783

* Software Engineer, Cloud Services- https://www.hashicorp.com/jobs/1735894

* Site Reliability Engineer, Cloud Services- https://www.hashicorp.com/jobs/1600002

* Engineering Manager, Vault/Consul/Nomad- https://www.hashicorp.com/jobs/1845061

* Infrastructure Engineering Manager, Terraform Cloud- https://www.hashicorp.com/jobs/1801371

At HashiCorp, we are committed to hiring and cultivating a diverse team. If you are on the fence about whether you meet our requirements, please apply anyway!

HashiCorp (https://www.hashicorp.com) | SRE, Software Engineers, Engineering Managers | REMOTE | Full-time

HashiCorp builds DevOps tools for modern applications. We build open source tools including Vagrant, Packer, Terraform, Consul, Vault, and Nomad. We also have a set of cloud services (Vagrant Cloud, Terraform Registry, Terraform Cloud, etc) that are high scale SaaS services. The HashiCorp products are downloaded tens of millions of times per year and power tens of thousands of organizations. Join our growing team and help organizations deliver modern applications using DevOps practices and enable dynamic multi-cloud infrastructure.

For a complete list of openings, see https://www.hashicorp.com/jobs

Some of our openings include: Software Engineer, Terraform Cloud- https://www.hashicorp.com/jobs/1788064 Software Engineer, Systems- https://www.hashicorp.com/jobs/1588975 Software Engineer, Cloud Services- https://www.hashicorp.com/jobs/1620078 Site Reliability Engineer, Cloud Services- https://www.hashicorp.com/jobs/1600027 Engineering Manager, Systems- https://www.hashicorp.com/jobs/1625448

At HashiCorp, we are committed to hiring and cultivating a diverse team. If you are on the fence about whether you meet our requirements, please apply anyway!

I gave a talk at GoSF about Radix trees, and how they are used heavily in HashiCorp products (Terraform, Consul, Vault, Nomad, etc). The slides are available here for those interested: https://speakerdeck.com/armon/radix-trees-transactions-and-m...

Radix trees are one of my favorite data structures, and widely under used. For many "dictionary" type lookups, they can be faster and more efficient than hash tables. While hash tables are commonly described as being O(1) for lookup, this ignores the need to first hash the input, which is typically an O(K) operation, where K is the length of the input string. Radix trees do lookups in O(K), without needing to first hash and have much better cache locality. They also preserve ordering allowing you to do ordered scans, get min/max values, scan by shared prefix, and more.

If you combine them with an immutable approach (such as https://github.com/hashicorp/go-immutable-radix), you can support more advanced concurrency, such as lock free reads. This is important for highly concurrent systems to allow scalable reads against shared memory.

Hey, I'm one of the co-founders of HashiCorp.

To keep it brief, we are more committed to Nomad today than before. The team has doubled in the last year, and we plan to grow further next year as well. Our goal has always been to build a simple, general purpose scheduler, that composes well with the rest of the HashiCorp ecosystem.

Kubernetes is an important ecosystem and a platform we tightly integrate with across our other products (Terraform, Consul, Vault). We've always believed that our tools would be "mixed and matched" with different technologies, and that pragmatically we should support the broadest range of integrations.

Nomad is an important piece of our ecosystem, and we have many open source users, enterprise customers, and our SaaS offerings are built on it. Rest assured, it's not going anywhere!

That is exactly right. At the bottom of the blog post we touch on this, but if you are using consul-template to provide secrets via a configuration file, it can either restart or reload (signal) the application to pickup the changes. Alternatively, an application could be Vault aware and use the SDK programmatically.

Hey peteski22,

Exactly what you suggested would work! Having an AppRole that never expires would allow the trusted orchestrator to authenticate on each run, and then generate and inject ephemeral credentials.

Vault is free and open source. We do have an enterprise product, but all the dynamic secrets capabilities exist in the open source!

There are some good suggestions in the replies here, but I'd recommend looking at our secure introduction guide here: https://www.vaultproject.io/guides/identity/secure-intro.htm...

Effectively, there are only two possible approaches. Platform specific integrations, such as AWS IAM, Kubernetes, Nomad, etc. Or Trusted Orchestrators, which help inject the initial "secret zero".

When Vault connects to the endpoint system to create a dynamic user, it presents a set of credentials only known to it. You have to authorize Vault to create dynamic users, so a malicious actor would need to somehow obtain a similar level of privilege.

Vault typically prefixes something to the username as well (e.g. "vault-...") and also audits the creation of dynamic users so you can either look for the prefix or cross check the audit logs.

We work with many Fortune 2000 customers, and having 500K secrets is on the extreme side and most certainly puts you in an infrastructure where you have 50K-100K+ machines under management.

In terms of the "compute cost", for an infrastructure of that size this is a negligible amount of overhead. For dynamic secrets that live 30 days, rotating 500K secrets works out to 1 secret every 5 seconds.

The advantage would be avoiding an incredible number of static credentials sprawled across a very large estate, plus having a unique audit trail that lets you identify points of compromise. Treating those credentials as dynamic will also reduce the human overhead of managing so many credentials, instead focusing on roles and high level intents.

I question if there is an non-disclosed bias given the anonymous user, created just in advance of the comment.

Kerberos was a major inspiration for us!

The goal of Vault was to be a modern Kerberos, but invert the integration model. I think that is the Achilles heel of Kerberos, since it has a complex API and only works if the endpoint systems are tightly integrated.

Vault operates in much the same way, and could be viewed as a "KDC". However, instead of requiring the Authentication Service (AS) to be Vault aware, Vault uses authentication plugins to do the integration in the other direction. Similarly, instead of network services being Vault aware, Vault uses secret plugins to do the integration with endpoint systems.

This lets Vault easily be extended to support new authentication systems and endpoint systems, without needing those systems to be modified. Otherwise, its conceptually very similar to Kerberos!

Edit: I'm a co-founder of HashiCorp, and one of the early authors on Vault.

I'm one of the co-founders of HashiCorp, and I wanted to just chime in on the release. We may have done a disservice in the blog post by highlighting the new features landing in the Enterprise versions, because it obscures all the new functionality and work that has gone into the open source. We call out some of these at the bottom of the post, but just for the sake of clarity:

* Identity / Group Management. This sounds like a minor bullet point, but an incredible amount of work went into supporting a unified identity system that could span multiple external identity brokers. Our goal was to allow Vault to act as a broker between many Identity Providers and apply a consistent access control and group management scheme to all of theme. This was something we originally landed in Enterprise with 0.8 so that we could bake it with a smaller number of customers and have since brought to open source as we feel more confident in the implementation and APIs.

* GCE integration. This is part of our ongoing partnership with Google and our goal of natively integrating Vault with all the common platforms our users want to use.

* K8S integration. Again, this is about the last mile integration into the platforms users want and is in the open source.

* Oracle database integration. The database backend provides dynamic secret functionality across a number of RDBMS and NoSQL systems, and we are continuing to broaden the supported systems.

* RSA support for transit encryption. The transit backend allows Vault to do key management and cryptographic offload, and we are continuing to extend the algorithms and operation types that are supported.

* Lazy loading of leases. Common feedback from our large scale users was that failover could take many minutes. We've improved the loading of leases and other metadata to be asynchronous resulting in a dramatically faster failovers for large scale deployments.

* Lots of other bug fixes and minor improvements.

This is only a small sample of what has been added in the Vault 0.8.3 and 0.9 releases as open source. In the last 6 months, we’ve added more to the OSS than ever before, because Vault Enterprise has allowed us to grow the team. By adding a small subset of features to our Pro and Premium Enterprise offerings we are able to continue developing Vault, and we take a similar approach for our other tools. I totally understand the frustration about our opaque pricing, and this is because we are still making pricing adjustments as we refine our fit and better understand the many different use cases customers have.

Mitchell has publically stated and I'll repeat, that our goal is to publish both our framework of determining what is open source versus commercial (largely a focus on compliance, governance, and collaboration) as well as our public pricing. Our goal is to be as transparent as possible, while still accommodating the need to understand product/market fit, refine pricing, and be a solvent business.

I'm curious if the author was aware of our implementation at HashiCorp (https://github.com/hashicorp/memberlist). We use that implementation in Serf (https://www.serf.io) to provide membership, failure detection, and an event system. Serf is in turn, used by both Consul (https://www.consul.io) and Nomad (https://www.nomadproject.io). The point being, our implementation has seen significant production hardening and is known to work on clusters with 10K+ machines.

We've made significant enhancements to SWIM to make it work at scale, but overall it's a simple algorithm to understand and works quite well in practice. Glad to see more implementations of it!

The used synthetic benchmark is not a great indicator of performance in most (any?) production environments. It's generally useful and interesting to understand the tradeoffs between an LSM and B-Tree. Particularly, if you have an update or delete heavy workload the compaction cost of an LSM can become an issue. B-Tree's don't suffer from compaction issues, so in some sense trade consistently slower writes instead of amortizing out an I/O intensive compaction.

Some of the worst production experiences I've had came from exhausting I/O on the database, and then having LevelDB / RocksDB / LSM stores kick off their compaction. B-Tree will give you a very consistent redline in terms of performance generally.

TL;DR: There are trade offs between the two, but this benchmark is not particularly insightful given that it doesn't really test any of the interesting boundary conditions or real world query patterns.

I've given a talk at Papers We Love about Bloom Filters (http://paperswelove.org/2015/video/armon-dadgar-on-bloom-fil...) for those wanting to learn more. There have been quite a few follow on extensions to standard bloom filters that address some of the shortcomings mentioned here.

For example, the cost of having K hash functions can be avoided with some simple math. This is discussed in "Less Hashing, Same Performance" (http://www.eecs.harvard.edu/%7Ekirsch/pubs/bbbf/esa06.pdf). You can also avoid the FPP rate from saturating at 100% using a technique called "Scalable Bloom Filters" (http://gsd.di.uminho.pt/members/cbm/ps/dbloom.pdf). The basic idea is you start with a small BF and layer on progressively larger filters. This also reduces the memory you need to allocate up front without worrying about the FPP rate saturating. As others have mentioned, Counting Bloom Filters allow for counts and deletion as well. The basic point is that there has been a _lot_ of additional work on bloom filters. One of my favorites is "Adaptive Range Filters" (http://www.vldb.org/pvldb/vol6/p1714-kossmann.pdf).

At an advertising company I previously worked for, we relied on bloom filters and hyperloglogs quite heavily for our realtime analytics pipeline. We open sourced our C daemons (https://github.com/armon/bloomd and https://github.com/armon/hlld) which have been running in production for several years.

The article can probably be succinctly summarized as "Premature optimization is the root of all evil". Most of the authors points are valid, in that caching adds more complexity.

That said, caching is absolutely critical to almost every piece of software ever. Even if you explicitly caching isn't used, a wide variety of caches are likely still being depending upon including CPU caching (L1, L2, L3), OS filesystem caching, DNS caching, ARP caching, etc etc.

Caching certainly adds complexity but it's also one of the best patterns for solving a wide range of performance problems. I would recommend developers spend more time learning and understanding the complexities so that they can make use of caching correctly and without applying it as a premature optimization.

To clarify, statsite does spawn threads to do some of the aggregation and flushing at the end of the collection interval, but yes it is possible to saturate the main loop at a very high ingest rate.

This looks pretty interesting! We use a library we've built called relay (https://github.com/armon/relay) to do something similar also in Go.

I like the higher level task based approach of this library, since this is a pretty common pattern when using task brokers.

The issues discussed by aphyr have been fully fixed as of Consul 0.2, several months ago. In the initial release of Consul, only the "default" mode was available and has the now-documented caveat of a stale read possibility. The various consistency modes are now discussed here: https://www.consul.io/docs/internals/consensus.html#default

With respect to ZooKeeper, there are different approaches. ZK provides a low-level primitive on which you can build. Consul provides similar primitives, but it ships with many features out of the box that don't require any development effort. It's a "batteries included" approach.

Specific examples:

- Real-time configuration with Consul + consul-template

- DNS based service discovery

- Scalable Nagios replacement

- Dynamic HAProxy / Varnish configuration

- Application configuration with Consul + envconsul

- Triggering config management tools with the event system

That is just a handful of uses for Consul that don't require writing any code. Doing similar things with ZK is possible, just requires a lot more work.

You are right, the communication is still unicast in nature. I should clarify to say that there isn't a persistent 1-to-1 communication, the nodes we gossip with are randomly selected on each interval. There is no connection or session establishment between peers.

I guess it depends on your definition of roll your own. We didn't invent AES-GCM or implement it. We are using the implementation shipped with the Golang stdlib.

We compare Consul to SmartStack here: https://www.consul.io/intro/vs/smartstack.html

With a tool like consul-template, you can have the same behavior that Synapse gives you. https://hashicorp.com/blog/introducing-consul-template.html

In the general case, clients of Consul don't need special handling code for events, it's handled by Consul for you.