russjones

Teleport (YC S15) | Backend and Fullstack Engineers | US, Toronto, London, Remote OK | https://goteleport.com

Do you enjoy building security and deployment tools for other engineers? Join us to hack on https://github.com/gravitational/teleport. Most of our code is Go, we have very little technical debt, our codebase is clean and small.

If you are a backend or fullstack engineer, we expect you to be comfortable with the following:

  * Go for backend and Go+TypeScript for fullstack engineers.
  * Linux, networking, and security.
  * Scalability and/or security experience for systems software is welcome.

We're looking for senior engineers to join the Teleport team. Teleport is a company started by engineers to build products for engineers. We are a stable and growing company.

We offer:

  * Competitive salary and equity.
  * 401k with company match.
  * Excellent health insurance.

Apply: https://jobs.lever.co/teleport

What to expect once you apply:

  * You will join a 30 minute intro call and we will walk you through compensation, the interview process, and requirements.
  * You join a private Slack channel and submit a coding challenge in Go (backend) or Go+TypeScript (fullstack) depending on the position using Github.

Teleport (YC S15) | Backend and Fullstack Engineers | US, Toronto, London, Remote OK | https://goteleport.com

Do you enjoy building security and deployment tools for other engineers? Join us to hack on https://github.com/gravitational/teleport. Most of our code is Go, we have very little technical debt, our codebase is clean and small.

If you are a backend or fullstack engineer, we expect you to be comfortable with the following:

  * Go for backend and Go+TypeScript for fullstack engineers.
  * Linux, networking, and security.
  * Scalability and/or security experience for systems software is welcome.

We're looking for senior engineers to join the Teleport team. Teleport is a company started by engineers to build products for engineers. We are a stable and growing company.

We offer:

  * Competitive salary and equity.
  * 401k with company match.
  * Excellent health insurance.

Apply: https://jobs.lever.co/teleport

What to expect once you apply:

  * You will join a 30 minute intro call and we will walk you through compensation, the interview process, and requirements.
  * You join a private Slack channel and submit a coding challenge in Go (backend) or Go+TypeScript (fullstack) depending on the position using Github.

You only need a browser if you are using SSO. If you are using local accounts it's all done in the CLI.

Hi scarygliders,

That's a good question and valid skepticism. The UI serves two purposes.

The first is to manage your cluster. For example, you may want to change a users role (to change what servers they can access) or play back a session recording to see what a user did within a session. You can do all of these things from the CLI, but sometimes you don't want to remember the exact command and flags, that's where the UI comes in.

The second is to have access to servers from an environment that either does not have robust SSH support, doesn't allow you to install software, or maybe an environment (temporary) into which you don't want to install software. For example, when the UI was first written Windows did not have great support for SSH certificates, but the UI would allow Windows users to access Teleport clusters.

Lastly, the UI is completely optional, in addition even the CLI tool is (almost) optional. Once you've fetched your certificate, you can use your OpenSSH client as normal. In-fact a lot of people use Teleport this way.

Disclaimer: I work at Gravitational.

We're trying to raise structured behavioral information about what is happening in a session to the cluster administrator.

That means we don't just provide information about what's executing, but also what files are being opened and TCP connections being established. Other avenues of expansion you may see this feature venture into: how were files changed, support for other protocols, support for other events (bind, listen, accept).

However we are not claiming this approach is not subvertable, but we do want to raise the bar for attackers and make it easier for cluster administrators to understand what is happening within their system.

Thank you Abe! The kind comments are appreciated!

So far we have gotten positive feedback. While this feature does not protect against root doing something malicious, it does allow admins to capture what root was doing up until they did something malicious and link that information to an identity (if using SSO).

Along with this feature we rolled out a Workflow API [1] that can be used to request role elevation. Once you add in session termination (which we are aiming for in the next release), you will have a powerful set of features that will allow you to start users out with limited access to your cluster with the ability to request more privileges and potentially automatically termination your session (and user) if you're found to be doing some malicious.

[1] https://gravitational.com/blog/workflow_api/

Understandable confusion, I've seen it referred to both ways. We decided to go with BPF for this blog post since that appears to be the official abbreviation.

From "BPF Performance Tools" by Brendan Gregg:

"Extended BPF is often abbreviated as eBPF, but the official abbreviation is still BPF, without the "e," so throughout this book I use BPF to refer to extended BPF. The kernel contains only one execution engine, BPF (extended BPF), which runs both extended BPF and "classic" BPF programs."

Author of the post here, happy to answer any questions.

Noted.

I ran an eBPF program called opensnoop [1] to capture what files were opened during login to a system and then re-launching bash. Looks like both are read during initial login but only .bashrc for non-login shells. Output is below.

  24435  bash                3   0 /etc/profile
  24435  bash                3   0 /etc/profile.d/
  24435  bash                3   0 /etc/profile.d/256term.sh
  24435  bash                3   0 /etc/profile.d/colorgrep.sh
  24435  bash                3   0 /etc/profile.d/colorls.sh
  24435  bash                3   0 /etc/profile.d/lang.sh
  24435  bash                3   0 /etc/profile.d/less.sh
  24435  bash                3   0 /etc/profile.d/which2.sh
  24435  bash                3   0 /etc/profile.d/sh.local
  24435  bash                3   0 /home/centos/.bash_profile
  24435  bash                3   0 /home/centos/.bashrc
  24435  bash                3   0 /etc/bashrc

  24736  bash                3   0 /home/centos/.bashrc
  24736  bash                3   0 /etc/bashrc
  24736  bash                3   0 /etc/profile.d/
  24736  bash                3   0 /etc/profile.d/256term.sh
  24736  bash                3   0 /etc/profile.d/colorgrep.sh
  24736  bash                3   0 /etc/profile.d/colorls.sh
  24736  bash                3   0 /etc/profile.d/lang.sh
  24736  bash                3   0 /etc/profile.d/less.sh
  24736  bash                3   0 /etc/profile.d/which2.sh

[1] http://www.brendangregg.com/blog/2014-07-25/opensnoop-for-li...

Maybe I did not provide the best example, but the point I was driving at is that the restrictions here are at the application level and not at the system level.

If you take the approach you are taking, similar to ForceCommand, you have to make sure none of the commands in your PATH allow code execution. For example git, vim, less, awk, and find, all of which I use on a daily basis, can all be used to re-launch a restriction free bash.

Once you remove all the commands you have to, how much of a shell do you have left?

Ha, nope, didn't realize this until everyone started mentioning it in the comments.

Author here, happy to answer any questions about the post.

We had a security audit of Teleport performed by Cure53 with the release of Teleport 2.2 which we released publicly. You can view the report here:

https://cure53.de/pentest-report_teleport.pdf

Hi walrus01,

If exposing Teleport (or OpenSSH or any piece of software to be honest) to the internet is a significant concern, the solution I recommend is to put it behind spiped. Spiped has a very small and well written code base and well suited for this problem.

Because Teleport is fully interoperable with OpenSSH you just follow the instructions in the README to get it working: https://www.tarsnap.com/spiped.html

Hi Operyl, Russell from Gravitational here.

We've made improvements in resource utilization for Teleport 2.0 and hunting down any further resource utilization issues is definitely one of my focus areas for Teleport 2.x.

If you run into issues like this again please reach out and send us your teleport.yaml, we want to run these issues down and resolve them.

Hi nikolay, we don't have a CloudFormation template, but we do have a containerized version of Teleport that is fully configured with multiple clusters that you can use to test with.

https://github.com/gravitational/teleport/tree/master/docker

Hi atonse, Russell from Gravitational here.

As far as configuring your VPC having the bastion (Proxy) as the only server with a public address is reasonable. One of the nice things about Teleport is that the Teleport Proxy itself doesn't have access to much, so exposing it to the Internet is fine. The Auth Server is the one that holds sensitive information and we recommend you create a security group for it and only allow it to be accessed from Teleport Proxies or Teleport Nodes.

With respect to keys, they are stored and accessed via the Auth Server in Teleport. We recommend you have strong access controls on the Auth Server. If you are using the default backend (BoltDB) or directory based backend that's all you need to do. If you are using etcd we recommend you have strong access controls on the server that runs etcd as well as etcd itself, we have an example in our Teleport repo for etcd configuration if you're interested[1]. If you are using DynamoDB, we recommend having a strong IAM policy. We are not using Vault at the moment.

[1] https://github.com/gravitational/teleport/tree/master/exampl...

Thanks, the DO's and DON'TS were heavily inspired by a talk given by Colin Percival called "Everything you need to know about cryptography in 1 hour". You can check it out here: https://www.bsdcan.org/2010/schedule/attachments/135_crypto1...

It's really good!

I don't mind the criticism constructive or otherwise.

I have already sketched out where to take part 2 and filled in some of the sections. That being said the 80/20 rule applies here like everywhere else.

That being said, if anyone wants to collaborate on part 2, feel free to ping me at [email protected] and we make a mailing list.

Fixed that, thanks for pointing it out.

Author of the guide here, don't worry, we will!

Author of the post here, happy to answer any questions or provide more details about our recommendations.

This is a good point. That's why we recommend either a robust CI/CD system (with good test coverage) or manually applying updates to a staging environment where you can test if the updates broke anything end-to-end before you roll them out to customers.

Author here, happy to answer any questions/discuss the recommendations we provided.

Russell from Mailgun here.

From our perspective, most people start (or at least should start) with the premise that email is not secure. There are things that we can do to make email more secure, but as an email service provider, we have to balance delivery with security. Most customers would rather have their email delivered to an inbox than us only deliver to an inbox where we can validate the TLS certificate presented by the server. In-fact, delivery is the entire reason companies like Mailgun exist.

That doesn't mean we don't care about security. We've rolled out opportunistic TLS about a year ago, we're working on rolling out mandatory TLS soon (with strict certificate verification), and have a slew of security feature coming down the pipeline that I think (some) customers will be really excited about. However, when it comes to delivery, the truth is that features like mandatory TLS are not really requested or demanded by customers. In the email world, delivery trumps security which is not surprising when you work from the premise that it is not secure to begin with.

Hi 0x0, Russell from Mailgun here. Do you have a self signed certificate by any chance? We typically downgrade a connection if we are presented with a self signed certificate. That might be what you are seeing.

If you drop us an email at support [at] mailgun.com with your domain I can take a look at the specific issue we are having delivering over TLS to your domain.