News for hackers. Stuff that matters.

A comment on Pachyderm (YC W15) Challenges Hadoop with Containerized Data Lakes.

Re:Pachyderm (YC W15) Challenges Hadoop with Containerized D... by eterps on Tuesday May 10, 2016 @03:09PM (#11669718)
It uses Kubernetes underneath, which is nice for scaling but how do you run automated functional, end-to-end or integration tests with this on your development (or CI) machine?
[ Parent | on HN ]
Re:Pachyderm (YC W15) Challenges Hadoop with Containerized D... by manojlds on Tuesday May 10, 2016 @03:32PM (#11669918)
Something like this might help - https://github.com/redspread/localkube
[ Parent | on HN ]
Re:Pachyderm (YC W15) Challenges Hadoop with Containerized D... by sjezewski on Tuesday May 10, 2016 @03:15PM (#11669770)
Great question!

For local development - we recommend connecting to a dev k8s cluster, which is easy enough to setup.

CI is actually something I get really excited about. Similary for CI you can have a separate k8s cluster for any tests you need to run. But because we version everything, you can run 'unit tests' on sample data sets that you know should be rock solid ... and also 'integration tests' on the _same_versioned_data_ you would see in production.

[ Parent | on HN ]
Re:Pachyderm (YC W15) Challenges Hadoop with Containerized D... by jdoliner on Tuesday May 10, 2016 @03:22PM (#11669836)
Just to clarify, a dev k8s cluster doesn't require anything more than a stock installation of Docker. K8s has actually gotten pretty easy to test against recently thanks to being deployable as containers.
[ Parent | on HN ]
Re:Pachyderm (YC W15) Challenges Hadoop with Containerized D... by eterps on Tuesday May 10, 2016 @03:45PM (#11670036)
Do you have some examples or links to share?
[ Parent | on HN ]
Re:Pachyderm (YC W15) Challenges Hadoop with Containerized D... by jdoliner on Tuesday May 10, 2016 @03:48PM (#11670057)
This is the script we use to start k8s for tests: https://github.com/pachyderm/pachyderm/blob/master/etc/kube/...

There's k8s docs on this somewhere but unfortunately they move around a lot so I can't find a link right now. It's just 1 docker run command though and you're good to go.

[ Parent | on HN ]