Re:Show HN: Open-source serverless security lake powered by ...
by shaeqahmed on Saturday August 06, 2022 @01:46PM
(#32370098)
Matano is designed specifically for petabyte-scale security log analytics use cases, so performance and costs are a top priority. Our data pipeline borrows from Vector's Rust based data transformation language [0] for maximal performance, with each parallel function invocation capable of processing upwards of 20MiB/s [1] thanks to auto-vectorizaton.
Roughly this comes out to $1/(TB/day) for ingest compute costs which is much cheaper than a commercial solution. We are also working on moving over our Lambda's to ARM for even better cost-effficency.
[0] https://vector.dev/docs/reference/vrl/ [1] https://vector.dev/docs/setup/going-to-prod/sizing/#sizing
Re:Show HN: Open-source serverless security lake powered by ...
by OrvalWintermute on Saturday August 06, 2022 @06:49PM
(#32372299)
> Roughly this comes out to $1/(TB/day) for ingest compute costs which is much cheaper than a commercial solution. We are also working on moving over our Lambda's to ARM for even better cost-effficency.
Given a prior exploration of a security visibility stack on AWS at scale, this is likely a colossal sized operational expense.
Honestly think at that point you’d be better off and cheaper to go with a commercial security data lake..