shaeqahmed

Matano is completely serverless and stores all data in ZSTD compressed parquet files in dirt-cheap object storage, allowing you to bring your own analytics stack for queries on large amounts of data for things like investigations and threat hunts. Since we store data in a columnar format and plug in query engines like Snowflake that are optimized for analytical processing the queries on specific columns will run much faster than they would run if executed on a search engine database like Elasticsearch which would require maintenance to scale.

I think it's important to understand that search engines and OLAP/data warehouse query engines have fundamental architectural differences that offer pros/cons for different use cases.

For enterprise security analytics on things like network or endpoint logs which can hit 10-100TB+/day, using anything other than a data lake is simply not a cost-effective option. Apache Iceberg was created as a big data table format for exactly this type of use case at companies like Netflix and Apple.

Yep, SIEM is just a superset of Log Management as it needs to do things like alerting + correlation + detection etc. in addition to ingesting logs to be considered a SIEM.

It is a common use case to send application logs along with security logs to something like Matano or Splunk for analysis as well, so feel free to use Matano to analyze your non-security logs!

Do keep in mind this will be a better fit if you have structured logs (you can also use VRL transformation to parse them at ingest) as the query language will be SQL.

Thank you for typing up a long detailed response. I think a lot of the points and concerns you bring up are valid, and we are mostly agreed upon.

In Matano however, we see Python as a viable component in security operations for narrowly tracking atomic signals while the language for writing detections and hunting threats will be SQL, which works perfectly well for use cases like the detection example you provided, albeit verbose. We have thought of also building a transpiler that would let analysts actually use the succinct syntax of SPL and compile that to SQL under the hood. This could be a great way to get adoption in companies where using Python would be difficult.

If you are interested, I would love to find some time to chat and share thoughts. Can you email me at shaeq at matano dot dev?

Long term, I believe Python (along with good ol' SQL for correlation) is the best language to model the kind of attacker behaviours companies are dealing with in the cloud and a lot of the difficulties with it are not inherent but around tooling. For example, in our cloud offering we plan on building abstractions that let you search for an IP or domain and get results with a click of a button as well the ability to automatically import Sigma rules and test Python logic directly with an instant feedback loop of a "low-code" workflow.

Currently we focus on more modern companies with smaller teams that have engineers that can write Python detections and actually prefer it over a custom DSL that needs to be learned and has restrictions.

Keep in mind there are more people in general that know Python than are trained in a vendor-specfic DSL so perhaps long term the role of a security analyst will evolve to overlap with that of an engineer. We are already seeing more and more roles require basic proficiency in Python as attacks on the cloud become increasingly complex :)

The code is written in high performance multi-threaded Rust and uses the [1] Arrow compute framework. We also batch events and target about 32MB of event data per lambda invocations. As a result it can process tens of thousands of events per second per thread, depending on the number of transformations.

That said, we are working on performance estimates and a benchmark on some real world data for Matano to help users like you better understand the cost factors. Stay tuned.

[1] https://github.com/jorgecarleitao/arrow2

We launched before Amazon Security Lake :)

Amazon Security Lake's main value prop is that it is a single place where AWS / partner security logs can be stored and sent to downstream vendors. As such, Amazon only writes OCSF normalized logs to the parquet-based data lake for it's own data in a fully managed way (VPC flow logs, Cloudtrail, etc.) and leaves it to the customers to handle the rest.

For partner sources, the integration approach has been to tell customers to set up infrastructure themselves to accomplish OCSF normalization, parquet conversion, etc. For example, here is okta's guide using Firehose and Lambda, https://www.okta.com/blog/2022/11/an-automated-approach-to-c...

The Amazon Security Lake offering is built on top of Lake Formation, which itself is an abstraction around services such as Glue, Athena, and S3. Security Lake is built using the legacy Hive style approach and does not use Athena Iceberg. There is a per-data cost associated with the service, in addition to the costs incurred by other services for your data lake. Looks like the primary use case of the service is being able to store first-party AWS logs across all your accounts in a data lake and being able to route them to analytical partners (SIEM) without much effort. It does not seem very useful for an organization that is looking to build its own security data lake with more advanced features, as you will still have to do all the work yourself.

Matano, has a broader goal to help orgs in every step of transforming, normalizing, enriching and storing all of their security logs into a structured data lake, as well as giving users a platform to build detection-as-code using Python & SQL for correlation on top of it (SIEM augmentation/alternative). All processing and data lake management (conversion to parquet, data compaction, table management) is fully automated by Matano, and users do not need to write any custom code to onboard data sources.

Matano can ingest data from Cloud, Endpoint, SaaS, and practically any custom source using the in-built Log transformation pipeline (think serverless Logstash). We are built around the Elastic Common Schema, and use Apache Iceberg (ACID support, recommended for Athena V2+). Matano's data lake is also vendor neutral and can be queried by any Iceberg-compatible engine without having to copy any data around (Snowflake, Spark, etc.).

Some big differences:

- Matano has realtime Python + SQL detections as code with advanced correlation support. Chronicle uses inflexible YARA-like detection rules iirc

- Matano supports Sigma detections by automatically transpiling them to the Python detection format

- Matano has an OSS Vendor Agnostic Security Data Lake and can work with multiple clouds / let's you bring your own query engine (Snowflake, Spark, Athena, BigQuery Omni). Chronicle is a proprietary SIEM that uses BigQuery under the hood and cannot be used with other tooling.

There are no limits on data retention or ingestion with Matano, it's your S3 bucket and the compute scales horizontally.

Thank you! We definitely believe in open source and don't need AGPL. Sending you love as you deal with that Splunk instance.

P.S. feel free to open some issues for any log sources you'd like to see supported in Matano

Thank you! Yes with AppTrail we wanted to solve the pain points around SaaS audit logs but since it was a product that needed to be sold and integrated into B2B startups rather than the enterprises that felt the pain points and needed audit logs in their SIEM, we couldn't find a big enough market to sell it.

We realized that the big problem was that most SIEM out there today did a poor job with pulling and handling the data from the multitude of SaaS and Cloud log sources that orgs have today, and decided to build Matano as a cloud-native SIEM alternative :)

We are working on a solution for GCP and Azure :) GCP recently announced Iceberg support with BigLake and support for federation across multi-cloud lakes so it would be perfect use cases.

If you are interested in using Matano for GCP, feel free to reach out and join our Discord community! We are FOSS so would love to collaborate on a solution.

I completely agree with you and the need for a fully integrated solution with great visualizations without hosting additional tools that aren't purpose built! Unfortunately there are very few SIEMs that get this right today..

Here's how we are thinking of it. We think it's important for a successful security program to first have high quality data and this is why we want help every organization build structured security data lakes to power their analysis using our open source project. The Matano security lake can sit alongside their SIEM and be incrementally adopted for a data sources that wouldn't be feasible to analyze otherwise.

Our larger goal as a company though is to build a complete platform that allows a security data lake to fully replace traditional SIEM -- including a UI and collaborative features that give you that great feedback loop for fast iteration in detection engineering and threat hunting as you mentioned. Stay tuned I think you will be excited by what we are building!

Many enterprises using Splunk are already being forced to purchase products like Cribl to route some of their data to a data lake because writing it all to Splunk is just way too expensive at that scale 1-100TB+/day (7 figures $).

But a data lake shouldn't just be a dump of data right? Matano OSS helps organizations build high value data lakes in S3 and reduce their dependency on SIEM by centralizing high throughput data in object storage using Matano to power investigations. To give you an example, one company is using Matano to collect, normalize, and store VPC Flow logs from hundreds of AWS accounts which was too expensive with traditional SIEM.

Matano is also completely serverless and automates the maintenance of all resources/tables using IaC so it's perfect for smaller security teams on the cloud dealing with a large amount of data and wanting to use a modern data stack to analyze it.

We think building a more efficient solution using data lakes is a win-win because it unlocks additional use cases for customers and allows them to analyze larger datasets within the same budget.

Solutions that offer a magnitude of order better performance than what is available today are critical for the industry because the amount of data teams are dealing with is growing much faster than their budgets!

Nope, Matano is named after one of the deepest lakes in the world - Lake Matano of Indonesia ;)

I'm on the fence about this one. When I heard it was announced, and that it was created by none other than the creators of the amazing AWS CDK, I was really excited by what could be possible. Having worked on complex infra automation using CDK (we use it extensively for our open source project for analyzing security logs on AWS: https://github.com/matanolabs/matano), I was excited because of the room for improvement with an integration that is language-native.

But after having looked into the abstraction that Winglang, and other "infrastructure-from-code" providers have come up with, I'm admittedly very skeptical. As other have mentioned, cloud primitives are almost by nature a leaky abstraction with many bells and whistles to be tuned. So I'm not sure it is a good idea, or feasible in a complex production application, to build on these very high level primitives such as cloud.Queue without limiting yourself to the lowest common denominator of features. But perhaps this issue is solvable by creating a nicer SDK.

What bothers me the most is having to to write code in a completely new language, that kind of treats runtime code as a second class citizen to be embedded in a configuration oriented language that looks like Typescript with some magic added in. Imo, this is far too much friction and risk vs. the benefit that could come from something like this over using your language of choice along with CDK.

I'm still rooting for Wing, and hoping they can figure out these issues, because the problem they are solving is a massive one. I think Winglang has the potential to do for cloud, what Rust did for memory safety by doing smart things at compile time and enforcing policies that could easily be missed by developers. For example, automatically deriving least privilege and minimal permissions for all infrastructure could be a great way to improve security out of the box.

okay but what about biometric auth via fingerprint - passwordless & zero context switches (provided your device supports this)

Yes, they would be handled automatically. Data ingestion is supported through S3 or Kafka, where files are picked up and ETL'D into structured Iceberg tables conforming to an ECS-like schema.

Feel free to join our Discord, happy to walk you through the steps and learn about your use case.

Matano is designed specifically for petabyte-scale security log analytics use cases, so performance and costs are a top priority. Our data pipeline borrows from Vector's Rust based data transformation language [0] for maximal performance, with each parallel function invocation capable of processing upwards of 20MiB/s [1] thanks to auto-vectorizaton.

Roughly this comes out to $1/(TB/day) for ingest compute costs which is much cheaper than a commercial solution. We are also working on moving over our Lambda's to ARM for even better cost-effficency.

[0] https://vector.dev/docs/reference/vrl/ [1] https://vector.dev/docs/setup/going-to-prod/sizing/#sizing

Using and maintaining Matano is a fraction of the cost compared to popular non-serverless alternatives like ELK or Spunk. Matano is specifically designed for petabyte-scale security analytics use-cases that don't fit in a traditional SIEM.

The serverless data ingestion pipeline means you don't need to over-provision for ingestion (Logstash and Splunk Forwarders are notorious for related costs / ops in high scale use-cases) in the write path. For reads, since Matano queries Iceberg tables backed by highly-compressed parquet files on object storage you won't pay anything close to what you would for a database or search engine based SIEM.

It is similar, although Snowflake is more of a query engine whereas we are a cloud security data platform built on an open data model (Apache Iceberg). We help you ingest and normalize data from common security sources into a data lake and offer a serverless platform to deploy & run Python detections-as-code on these events in realtime.

Although the only supported query service is currently Athena, we plan to integrate with popular vendors like Snowflake and Dremio. Thanks to the growing industry support for Iceberg, we believe vendor lock-in should be a story of the past for security data.

Its a data lake in which you store security logs. That includes Cloud/SaaS audit logs, network security logs (Zeek, Suricata, Snort), VPN/Firewall logs, and more.

Cool product and pricing model

> Cloud Log Lake

That's the first time I'm hearing a Clickhouse backend described as a lake. Care to explain?

> Customers also use Convoy as a broker for inter-service communication.

Very cool product! But this part threw me off a bit, wouldn't a distributed message broker like Kafka be a much better option for this use-case?

A CDC alone wouldn't. You would need to track and store that metadata in an additional database along with transaction ID, so it can be joined with the raw change stream to form a complete audit log.

Looks cool, I've tested out some of these React-inspired Rust UI frameworks and just can't seem to get over the convoluted macros with no syntax highlighting. Why hasn't anyone taken an approach to create something like .rsx (jsx equivalent)?