cm

How would this work for my 18th century agrarian business?

Congrats on the launch! Your site is loading now, albeit a bit slowly :)

We don't currently have use cases that require heavy transformations (see this blog post I wrote to explain why: https://blog.stitchdata.com/why-our-etl-tool-doesnt-do-trans...).

However, since Singer is built around piping data between applications, your suggestion - to code something that sits between taps and targets - makes perfect sense. The whole "flow" would look like:

$ tap-mydatasource | do-aggregations | target-mytarget

We'd be eager to hear from anyone who tries this approach!

There are a couple reasons why we included schemas in the spec:

- JSON doesn't have a robust set of data types, and specifically lacks a datetime/timestamp type. With a schema, Taps can, for example, denote fields in the JSON that contain datetimes represented as strings, and then targets can convert those to proper datetimes and handle them accordingly.

- Dealing with un-structured or flexibly-structured data is hard. Requiring a schema forces a Tap author to think about the structure of the data up front. By validating each data point against a schema, the Tap author should be able to more quickly identify nuances in the data set - like missing fields, nullable fields, mixed-type fields, etc - and either decide to clean them out of the data (if appropriate), or provide the right schema to inform downstream applications about them. Identifying and handling these problems requires an understanding of the source data set, so it is best done as close to the data source as possible.

It's accurate to say that we're more focused on the extraction and loading parts of ETL. In our experience, almost all useful data analysis requires data to be transformed at multiple stages - for example, once to cleanse and normalize the data, and again to aggregate it. Our goal is to leave the data in the rawest form possible without losing accuracy or precision, so that further transformation can occur, likely using SQL. We recognize this isn't perfect for every use case, but our customers love it for getting rapid access to data that would otherwise be locked away in SaaS applications and transactional databases.

We do have an API for sending data into the Pipeline, documentation for it can be found here: https://docs.stitchdata.com/hc/en-us/categories/203326787-Im...

Additionally, we'll be releasing a Java client library any day now, with other languages and platforms to follow.

I'm an engineer at Stitch. Our approach to transformation is to do just enough to move data from one system to another without losing precision or fidelity. So, we transform datatypes and structures into more appropriate forms for the target system, but we don't have any transformation operators like aggregation or windowing.

We have found that this approach works well for our users, who prefer to get the rawest possible data, and the systems we target like Redshift that are themselves powerful transformation engines. This gives the user unlimited flexibility for defining transformations, and a full audit trail for understanding how their data has changed.

We are always evolving, though, so if there's a use case that you think requires this approach, I would be eager to hear more about it.

I added some explanation about this - thanks for the feedback

I work with robertjmoore. I, along with a few other colleagues at our small office, voted for his past three blog posts. Many of those votes probably came from our single office IP address. Many were probably also placed after clicking a direct link to the HN posting. There couldn't have been more than about 10 votes like this, because we don't have many employees. And, in the interest of full disclosure, I don't vote for many HN posts besides those written by authors I know.

Is this the behavior that HN's vote ring detector is trying to discourage? I understand that these things are a slippery slope - but if so, it's too bad, because quality content like robertjmoore's last three posts is getting lost, and I would imagine that other authors at small companies like ours are unwittingly falling into the same trap.

If there have been other posts explaining the DOs and DONTs of the HN vote ring detector, I apologize in advance for not having read them.

RJMetrics is hiring software development interns in Philly

http://www.rjmetrics.com/jobs

You're neglecting to factor in bandwidth and EBS-usage (because, keep in mind, EC2 micro instances MUST be EBS backed). 10GB of outbound data transfer is another $1.10, plus another $.50-$1.00 in EBS costs per instance.e

I believe standard amazon gift cards can be used to pay for AWS charges, but it would be after-the-fact, not prepaid.

I wrote this up when I was dealing with the same issue a few months back - a simple deploy script for PHP applications using git: http://themetricsystem.rjmetrics.com/2010/02/01/simple-deplo...

"Fuck Everything, We're Doing Five Blades" - James M. Kilts, CEO and President, The Gillette Company.