News for hackers. Stuff that matters.
Re:Show HN: X to Elasticsearch Sync by styfle on Saturday November 18, 2017 @12:01PM (#15729743)
A couple questions with regards to the SQL -> ES:

1. Does it sync deltas or do you have to import the whole table each time?

2. Does it listen for changes to a table or does it require a manual invocation?

3. Can you explain more about the algorithm/implementation?

Re:Show HN: X to Elasticsearch Sync by sidi on Saturday November 18, 2017 @03:12PM (#15730427)
1. and 2. It syncs deltas if you are either using Postgres (9.4 or above) or MongoDB. For other sources like MySQL or MSSQL, it needs to import the whole table each time. The tailing capability is DB dependent, for instance, in MongoDB's case - we use the oplog to do this.

The way it works is after the initial import if you are using the `--tail` option, it listens to further changes via oplog. More on setting oplog access over here - https://github.com/appbaseio/abc/blob/dev/docs/importer/adap...

For Postgres, similarly a replication role and a replication slot needs to be created for decoding the write ahead log. More on that here - https://github.com/appbaseio/abc/blob/dev/docs/importer/adap....

This seems like a good primer on replication slots - https://postgresqlspace.wordpress.com/2015/06/18/replication....