Building a Full-Text Search App Using Docker and Elasticsearch

Re:Building a Full-Text Search App Using Docker and Elastics... by jcadam on Thursday February 01, 2018 @12:21PM (#16283664)

The sync issues between an RDBMS and ElasticSearch are definitely the most annoying aspect of getting something like this working.

In the case of my current project, I've got an 'indexer' service whose sole job is to await messages from RabbitMQ to create/update/delete records to/from the index.

But sometimes an update may get missed for any number of reasons. So, I've got a systemd timer that fires off a little script to go through my audit tables in Postgres and make sure all changes since the last run were actually, in fact, synced to ElasticSearch.

And I'm still wondering if what I'm doing is 'good enough.' How else could this thing fail?

[ Parent | on HN ]

Re:Building a Full-Text Search App Using Docker and Elastics... by amelius on Thursday February 01, 2018 @01:04PM (#16284068)

What if your server crashes after you commit to the database, but before you submit to elastic-search? Even if submitting is flawless, this still presents a problem.

[ Parent | on HN ]

Re:Building a Full-Text Search App Using Docker and Elastics... by jcadam on Thursday February 01, 2018 @01:12PM (#16284152)

Depends on which server :D The Database, ElasticSearch, RabbitMQ, and the application services are on different VMs (in my case, Linodes).

Committing to the database will trigger an entry (via a trigger function) in an audit table. So there is still a record of the transaction. After the system is restored to a healthy state, the next time the sync script is run (either automatically or manually if I'm feeling anxious), it should catch up.

Of course, in a worst-case scenario, I could always rebuild the index :O.

[ Parent | on HN ]

Re:Building a Full-Text Search App Using Docker and Elastics... by AznHisoka on Thursday February 01, 2018 @09:03PM (#16287792)

A server crash should be a rare event but if you reindex data regularly, it should not be a huge worry.

[ Parent | on HN ]

Re:Building a Full-Text Search App Using Docker and Elastics... by bdcravens on Friday February 02, 2018 @12:36AM (#16288654)

In Rails, the searchkick gem seems to do a good job of solving this pain point

[ Parent | on HN ]