如何将mysql数据实时迁移到ElasticSearch
我有一个包含几个表的 mysql 数据库，我想将 mysql 数据迁移到 ElasticSearch.通过批处理作业将整个数据库迁移到 ES 很容易.但是我应该如何从 mysql 实时更新 ES.即如果 mysql 中有更新操作，那么我应该在 ES 中执行相同的操作.我研究了 mysql binLog，它可以反映 mysql 的任何更改.但是我要把binLog解析成ES语法，我觉得真的很痛苦.谢谢！(与 Solr 相同的情况)
I have a mysql database with couple tables, I wanna migrate the mysql data to ElasticSearch. It's easy to migrate the whole database to ES via a batch job. But how should I update ES from mysql realtime. i.e if there was a update operation in mysql then I should do the same operation in ES. I researched mysql binLog which can reflect any changes from mysql. But I have to parse binLog to ES syntax, I think it's really painful. Thanks! (the same case with Solr)
推荐答案
有一个现有项目可以获取您的 binlog，将其转换并将其发送到 Elasticsearch，您可以在以下位置查看:https://github.com/siddontang/go-mysql-elasticsearch
There is an existing project which takes your binlog, transforms it and ships it to Elasticsearch, You can check it out at: https://github.com/siddontang/go-mysql-elasticsearch
另一个是这个:https://github.com/noplay/python-mysql-复制.
但是请注意，无论您选择哪种方式，在索引 binlog 之前预先创建索引和映射都是一个好习惯.这让您可以更好地控制自己的数据.
Note, however, that whichever you pick, it's a good practice to pre-create your index and mappings before indexing your binlog. That gives you more control over your data.
更新:
这是关于该主题的另一篇有趣的博客文章:如何使用 Logstash 保持 Elasticsearch 与关系数据库同步
Here is another interesting blog article on the subject: How to keep Elasticsearch synchronized with a relational database using Logstash
相关文章