Manage your Elasticsearch rivers with Sluice

As you may have noticed, I am an addicted user of Elasticsearch and have already written some river plugins for indexing different data sources such as Amazon S3 buckets or Google drives.

As a river plugins developper, I usually find myself in situations where I should test many Elasticsearch version and configuration combos – for that I have to reinstall fresh copies of my plugins. Also, as a river user, I have to create many rivers – and it’s often a trial and error process in order to find the correct configuration in regards of content to index, data freshness, index and request performance, mapping issues and so on…

So this lead me to the point that I felt tired about those long CURL commands needed to configure all this stuffs and decided to react ! Today I’m introducing you a new Elasticsearch plugin I’ve wrote to make my life easier (and maybe yours if you’re using rivers too ;-)): Sluice !

Features

So what’s Sluice ? As stated above, Sluice is also an Elasticsearh plugin whose goal is to help you manage your rivers : it simplifies installation of the required plugins but also helps you setup and tune your rivers. The idea with Sluice is to no longer have CURL commands to type, just install the Sluice plugin and then use its simple User interface.

Sluice is hosted on Github and install as a regular plugin by typing the following command in a shell :

$ bin/plugin --install com.github.lbroudoux.elasticsearch/sluice/0.0.1

Now restart your Elasticsearch instance and point your browser to http://localhost:9200/_plugins/sluice. You should see the following dashboard appear.

sluice-dashboard

You see here that Sluice checks installed river plugins among supported ones and offers simple way to install river plugins not already installed. Just click the Install link and its cares about retrieving and setting up Amazon S3 river plugin for example. You just then need to restart your ES node.

Picking up the dedicated section, you may also have the list all the river instances created into your ES cluster. For now, you can just edit and modify existing river – not remove them.

sluice-rivers

Finally, it offers a convenient way to add a new river. Configuration attributes of the river are grouped together with clear explanation of its meaning and supported format.

sluice-river-edit

Easy, no ? For the moment, supported River plugins are :

  • Amazon S3 River plugin,
  • Google Drive River plugin

Limitations

Sluice has only a first release named 0.0.1 and it’s far from being feature complete !
The current limitations are :

  • Only work with local development instances (yep ! http://localhost:9200 is hard coded… so ugly ! :-()
  • No way of removing rivers,
  • No way to start/stop rivers,
  • ES reboot is required after plugin installation

Future plans

Many useful features come to my mind – the order of the list has no relation with priority :

  • Configuration of Elasticsearch cluster endpoint,
  • Ability to remove or duplicate rivers,
  • Support of other river plugins such as the excellent FSRiver or the TwitterRiver,
  • Ability to start/stop or force a refresh of river settings while running,
  • Ability to get the CURL command of river for recreating it later (useful when tuning has been done in Dev or QA and that river creation should be scripted on production),
  • Rivers indexing statistics on dashboard !

Do not hesitate giving me feedback and sharing your feature ideas for future release of Sluice !

Advertisements

2 thoughts on “Manage your Elasticsearch rivers with Sluice

    1. Actually, the plugin only supports ES 1.3.x like Amazon-S3 and Google-Drive rivers do. Rivers were broken in first releases of ES 1.4.x so upgradre of plugins could not be done … I have not checked latest revisions nor have enough time lately to check ES 1.5.x. Hope to find so time in fourthcoming weeks.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s