Elasticsearch 7

Below quotes come from this article (October 2019).

Elasticsearch started life as a document database sitting atop the Lucene text search engine library. It was soon joined by related applications, and the preferred acronym for the Elasticsearch family of products was ELK: Elasticsearch; Logstash, the data pipelining tool, principally used to hoover data from logging into an Elasticsearch database; and Kibana, the data visualization construction kit.

The ELK trio has since been joined by a small platoon of “data shipper” utilities: the Beats products. Similar to Logstash, the Beats products move data from an outside source into an Elasticsearch database. They differ in the source of the shipped data. Filebeat is designed to read and forward the contents of log files (like Logstash, but without Logstash’s transformation and aggregation capabilities). Metricbeat reads system metric data gathered from Windows, Mac, or Linux hosts. Metricbeat can also gather enterprise application metrics from Microsoft SQL Server, MySQL, PostgreSQL, and other sources.

Aha, SQL databases can now also be imported.

Kibana improvements
Finally, it is worthwhile to pause a moment and discuss Kibana, because it has grown from being just a data visualization tool to a kind of Elasticsearch uber-dashboard. Kibana was originally a platform for creating and displaying real-time data visualizations – line graphs, bar charts, pie charts, etc. – drawn live from an Elasticsearch database. Though still a visualization builder, Kibana now provides consoles for management, development, machine learning, data exploration, and much more. For example:

• From Kibana’s index management console, you can display stats such as field mapping (index schema), index summary metrics (number of documents, on-disk storage consumed, etc.), index default parameters (number of shards, nested field limits, etc.), and more.
• The Canvas workpad area lets you create content for Canvas. A complementary visualization tool, Canvas can create Kibana-style data graphics, arrange them in a form suitable for presentations, and export them to file formats for import into presentation or slide-deck software.
• The Maps console lets you process and display geographical data (the console opens to a world map). Data is displayed as layers veneered atop and bound to the mapping structure below, so you can do things like set the color of a region (state, country, etc.) based on, say, the number of documents in a data set whose geospatial coordinates are in that region.
• The development tools dashboard provides a console for entering and executing queries against a database. (This is what used to be available as the widely popular Sense plug-in for the Chrome browser.) The dev tools console can also provide the performance of various internal database components.

These are just a sampling [of] all of Kibana’s features[…]

While Elasticsearch is evolving rapidly, its evolution is not simply engineers ladling out a stew of new features. They are also upgrading infrastructure in ways that both improve cluster performance and simplify an otherwise complex cluster configuration process.

This is not to say that properly configuring an enterprise-scale Elasticsearch cluster has suddenly been made straightforward; there are numerous considerations to be accounted for. But it is true that installing and running a working Elasticsearch database that’s suitable for development is surprisingly easy. And the new features that have appeared with Elasticsearch 7 make the trip from that initial installation to a distributed, enterprise-level, multi-node Elasticsearch cluster a much swifter journey.