Version 4.0 of Echo changes some fundamental ways the Stack operates to provide increased performance, efficiency and use-case support.

With +2000 commits since the release of Echo 2.0.0 LTS, we are proud to announce the release of Echo 4.0.0 GA, based on Elastic Stack 5.2.0. This is the latest stable release and is available as LTS throughout mid 2019.

Performance and user experience

Indexing performance

Indexing throughput has improved dramatically thanks to a number of changes in the underlying datastore including better numeric data structures, reduced contention in the lock that prevents concurrent updates to the same document, and reduced locking requirements when writing data to the transaction log. An internal change to how real-time document fetching is supported means there is more memory available for the indexing buffer and much less time spent in garbage collection.

Depending on your use case, you are likely to see somewhere between 25% - 35% improvement to indexing throughput.

Faster user experience

Echo is designed to provide a fast user experience to security analysts and operators and version 4 takes things even faster with 20% reduction in navigation and query time (average).

We have reworked key parts of the user interface to offer a faster navigation and usage of Echo. The underlying data stores were reviewed, optimized and further tweaked to offer shorter index time with increased ingestion load.

Search and Aggregations

Those beautiful charts which graph some security metrics over the previous day, month, or year, are going to get a serious speed boost with Instant Aggregations. Most of the data in those charts comes from indices which are no longer being updated, but our datastore had to recalculate the aggregation from scratch on every request because it wasn’t possible to cache a range like { “gte”: “now-30d”, “lte”: “now” }. Now the datastore is much cleverer about how a range query is executed, and will now only recalculate the aggregation for indices that have changed.

Aggregations have seen more improvements: histogram aggregations now support fractional buckets and handle the rounding of negative buckets correctly, terms aggregations are calculated more efficiently to reduce the risk of combinatorial explosion.

On the search side, the default relevance calculation has been changed from TF/IDF to the more modern BM25 to ensure better matches to your security queries.

New and updated visualizations

Metric visualization

The Metric visualization has been updated to offer color coding based on the displayed value.

Also, it now supports calculated metrics allowing the display of rates and ratios at ease.

Swimlanes visualization

This version introduces the Swimlanes visualization which offers a unique type of heatmap to allow faster identification of patters and out-of-band events.

Timeline visualization

The Timeline look & feel has been updated and more options added to help you visualize events over time.

Sankey visualization

The Sankey visualization now offers quick filtering by clicking on any of the nodes.

Circuit breakers

We have added a circuit breaker which limits the amount of memory which can be used by in-flight requests, and expanded the request circuit breaker to track the memory used by aggregation buckets and to abort pathological requests which request trillions of buckets. While an out-of-memory exception is much less likely than before, if one does occur, the node will now die-with-dignity instead of limping along in some undefined state.

Datastore Resiliency

There are a host of changes that have gone into this release to make Echo safer than ever before. Every part of the distributed model has been picked apart, refactored, simplified, and made more reliable. Cluster state updates now wait for acknowledgement from all the nodes in the cluster. When a replica shard is marked as failed by the primary, the primary now waits for a response from the master. Indexes now use their UUID in the data path, instead of the index name, to avoid naming clashes.

Security and multi-tenancy

This version placed special emphasis on security and multi-tenancy.

Echo certificate authority

Echo uses Docker and micro-services as its infrastructure, in order to ensure all of the stack parts communicate securely we have added our CA as part of the stack to issue both internal and external interfaces with TLS certificates.

External facing certificates can easily be exchanged for in-house ones using the different configuration options.

TLS everything

All Echo services and endpoints now require TLS and are served in an encrypted manner to prevent data leakage and unauthorized usage.

Secret management

Echo now provides REST API for secret management, you can safely store secrets in Echo. Secrets are encrypted using aes-256-ctr and offer both minimal latency times combined with enhanced security.

We use secret management as part of the stack to ensure all passwords, sensitive data, certificates, etc… is safely stored outside the reach of unauthorized access.

Object level permissions

One of the most asked feature by our users has now landed in version 4. Controlling who can view/edit/delete objects could not be easier. Roles are assigned with permissions specifying which operations they’re allowed to perform on objects such as dashboards, saved searches, visualizations and more.

Event level permissions

Users can now apply event level permissions and offer multi-tenancy by limiting which roles can view and edit raw data events. Event level security is based on already known filters which are applied on a role level and follow users where ever they navigate within the system. A simple example may be limiting a SOC operator to view only Security Alerts related data while keeping sensitive employee data hidden.

Single-sign-on

As part of our efforts on Echo’s security model and secret management, we now support Single-Sign-On (SSO). Currently only SAML is supported, but more providers (including custom) will be added in the near future.

Users can setup SSO using the Management section.

Breaking changes

The following breaking changes are part of the release of version 4.

Vector map visualization removed

Due to poor user adoption, we have decided to remove the vector map visualization and review if and what vector visualizations would fit with our users and serve their data exploration best.

Strict mappings

Up to version 4, Echo used a loose mappings approach in which all data fields were indexed and made available for search. This lead to increased hardware and data storage requirements which based on our and our customers’ experience are not cost effective. We have decided to move to a strict mappings approach, meaning that only modeled data fields are indexed and available for free-text search. It doesn’t mean that we throw the data away, all data fields are included as part of the event, the change only affects the ability to free-text search on these fields.

Users can manually opt-in to index additional data fields using the Management section.