In our series on the Go client for Elasticsearch, we've covered the architecture and design of the client and how to configure it. In this post, we’ll look at encoding and decoding JSON payloads and efficiently indexing documents in bulk.

blog-banner-board-game-go.jpg

Deep dives

Elasticsearch

Data ingestion

Search applications

Search analytics

Real-time analysis

Deployment

Cloud search

OpenTelemetry

Visualization

Relevance

Search

blog-thumb-board-game-go.jpg

The Go client for Elasticsearch: Working with data

Find out how Siren uses the real-time capabilities of the Elastic Stack to power their investigative intelligence platform. Learn how they use Elastic to tackle big challenges: tracking relationships between diverse data and diverse analyst needs.

At Siren, we build a platform used for “investigative intelligence” in Law Enforcement, Intelligence, and Financial Fraud. Investigative intelligence is a specialisation of data analytics that serves the needs of those that are typically hunting for bad actors. Such investigations are the primary focus of law enforcement and intelligence, but are also critical to uncovering financial crime activities and for threat hunting in cybersecurity.At the heart of Siren is the Elastic Stack. With its real-time responses and ability to deal with ever-increasing amounts of structured and unstructured data, <a href="/elasticsearch">Elasticsearch</a> provides the ideal fabric for investigative analysis. Furthermore, thanks to the Siren and Elastic partnership, all the advanced features of an Elastic Platinum subscription can be used in Siren as part of US Federal Siren deployments, giving investigators unprecedented flexibility, capabilities, and operational resilience.At Siren, we set out to use Elasticsearch to tackle two of the biggest problems in investigative intelligence: disjointed data and disjointed tools. With that in mind, in this blog post, we would like to share part of our approach with the broader community of technologists and architects who are trying to get the most out of their data.<h2>Two challenges for modern investigative analytics</h2>At the data level, the main challenge in investigative intelligence is disjointed data: investigators need to be able to follow non-obvious relationships across a myriad of diverse data sets and data sources.&nbsp;A more subtle challenge, however, is the diversity of the analysis needs, which traditionally forced organizations and analysts to either use many disconnected tools and backends (APIs) or embark on building expensive and rigid ad-hoc integrations.&nbsp;For example, investigators certainly need link analysis — to see and explore the connections between records — but they also need fast drilldowns, business intelligence (BI)-style visuals, and text search and analysis for unstructured data.&nbsp;Our challenge from the onset has been: How can we deliver a unified investigative analytics platform that is also architecturally modern and easy to deploy?<h2>Stepping up to the challenge</h2>With its real-time responses, its array of <a href="/guide/en/elasticsearch/reference/7.9/search-aggregations.html">real-time analytics functions</a>, and its powerful search engine — capable of dealing with fuzzy searches and noisy data — Elasticsearch immediately stood out as the backend providing the ideal starting point for investigative analysis.That is why we built the Siren platform — a unified tool that enables big, disconnected data analytics — on top of Elasticsearch and Kibana.<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt2d030b406ac6417a/5f761aeddac9270efe35f0c7/blog-siren-platform.png" data-sys-asset-uid="blt2d030b406ac6417a" alt="The Siren platform" style="display: block;margin: auto;width: 500px;" width="500"/><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt189f1fb680ad09c5/5f6a6a252b93e84e4c32a297/blog-siren-graph-analysis-example.jpg" data-sys-asset-uid="blt189f1fb680ad09c5" style="display: block;margin: auto;"/><h2>Analysing disjointed data in Elasticsearch: Data model, joins, and link analysis</h2>The investigative world is made up of disjointed data that needs to be connected. People (for example) are connected to vehicles they own, which are connected to locations where they’ve been, which may be connected to events, and so on.In Elasticsearch these are typically recorded in separate indices, possibly coming from all sorts of sources. Siren leverages the real-time speed of Elasticsearch to tie this data together for investigators, regardless of source or index.<h3>Tying data together with an associative data model</h3>In Siren Investigate — Siren’s frontend built on <a href="/kibana">Kibana</a> — administrators or advanced analysts define an Associative Data Model on top of their existing data, and this data model then drives all the analytic operations.&nbsp;The Data Model editor is visual, allowing you to define how tables are interconnected, typically by shared keys, which are then used to join the records. For example, an Associative Data Model in law enforcement can be defined to connect tables which contain persons with vehicles, cases, automatic camera licence plate readings, and more.One uses the visual editor to specify the primary and foreign keys to be used as associations. For example, here are the connections for the Crimes index:<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltc03f34ff4e2c9684/5f6a6a50b038186a244ba518/blog-siren-visual-editor.jpg" data-sys-asset-uid="bltc03f34ff4e2c9684" style="display: block;margin: auto;"/>And the overall model can also be seen as a single picture, such as this graph visualization of the connections between persons and other entities.&nbsp;<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltdc27c3420fabf654/5f6a6aab128a9c4e4d50c138/blog-siren-analyzed-emails.jpg" data-sys-asset-uid="bltdc27c3420fabf654" style="display: block;margin: auto;"/><figcaption>A view of the connections between persons and other entities</figcaption>In another example — a cybersecurity scenario — it is common to use concepts such as IPs, MD5 hash values, emails or user IDs to tie together security logs. The following screenshot shows the relationships between different IPs.<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltcdbf230d9b2e27d3/5f6a6a80128a9c4e4d50c134/blog-siren-ip-relations.jpg" data-sys-asset-uid="bltcdbf230d9b2e27d3" style="display: block;margin: auto;"/><figcaption>A view of the relations between different IPs</figcaption>All the examples above share specific identifiers, but, as we’ll see later, “fuzzy” relations can be similarly accounted for (e.g., relations coming from Natural Language Processing (NLP) or Entity Resolution).<h3>Data model-powered associative drilldowns (and link analysis)</h3>The data model enables a special kind of investigative capabilities called associative drilldowns and link analysis&nbsp;investigations.Let’s see this in action with a financial investigation example, where we have companies that have received investments by investors, as well as articles that mention companies (and often their investments). This is represented by the following data model:<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blta49497ea09bfc836/5f6a6a66b038186a244ba51c/blog-siren-articles.jpg" data-sys-asset-uid="blta49497ea09bfc836" style="display: block;margin: auto;"/><figcaption>Here, the articles-to-companies relation comes from the NLP engine.</figcaption>Thanks to the data model in Siren, we can drill down based on what’s connected to a set of records.&nbsp;For example, in the screenshot below, the relational navigator, the user sees in real time how many records are connected (351,243 articles and 41,298 investments) to the current set and is navigable with the click of a button.&nbsp;<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltfb61ae1cd1578159/5f6a6af394533468847809f2/blog-siren-drilldown-1.jpg" data-sys-asset-uid="bltfb61ae1cd1578159" style="display: block;margin: auto;"/>Drilling into investments in the relational navigator button brings us to a tailored dashboard, where we can drill down further.<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt3a50ef7f9d89073b/5f6a6b0a8350724defa21ab0/blog-siren-drilldown-2.jpg" data-sys-asset-uid="blt3a50ef7f9d89073b" style="display: block;margin: auto;"/>The process can be then repeated — for example, to move from here to the 2,535 investors who made these investments in 2012.Under the hood, these real-time interactive associative capabilities and the relational button make use of the <a href="https://siren.io/elasticsearch-investigative/">Siren Federate Plugin</a>, which extends the Elasticsearch query DSL to include cluster scalable join/correlation capabilities.&nbsp;Siren Federate also enables working across different backends: it has a series of drivers that can see data in remote backends as if they were in Elasticsearch (<a href="https://www.linkedin.com/pulse/alternatives-etl-elasticsearch-virtualize-reflect-varun-sharma" target="_self">virtual indices</a>).<h3>Pivoting to graph mode: Siren Link analysis</h3>Being able to do associative drilldowns is great, but there are questions that no dashboard can answer.&nbsp;For example:<ul><li>Which investors invested in which company?</li><li>Are they investing together or in groups?</li><li>Are there groups that appear to be investing in competing companies?</li></ul>For these questions, Siren's ability to move from dashboards to link analysis is key.I simply dragged and dropped the filtered Investments dashboards and I can quickly see how they connect<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltdf679f6942e75875/5f6a6b75f0d2b04fae76ca69/blog-siren-link-analysis.jpg" data-sys-asset-uid="bltdf679f6942e75875" style="display: block;margin: auto;"/><h4>Elasticsearch aggregations for big (graph) link analysis&nbsp;</h4>Elasticsearch aggregations can be used on demand to summarize the graph. The sidebar of the link analysis visualization allows you to choose the aggregation criteria to display edges which summarize (e.g., count/rollup) all the nodes between two entities.&nbsp;In this example we’re counting the number of articles which co-mention the two companies but also the significance of the co-mention — as outputted by the incredibly useful <a href="/guide/en/elasticsearch/reference/7.9/search-aggregations-bucket-significantterms-aggregation.html">significant terms</a> aggregation in Elasticsearch.<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltaefb9e9523ec374e/5f6a6b20a4d7b56a23867264/blog-siren-agg-link-analysis.jpg" data-sys-asset-uid="bltaefb9e9523ec374e" alt="blog-siren-agg-link-analysis.jpg"/> <figcaption>Link analysis: Articles displayed as aggregate links between companies</figcaption><h3>Efficient shortest path queries in Elasticsearch</h3>Finding the shortest and most significant path across connected records (phone calls, messages, social links) is a typical example of a widely used investigative graph algorithm.Efficient shortest path in Elasticsearch is another operation made possible by the Siren Federate plugin technology. <a href="https://www.youtube.com/watch?v=UYlubDhpHGA&feature=youtu.be">Here it is in action finding connections between two users, six phone hops away</a>.Another very important investigative graph algorithm is the ability to find a “common communicator” among nodes. In the following screenshots we find that a common communicator exists between these three companies.&nbsp;<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltdd267d30223bad85/5f6a6bdf5e85ad4dee190e17/blog-siren-shortest-path.jpg" data-sys-asset-uid="bltdd267d30223bad85" style="display: block;margin: auto;"/><figcaption>Identifying Microsoft as the common communicator as mentioned in articles with other companies</figcaption><h3>Web services and advanced geo/time/spatial analysis</h3>Sometimes it’s not possible to get an answer with a simple operation, such as with a shared key join.&nbsp; Siren supports calling remote web services and fitting their results back in the data model.&nbsp;This capability can be used in many ways — for example, to pull in data on demand (e.g., to access remote knowledge or reference data) and ask for advanced computations on demand.&nbsp;Let’s take for example a COVID-19 simulation scenario: have two phones been in physical proximity for more than 15 minutes? In the next screenshot, Siren is configured to use a web service (which implements the complex logic required to deal with noisy and spotty data) and makes the results available for analysis.<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt0c091d1aabd3f757/5f6a6bf7a4d7b56a2386726a/blog-siren-geo-dashboard.jpg" data-sys-asset-uid="blt0c091d1aabd3f757" style="display: block;margin: auto;"/>As expected, geo time analysis is critical in investigative intelligence, and Siren builds on the extraordinary <a href="/maps">geo capabilities of Elasticsearch</a> to provide this in an analyst-interactive way. The following screenshots illustrate some of the capabilities, which include graph over time evolution and analyst-activated Elasticsearch stored layers.<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt4aab0edb1e52660e/5f6a6c0c6f40ab4648b5e50b/blog-siren-geo-world.jpg" data-sys-asset-uid="blt4aab0edb1e52660e" style="display: block;margin: auto;"/><figcaption>Using the timeline mode to view spatial and temporal data</figcaption><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt51aa4a14c3844db9/5f6a6c254a19ef4fa853eee3/blog-siren-geo-gb.jpg" data-sys-asset-uid="blt51aa4a14c3844db9" style="display: block;margin: auto;"/><h2>Conclusion&nbsp;</h2>Elasticsearch is the ideal centerpiece backend for large-scale, interactive structured and unstructured data analysis. It was a natural foundational choice for Siren in its mission to provide a unified intelligence analytics experience — and connect disjointed data.Interested? Try Siren now with our freely available <a href="https://siren.io/downloads/">Siren Community Edition</a> and our nice <a href="https://siren.io/getting-started/">getting started tutorial</a>.&nbsp;<h2>Learn more</h2><a href="/webinars/elastic-and-siren-protecting-people-assets-and-networks">Elastic and Siren: Protecting people, assets, and networks (video)</a><h2>About Siren</h2><a href="https://www.siren.io">Siren</a> provides investigative intelligence based on Elasticsearch to some of the world’s largest and most complex organizations.&nbsp;<h2>About Dr. Giovanni Tummarello</h2>Giovanni Tummarello, Ph.D is a Computer Scientist and entrepreneur, co-founder and Chief Product Officer at <a href="https://www.siren.io/">Siren.io</a>. He led the team at the National University of Ireland Galway <a href="https://scholar.google.com/citations?user=x-8b-ZkAAAAJ&hl=en">researching</a> on Knowledge Graphs, Search Engines and related UI/UX which then spun off into Siren. Previously, while at the FBK Institute in Trento Italy he led a Semantic Web team and co-founded business information company Spaziodati.eu.&nbsp;

blog-banner-generic-black.png

Customers

Customer story

Financial services

Government

Graph analysis

Security

blog-thumb-siren.png

Investigative analysis of disjointed data in Elasticsearch with the Siren Platform

Find out how JetBrains used .NET, Elasticsearch, and Kibana to analyze and visualize their community's past as well as make predictions about the future.

blog-banner-jetbrains-es.png

Kibana

Software & technology

Visualizing

Ingesting

Log analytics

Observability

Business analytics

Stack

blog-thumb-jetbrains-es-2.jpg

How JetBrains uses .NET, Elasticsearch, CSVs, and Kibana for awesome dashboards

Check out Elastic’s newest 2020 Elastic Search Awards honorees for Public Sector, EMEA and APJ.

Every day at Elastic we're hard at work building better products, solutions, and features. What inspires our innovation, and humbles us at the same time, are the exceptional projects and solutions the community has built using Elastic.
We're excited to recognize some of those extraordinary use cases as part of our regional Elastic Search Awards program.
The Elastic Search Awards categories for the Asia-Pacific-Japan (APJ) and Europe-Middle East-Africa (EMEA) regions include:
<ul>
 <li>Cause Awards for bettering the world</li>
 <li>Cluster Awards for inspiration and uniqueness</li>
 <li>You Know, For Search! Awards for a project's potential for growth and for fostering a breakdown of data silos</li>
</ul>
The<a href="/events/awards/winners"> Elastic Search Awards</a> honorees were announced during our virtual, three-day ElasticON Global event. We also honored the EMEA Elastic Certified Professional of the Year for exceptional contributions to the Elastic community. Joining our regional honorees were the global honorees of our Public Sector Search Awards. We <a href="/blog/introducing-the-2020-elastic-search-awards-honorees-for-the-americas-region">introduced</a> the Elastic Search Awards honorees for the Americas region in March.
"The originality demonstrated by the Elastic Search Awards nominees never fails to impress. It's always a challenge to select the honorees, and this year's applicants really put us to the test with an array of innovative contributions and Elasticsearch use cases,” says Madison Bahmer, the chief technology officer of <a href="https://www.istresearch.com/">IST Research</a> and member of the Elastic Search Awards <a href="/events/awards">judging panel</a>.
<h2>Elastic Certified Professional of the Year</h2>
We honored Peter Steenbergen, 37, of the Netherlands, as our 2020 Elastic Certified Professional of the Year for the EMEA region. Steenbergen has earned <a href="/training/certification/faq">Elastic certification</a>, harnessed Elastic for his own projects, and influenced the <a href="/community">Elastic community</a> by sharing and championing his Elastic knowledge. He's also the co-founder of <a href="https://uptimemate.com/">UptimeMate</a> and is an Elasticsearch consultant. Steenbergen says, “Elastic changed my life after being certified. I get questions from all around the world to help people with their problems or to help them get started with Elasticsearch.”
Rich Raposa, manager of the Elastic Certification Program and one of the Elastic Search Awards judges, says Steenbergen was one of the very first people to pass the <a href="/webinars/preparing-for-the-elastic-certified-engineer-exam">Elastic Certified Engineer</a> exam. “Ever since then I have been impressed with his enthusiasm and contributions. He is an active member of the Elastic community, and his Elastic certifications give him the credibility he so rightfully deserves,” Raposa says.
Watch the <a href="/videos/emea-elastic-search-awards-2020-certified-professional-of-the-year">video</a> of Peter's story.
<h2>APJ awards</h2>
Cause Award: <a href="https://www.blackdoginstitute.org.au/research/data-analytics/digital-phenotyping/">Black Dog Institute</a>, Australia. The Black Dog Institute is using Elastic in a project called “Instil” to predict the onset of mental illness in young people with data sent from volunteers' cell phones and stored in Elasticsearch.
Cluster Award: <a href="https://corp.wise-vine.com/">WiseVine</a>, Japan. WiseVine has created a database, built with Elasticsearch, to connect governments and private enterprise at the early stages of the public procurement process for better policy planning, and for a more equitable public procurement process.
You Know, For Search! Award: <a href="https://www.telkom.co.id/sites">Telkom Indonesia</a>, Indonesia. Telkom Indonesia, the largest telecommunications provider in Indonesia, is transforming into a digital telco, and has created a self-service Open API Platform with open source software and Elasticsearch as a means to provide accelerated time-to-market and streamlined user onboarding.
Watch the <a href="/videos/apj-elastic-search-awards-2020-honorees">video</a> of the APJ honorees tell their stories.
<h2>EMEA awards</h2>
Cause Award: <a href="http://www.scio.systems">SCiO</a>, Greece, and <a href="https://www.cgiar.org/">CGIAR </a>of France. SCiO provides AI-enabled services for the disruption of the agri-food value chain, and has developed <a href="https://gardian.bigdata.cgiar.org/">GARDIAN</a>, a data discovery framework built using Elasticsearch. The GARDIAN framework supports the <a href="https://bigdata.cgiar.org/">CGIAR Platform for Big Data in Agriculture</a>, a large-scale initiative to unlock important research publications and data sets about food security, nutrition, and natural resources.
Cluster Award: <a href="https://www.cedint.upm.es/es">Centro de Domótica Integral de la Universidad Politécnica de Madrid</a>, Spain.&nbsp;
Centro de Domótica Integral de la Universidad Politécnica de Madrid (CeDInt-UPM) researchers have developed an Internet of Things (IoT) platform using the tools provided by the Elastic Stack to control smart buildings, smart lighting, and smart greenhouses.&nbsp;
You Know, For Search! Award: <a href="https://www.intelligentplant.com/">Intelligent Plant</a>, United Kingdom. Intelligent Plant provides an open and secure Industrial Internet of Things (IIoT) portal known as the Industrial App Store. Elasticsearch provides the Industrial App Store with secure remote performance monitoring of equipment and processes through the analysis and visualization of real-time data.&nbsp;
Watch the <a href="/videos/emea-elastic-search-awards-2020-honorees">video</a> of the EMEA honorees tell their stories.
<h2>Public Sector awards</h2>
Cause Award: <a href="https://www.nationalcrimeagency.gov.uk/">National Crime Agency</a>, United Kingdom. The National Crime Agency (NCA) is using Elasticsearch to help maximize its understanding of the crimes it investigates. By doing so, this will then help provide a fuller picture of the crime types and enable even better informed decision-making around its operational activity.
Cluster Award: <a href="https://www.stormfish.io/page/homepage">Stormfish Scientific</a>, United States. Stormfish Scientific relies on the Elastic Stack to develop its auroraXR™ platform for building secure, information-rich, multi-user, cross-reality environments. auroraXR™ enables military scientists and engineers, academia, and commercial partners to explore human cognitive processes to better understand how virtual reality and augmented reality technologies impact accuracy, timeliness, and confidence for empowering accelerated, mission-critical decision making.
You Know, For Search! Award: <a href="https://www.leidos.com/">Leidos</a>, United States. Leidos, a defense, aviation, information technology, and biomedical research company, has developed a data discovery platform, built with the Elastic Stack, to make petabytes of sensitive mission data searchable and available to US military and intelligence agencies.
Watch the <a href="/videos/public-sector-elastic-search-awards-2020-honorees">video</a> of the public sector honorees tell their stories.
Congratulations to all of the Elastic Search Awards honorees!

elastic-awards-blog-banner.png

ElasticON

Community

elastic-awards-blog-thumb.png

Congratulations to our newest 2020 Elastic Search Awards honorees

Find out how Comillas University in Spain has been using Elastic Cloud to make sure social distancing doesn't impact their students' ability to learn.

Our new <a href="/community/students-and-educators">Elastic for Students and Educator program</a> provides online resources and support to help you teach and learn no matter where you are. Hear from <a href="https://www.linkedin.com/in/lmerchante/">Luis Francisco Sánchez Merchante</a>, an educator based in Spain, as he reflects on the challenges he’s faced while teaching during a global pandemic.We live in strange days. We feel, as never before, like characters in a disaster movie. We have been affected by pandemics that spread across the world before, but it might be the first time that it has occurred during a time of well-being and technological superiority that made us feel immune. This is an opportunity to reconsider our priorities and an occasion to become aware of our position in the world. But we’ve paid a high price and we’re still paying it.Our life has been altered to extremes that none of us could have imagined just a year ago. We have been forced to change our routines. Changing habits in the personal sphere will be tough and it will take time to get used to. Unfortunately, we don’t have that time in the professional sphere.<h2>Too many questions</h2>The tourism sector in Spain reached 12.3% of the GDP in 2018. Because of its importance, any butterfly effect around this sector can result in a nationwide hurricane. We can imagine that this sector will receive special attention to make this transition as painless as possible. But what will happen to other less visible sectors?Education is an example of one of those sectors of vital importance that is not considered to be a driving force of the economy. So how will the current situation affect schools? Even more importantly, how will this situation affect education itself? Nurseries, primary and secondary schools, and universities are called upon to carry out deep changes — not only at the level of infrastructure but also, and above all, to their education methodology. The success of many academic plans is supported by personalized attention and the proximity between teachers and students: building a motivating environment that favors knowledge generation or leveraging contact networks in the academic community. Can we replace face-to-face education with a completely remote model?I guess it’s feasible, but it will require a huge effort. Replacing classroom lectures with Zoom or Teams sessions allowed us to overcome the semester with the least possible disruption for students. But this situation may be prolonged. Even if science finds the means to defeat the virus, there is no guarantee that in months or years a similar situation will not reoccur. Changes must be global. We might be living in a time where the trend that technology followed — favouring delocalisation over centralised systems — starts to apply to education.<h2>Cloud providers, educational partners</h2>Remote education is not only about updating the transmission channel or about keeping the same schedule and educational resources as before, but doing so with the student at home. The advantage of remote learning is that students can manage their own time by accessing class materials and completing requested tasks autonomously. This implies that if certain infrastructure is needed for a lab (access to a database, for example), having those servers up and running only during class is no longer sufficient. Nor is leaving the servers up indefinitely an alternative, because that will certainly be a waste of resources.The model that best suits remote education involves cloud technologies. Being able to start or stop services remotely with automatic resource management and deployments that can scale according to demand is the reason why cloud providers appeared. Most educational institutions do not have the capacity to build and administer their own data processing center. Since academic institutions are not money-making machines, collaboration between IT companies and academic institutions is the desired scenario.With the quick evolution of IT products (especially those in the big data domain), working on cloud makes it extremely easy to use those products in academic programs. Many products get major upgrades at least once a year, not only to fix bugs, but also to include new features or improve the user experience. After a few upgrades, the version referenced in the training material may have little to do with the current version that is used in the industry. This isn’t the worst situation; sometimes academic programs can be using legacy or deprecated software that students will never see during their professional life. But cloud providers offer a great solution for this issue. Due to their virtual nature, most cloud updates are not disruptive because new updated nodes can be added before removing old ones without stopping the service. This can also be accomplished on premise, but if you have been in that position, the technical, human and administrative complexity to schedule a major upgrade makes many companies ignore or postpone them sometimes indefinitely.An encouraging example of the direct collaboration between IT companies and academic institutions is the relationship between Elastic and <a href="https://www.comillas.edu/en/">Comillas University</a>. I’m a lecturer at the engineering school <a href="https://www.comillas.edu/en/icai">ICAI</a>, where we are teaching data visualization in several degrees and postgraduate courses. The purpose is that students learn to distinguish use cases and use the appropriate visualization tools. Even though everyone is aware of the most popular dashboarding applications that aim to cover most of the industry’s needs, it is precisely this completeness of vision that can make them waver in some areas.One of the most paradigmatic use cases in data visualization is the creation of dashboards for monitoring. Most versatile tools are not well equipped to present information that is updated in real time. However, this is highly demanded by the industry. If our students run into any of those use cases, we want them to have the expertise to make the best choice.I have always considered that collaborating with universities is the best investment that companies can make. During my time in the industry I tried to put it into practice. Not only does the community perceive these companies as entities that care about building future generations, but also students have the opportunity to become familiar with their products.As a recent graduate, if you have to select a cloud or a database provider, it is likely that your experience as a college student will influence your decision. Those were my thoughts as a rookie teacher when I joined the university two years ago. I was willing to establish cooperation agreements with as many companies as possible to provide a quality, industry-linked education for our students. Most of the outreach I made never materialized. But one of the few exceptions that saw the potential of this collaboration to raise awareness of their products was Elastic. The subject of data visualization was part of the master’s degree in Big Data Technologies and Advanced Analytics, and when the program launched there were only 25 students. From the beginning, Elastic provided extended use of <a href="/cloud">Elastic Cloud</a>, the official service for running the Elastic Stack in the cloud, without any intrusive ads or tradeoffs. Two years later, interest in data visualization has grown and is taught in several programs. In the last edition, around 200 students made use of Elastic Cloud to run <a href="/elasticsearch">Elasticsearch</a> and <a href="/kibana">Kibana</a>.<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt123ce97b9ba00f60/5f11d1f17523cd1c482ba8f9/blog-cloud-ed-kibana-dashboard.jpg" data-sys-asset-uid="blt123ce97b9ba00f60" alt="Some examples of Kibana dashboards created by our students " style="display: block;margin: auto;"/><figcaption>Some examples of Kibana dashboards created by our students</figcaption>This deployment has not only allowed us to train our students in the use of Kibana, but having a cloud service during the recent restrictions also helped us to complete the semester without altering the content of the sessions.<h2>Kibana as a business and teaching tool</h2>Teaching telecommunication engineers or post-graduates in Big Data and Advanced Analytics in the use of these tools is expected. But data visualization is not only taught in engineering degrees and scientific postgraduate courses, but it is also part of some double degrees such as the degree in <a href="https://www.comillas.edu/en/degrees/dual-degree-program-in-business-administration-and-management-ade-and-business-analytics-e-2-analytics">Business Administration and Management and Business Analysis</a> or the degree in <a href="https://www.comillas.edu/en/degrees/dual-degree-program-in-business-analytics-and-law-e-3-analytics">Business Analysis and Law</a>. Since companies are getting more and more data centric, it’s not difficult to picture an executive downloading data from an API. Or a lawyer using SQL to query a database. Or a business analyst making predictions using a deep-learning algorithm.Their skills will be different than those of a computer engineer who received several years of training in software architecture and coding, but being able to do simple website scraping, or understanding the concepts behind a machine learning algorithm, or being able to build a dashboard with social network data can be of great support for their core business. And that’s the ultimate goal: that our students leave college better prepared, mastering actual industry tools that will make them better professionals.<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blte3d59db3b0de6f0d/5f11d1a15bbd1b7f3aca86f5/blog-cloud-ed-success.jpg" data-sys-asset-uid="blte3d59db3b0de6f0d" alt="Program success rates" style="display: block;margin: auto;width: 600px;" width="600"/><figcaption>Success rates of these programs</figcaption>See what tools and resources are available to you through our new <a href="/community/students-and-educators">Elastic for Students and Educators</a>&nbsp;program.

blog-banner-virtual-stack.png

Education & non-profit

Cloud

blog-thumb-virtual-stack.png

The benefits of cloud education in pandemic times

At our ElasticON US Federal event, organizations across the federal government shared how they are leveraging Elastic to achieve their mission. Read more about the event, including takeaways from the key sessions.

After a year of accelerated change in the federal industry, the <a href="/elasticon/public-sector/federal#overview">Federal Public Sector event</a> focused on the progress you’ve made, the trends demanding your attention, and the Elastic capabilities that continue to guide federal agencies, offices, and departments towards a search-enabled future.&nbsp;
With a theme of&nbsp;Accelerate the mission. Start with search, the ElasticON US Federal event centered around some key takeaways:&nbsp;
<ul>
	<li aria-level="1">Data volumes will increase regardless of the size, scape, or scope of your federal focus. Having the tools to adapt processes at speed and scale can be the difference between mission success or failure.</li>
	<li aria-level="1">Whether your focus is to enable business visibility, optimize infrastructure and applications, or protect your digital ecosystem, the power of search is the catalyst for becoming an insights-driven organization.</li>
	<li aria-level="1">Your security posture should center on a consolidated, unified analyst experience for security and observability in all operations. Elastic bridges the gap between all silos, operations, and domains to deliver a central hub for detecting, analyzing, and acting on all threats before they put the mission at risk.</li>
	<li aria-level="1">Federal-specific concerns of speed, networking, accessibility, tracking, and data security are built into the latest releases so that Elastic can bring more responsiveness than ever to public sector projects.</li>
</ul>Using the highlights below as a guide, learn how organizations across the federal government are leveraging Elastic to quickly analyze data, unify intelligence, standardize datasets, and achieve the mission faster.
 <h3>Elastic Security in action</h3>Easily the most spirited conversation of the event was <a href="/elasticon/archive/2021/public-sector/federal/elastic-security-in-action">Elastic Security in action</a>, facilitated by Elastic’s Mike Nichols and John Harmon. Bringing together perspectives from both product and mission with Chris Johnson (Director, Cybersecurity Operations Center, NGA), Armando Seay (Co-Founder & Member Board of Directors, Maryland Innovation Security Institute), and Ed Yardley (Chief Engineer, Valiant), the session provided a comprehensive analysis of the cyber, cloud, supply chain, and compliance capabilities enabled by Elastic Security and how they stand up to the requirements of the federal government of tomorrow. The key to scalability, integration, risk assessment, and threat hunting is an ecosystem of cyber capabilities that can communicate freely to improve response time.
 <h3>Empowering agencies using Elastic as a service inside government</h3>Elastic solutions architects Tristan Ahmadi and Jeremy Woodworth were joined by the Director of Cyber Technologies at ECS, Mike Zakrzewski, to discuss <a href="/elasticon/archive/2021/public-sector/federal/empowering-agencies-using-elastic-as-a-service-inside-government">how Elastic Cloud Enterprise (ECE) has impacted the Elastic public sector community</a> and how it can continue to grow to fit ever-evolving mission needs. From threat detection under tightening budgets to networking disparate systems and adopting a risk-based assessment mindset, Elasticsearch has proven to be the foundational element of management and shared service use cases within the federal government.&nbsp;
 <h3>Integrating advanced analytics with Elasticsearch</h3>In a solution application deep-dive, two members of Deloitte’s Analytics & Cognitive Applied AI practice detailed <a href="/elasticon/archive/2021/public-sector/federal/integrating-advanced-analytics-with-elasticsearch">how they used Elasticsearch to help a large federal agency expedite document review</a>. Applicable to a wide variety of eDiscovery use cases, this session highlighted the flexible and creative development possible when pairing the Elastic Stack with emerging tech (e.g., ML and NLP) to solve complex government client problems. By focusing on enabling human review as opposed to replacing it, Deloitte built a process that helped find documents more efficiently, learn patterns from user decisions, and be more transparent.
 <h3>Achieving cyber mission assurance with near real-time impact</h3>Matt Hackman (ECS) walked through <a href="/elasticon/archive/2021/public-sector/federal/achieving-cyber-mission-assurance-with-near-real-time-impact">how Elastic and ECS support the Mission Assurance Decision Support System (MADSS) program for the Navy</a>. Focusing on observability, data enrichment, and powerful search in a containerized environment, Hackman guided viewers through the development process, arriving at improved productivity and user acceptance in the MADSS. By correlating data from diverse sources using web-based services and secure, automated data transformation services, MADSS improves responsiveness, predictions, and event analysis for critical network and infrastructure outages.
 <h3>Fireside chat with Nicolas M. Chaillan, Air Force Chief Software Officer (CSO)</h3><a href="/elasticon/archive/2021/public-sector/federal/fireside-chat-with-nicolas-m-chaillan-air-force-chief-software-officer">In a final session</a>, US&nbsp;Air Force’s Chief Software Officer, Nicolas Chaillan, shared his thoughts with Elastic’s VP of Federal, Steven Coles, on the DoD Data Strategy and the DoD Enterprise DevSecOps Initiative (DSOP). From integrating DevSecOps across the services to the importance of having a 360-degree view of data, this fireside chat underscored how the shift to agile and infrastructure as code improve&nbsp;efficiency and provide&nbsp;more robust security across the DoD.
<h2>Continue the conversation</h2>Whether you were able to join us live or not, we want to help you keep the learning and community connections alive. We’re always happy to put you in touch with other public sector users to discuss use cases. Email us at <a href="mailto:federal@elastic.co">federal@elastic.co</a>&nbsp;or join in the <a href="https://elastic-uspubsec.slack.com/join/signup#/">Elastic Public Sector Slack workspace</a>, an online forum dedicated to the US&nbsp;public sector community.&nbsp;
If you'd like to view all videos from the federal event as well as content on the state and local government, education, and international markets, <a href="/events/videos?eventType=ElasticON&usecase=All&language=English&industry=Government">all sessions are now available on demand</a>. These complete sessions give you the benefit of these experts' insights and details on current government challenges and use cases — no matter the mission and no matter the need.

blog-banner-fed.png

blog-thumbnail-fed.png

Perspectives from the federal public sector: Enable mission success with Elastic

Our ElasticON Public Sector event for State and Local Government & Education showcased the tech solutions organizations are using to deliver for their constituents faster. Read more about the event, including takeaways from the key sessions.

With the surge of endpoints and growing demands for access to data, public sector organizations have seen an increase in security threats since the onset of the pandemic. Against this backdrop, Elastic gathered industry leaders to learn how to manage these challenges and demands&nbsp;and build a path toward the future. The recent State and Local Government & Education ElasticON Public Sector event showcased the tech solutions that are delivering for constituents, faster. What should organizations keep&nbsp;top of mind to achieve these goals? Presenters focused on these takeaways:
<ul>
	<li aria-level="1">Regardless of the size or purpose of your organization, the volume of data at your disposal will increase. Having the tools to analyze that increased data load and to adapt processes at speed and scale will be critical for efficiency and effectiveness in the future.&nbsp;</li>
	<li aria-level="1">Whether your focus is to enable business visibility, optimize infrastructure and applications, or protect your digital ecosystem, the power of search is the catalyst for becoming an insights-driven organization.</li>
	<li aria-level="1">Small cybersecurity and IT teams require a consolidated, unified analyst experience rooted in security and observability in order to protect the large number of devices, employees, endpoints, and log data in their purview.&nbsp;</li>
	<li aria-level="1">As technology is spread across numerous jurisdictions, counties, and institutions, organizations are facing challenges with speed, networking, accessibility, tracking, and data security.&nbsp;</li>
</ul>To address these challenges, Elastic bridges the gap between silos, organizations, and domains to proactively detect, analyze, and act on threats before they put the institution at risk. And the latest Elastic releases make our capabilities more responsive than ever to the requirements of public sector projects. But don’t just take our word for it. Using the highlights below as a guide, learn how organizations across state and local government and education are leveraging Elastic to quickly analyze data, unify intelligence, and standardize datasets.&nbsp;
 <h3>Using a risk-based approach to provide cost-effective security</h3>This unified view of data, threats, and risk was the main focus of the presentation by <a href="/elasticon/archive/2021/public-sector/state-local-government-education/using-a-risk-based-approach-to-provide-cost-effective-security">Blaine Stubstad (Senior Information Security Engineer) and Adam Pena (Senior Security Engineer) from the State of Arizona Enterprise Security team</a>. Inundated with large amounts of data to analyze across numerous regulatory frameworks, employee devices, and agencies, and constrained by fiscal budgetary cycles, the team undertook a proof of concept with the goal of increasing the automation and reporting capabilities of its risk assessment and threat intelligence process. Using Elastic, they produced more accurate assessments and models of cybersecurity threats.&nbsp;
 <h3>Tales from a security specialist in Salt Lake County</h3>Cybersecurity has evolved quite a lot since <a href="/elasticon/archive/2021/public-sector/state-local-government-education/tales-from-a-security-specialist-at-salt-lake-county">Chris Adamson, Information Security Engineer with Salt Lake County</a>, was tasked to start collecting logs in 2015. What began with proof of concepts on several different solutions has since tinkered its way through to a five-node Elastic cluster. After years of expanding with Elastic, Salt Lake County has refined their dashboards to focus on only what is important to them, with consolidated logs in one place. What took hours and days now takes seconds with Elastic. And what was once a looming threat of phishing and ransomware has turned into an effective defensive posture of “mines and tripwires” to alert the team of attacks.
 <h3>Increased visibility: Opening the black box of networks</h3>For <a href="/elasticon/archive/2021/public-sector/state-local-government-education/increased-visibility-opening-the-black-box-of-networks">Oregon Health and Science University (OHSU) network architect Alexander Germain</a>, the challenge of connecting the many facets of campus and research networks across a state was time consuming and cumbersome. By implementing Elastic and Elastiflow, they are now able to see and fix performance and security issues before their users become aware of them. Increased visibility also helped OHSU detect security threats and address weaknesses in network infrastructure. This flexible approach made it applicable to a wide variety of use cases for visualizing external threats, conducting security investigations, troubleshooting, and getting insights into complex systems.
 <h3>Elastic Security in action</h3>For academic institutions, the need to secure vast amounts of user, behavioral, and endpoint data is constant and the teams charged with securing that data are all too often small. In this session <a href="/elasticon/archive/2021/public-sector/state-local-government-education/elastic-security-in-action">Kevin Wilcox, Information Security Specialist from Appalachian State University, and Tyfanie Wineriter, Database Administrator at The University of Oregon</a>, discuss the challenges of data security and the pressing need to search, analyze, and act on countless alerts each day. Using Elasticsearch, higher ed can do more with the teams, data, and requirements they have to create a proactive defense that produces actionable fixes, quickly.&nbsp;
 <h2>Continue the conversation&nbsp;</h2>Whether you were able to join us live or not, we want to help you keep the learning and community connections alive. We’re always happy to put you in touch with other public sector users to discuss use cases. Email us at <a href="mailto:sled@elastic.co">sled@elastic.co</a> directly or join in the <a href="https://elastic-uspubsec.slack.com/join/signup#/">Elastic Public Sector Slack workspace</a>, an online forum dedicated to the US public sector community.&nbsp;
And for those wanting to fully dive into the ElasticON experience, <a href="/events/videos?eventType=ElasticON&usecase=All&language=English&industry=Government">all sessions from the day are now available on demand</a>, highlighting how Elastic capabilities are meeting challenges and use cases from the field — no matter your charter and no matter when you need them.

blog-banner-SLED.png

blog-thumbnail-SLED.png

Overcoming data challenges for state and local government and education with Elastic

Elastic uses and tests our own products. That’s how our InfoSec team’s enterprise security data is helping our product team build and deliver quality, real-world detections.

At Elastic, we internally use, test, and provide feedback on all of our products. For example, the Information Security team is helping the Product team build a stronger solution for our customers.
The InfoSec team is an extremely valuable resource who acts not only as an extension of Quality Assurance/Testing, but also as a data custodian. In fact, our internal detections team utilizes internal Elastic InfoSec data to help build and test <a href="https://www.elastic.co/guide/en/security/7.13/prebuilt-rules.html">detection rules</a> that ultimately find their way into the Elastic Security product. 
Last month, I was afforded the wonderful opportunity of “riding along” with our InfoSec team to better understand how we use Elastic internally. Over the course of three days, spread across three weeks, I saw the underbelly of our internal systems, how they are used, and how the team uses Elastic Security every day.
[At the time of this ride-along, version <a href="https://www.elastic.co/blog/whats-new-elastic-7-12-0-schema-on-read-frozen-tier-autoscaling">7.12</a> of the Elastic Stack had been released and running in the wild for well over a month. However, the InfoSec team usually runs several minor iterations ahead of general availability.]
 <h2>Day one: Lay of the land</h2>On day one, I learned about all the tools InfoSec uses to keep Elasticians safe.&nbsp;
These include, but are not limited to, Case Management (Hive), Identity Management (Okta), various Threat Intelligence feeds, Slack, and Elastic Endgame. Internal Elastic data sources that are being pulled into Elasticsearch range from cloud-specific logs (AWS Cloudtrail, Azure Activity/Diagnostic Logs, GCP Stackdriver) to network-specific logs (load balancer, proxy, web server, Github, VPC Flow, authentication, and vulnerability) to more host-specific (Auditbeat/Filebeat, Endpoint Protection and Telemetry).
With all this data stored and available for searching, one key area of functionality the InfoSec team requires that is used every day is <a href="/guide/en/cloud/current/ec-enable-ccs.html">cross-cluster search</a>. With this setup, a single cluster serves as the search head, which can query and alert on events across all additional clusters. All out-of-the-box detection rules (currently numbering 525+) are enabled and running against the corresponding data sources. As a best practice, InfoSec focuses on cloud detections first (AWS, Azure, GCP). Based on industry trends, they have a specific emphasis around living-off-the-land (LOLbins) detections as well.
In addition to the standard machine learning jobs, InfoSec leverages 15+ custom&nbsp;jobs that pinpoint rare environmental occurrences. These machine learning job types include:
<ul>
	<li aria-level="1">Process/Executable (Process by System/Provider/Team)</li>
	<li aria-level="1">Process arguments by Process</li>
	<li aria-level="1">Process by Execution Location</li>
	<li aria-level="1">Login Location (Geo and IP)</li>
</ul>Currently, detection alerts are forwarded into a generalized Slack channel and might look like the following:
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt0cef0a6e8961cbfd/60d3a7e31e0505677a8837a6/blog-elastic-security-infosec-1.png" data-sys-asset-uid="blt0cef0a6e8961cbfd" alt="blog-elastic-security-infosec-1.png">
In general, this alert may consist of:
<ul>
	<li aria-level="1"> 
	A high-level description of the event
	</li>
	<li aria-level="1"> 
	Hyperlink to Kibana alert
	</li>
	<li aria-level="1">
	3-4 information fields (offending source, acting process, etc.)</li>
</ul>While internally at Elastic we leverage Slack for notifications, there are several alternative detection rule notification paths available for customers:
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltfdc0c4f89f7e1614/60d3a7f3b930a53616fa512b/blog-elastic-security-infosec-2.png" data-sys-asset-uid="bltfdc0c4f89f7e1614" alt="blog-elastic-security-infosec-2.png">
An analyst will pick up the alert, then pivot into Kibana and the Case Management tool to start their triage process. For this triage work, an analyst often walks through a process of elimination in determining what is the alert/event and what needs to be done with it. Analysts pick indicators of compromise (IoCs) to cross-reference and correlate across data sources such as VirusTotal, URLscan.io, and <a href="/guide/en/beats/filebeat/current/filebeat-module-threatintel.html" "="">additional threat intel feeds</a>. The analyst may leverage an IoC match for context and enrichment, and not necessarily strictly for alerting purposes. This helps to provide more value for the analyst to make a quicker decision on further investigative work or escalation toward proper resolution.
<h2>Days two and three: Using Elastic to protect Elastic</h2>On days two and three, I learned about our security-specific workflows and decision-tree processes when InfoSec is engaged in responding to an alert.
The Elastic InfoSec team has a weekly standing call with our Product Management team. Considering that we are <a href="/blog/securing-our-own-endpoints-with-elastic-security">customer zero</a> of all software build candidates, it is imperative to give proper, quick, and valuable feedback on what works and what can be improved upon. There have been several instances where our own InfoSec team helped design the vision for certain Elastic Security functionality.&nbsp;
One of these <a href="https://github.com/elastic/kibana/issues/66587">suggestions</a> was the ability to include any data from a detection engine alert in the related action so that InfoSec could surface as much alert context as was available to the analysts. Prior to this, it was only possible to include some fields related to the detection rule itself — but not all the fields related to the detection.&nbsp;
The relationship between InfoSec and Product Management is truly symbiotic. The InfoSec team provides extremely valuable insight and feedback for our Product Management team, and the Product Management team affords InfoSec the opportunity to run a minor version ahead of the rest of the company and customers. This “first view” access means both teams get to digest new features and functions, as well as act as another layer of quality assurance.
Analysts often do things that may be unexpected during the development of a product. One <a href="https://github.com/elastic/kibana/pull/98706">example</a> of this was in Timeline when using many “OR” conditions — resulting in not being able to see search results due to the query area taking up the entire screen. This was a frustrating usability issue that could arise when stitching together parent-child relationships. Thankfully it was an easy and quick fix for our developers.
As a part of the triage process, it may be determined that a file needs to be pulled from the offending host. In order to determine “what happened” or “what could have happened,” an analyst may need to evaluate if a file is deemed “safe” or “unsafe.” Internally, we have an <a href="/blog/how-to-build-a-malware-analysis-sandbox-with-elastic-security">Elastic-built malware sandbox</a> that provides an opportunity to safely execute a file while collecting valuable telemetry data around the file. This, in turn, can help provide a quick turnaround to our end users on the safety of a potential attachment or file. Our Elastic Endpoint agent acts as a valuable part of this all-inclusive VM. This agent has been a part of the most recent (third) round of MITRE testing and has shown to perform effectively for both our clients as well as our own internal needs. <a href="/blog/mitre-engenuity-attck-round-3-carbanak-fin7-vs-free-open-elastic-security">We posted these results on our blog</a>, and also provided a <a href="https://ela.st/mitre-round3">Kibana Dashboard</a> for the public to navigate the results for their own reference.&nbsp;
Our InfoSec team also coordinates and helps manage the Elastic Bug Bounty Program. Info on disclosure of security issues can be found at <a href="/community/security/">elastic.co/community/security/</a> as well as the <a href="https://discuss.elastic.co/c/announcements/security-announcements/31">Security Announcements forum</a>. The bug bounty program is currently being reworked and expanded to be relaunched approximately within the next month.
 <h2></h2>

blog-banner-elastic-on-elastic.png

Metrics

SIEM

blog-thumb-elastic-on-elastic.png

How the Elastic InfoSec team uses Elastic Security

When UC Davis needed to enhance its SOC, they turned to Elastic to replace their legacy SIEM tool. Now, the SOC is able to address the visibility problems they experienced with their legacy tool while maximizing cost benefits.

The University of California at Davis is an agriculturally focused university of more than 30,000 students. Founded in 1905, the university performs federally funded research for the U.S. Department of Defense, U.S. Department of Agriculture, and other agencies. It’s also home to&nbsp; an electric power substation, police and fire departments, and even an airport.&nbsp;
All of this combined is a digital security challenge for Jeff Rowe, the university’s cybersecurity architect. There are 5,000 servers, and the university's Security Operations Center monitors 170,000 user accounts for cybersecurity threats originating across the globe.
With this level of complexity, UC Davis needed to enhance its security even more. The first step for the university was to centralize security logs in one place by eliminating multiple data silos that were difficult to maintain and search.
“Everything was spread out across multiple systems … Some of the systems were hard to maintain,” Rowe says. “This is primarily what we wanted to address with a new project.”
According to Rowe, the university wanted Elastic to replace their legacy security SIEM tool, ArcSight, which was handling about 300 gigs a day of security logs for their Security Operations Center (SOC).
<h2>The UC Davis journey to Elastic</h2>ArcSight was labor intensive and expensive, Rowe says. In addition, there was no federated access control, which meant that ArcSight couldn't be made available to some UC Davis partners.
Because of these shortcomings, the time had come to move to a “next gen security logging platform.” They evaluated Elastic, Splunk, SumoLogic, and LogRhythm. UC Davis chose Elastic because it solved their data visibility issues, was easier to maintain, and was cost effective.
The migration to Elasticsearch was completed in about six months — about a half a year sooner than planned. The university is now ingesting, on average, 800GB of data a day into Elasticsearch. The SOC retains logs at various hot, warm, and cold phases to maximize cost benefits.&nbsp;
“We can get a lot of data and it helps address our visibility problem that we've always struggled with before,” Rowe says.
<h2>Elastic reduces costs, enhances security</h2>According to Rowe, Elastic supplies a wealth of benefits to the university:
<ul>
	<li aria-level="1">Provides a high-performance, fault-tolerant logging platform</li>
	<li aria-level="1">Reduces costs</li>
	<li aria-level="1">Enables federated, role-based access control</li>
	<li aria-level="1">Enhances security</li>
	<li aria-level="1">Empowers student analysts working with the SOC to get well-placed jobs after graduation</li>
</ul>Watch the <a href="/elasticon/archive/2020/global/protecting-against-cyber-attacks-at-uc-davis-with-elastic">full presentation</a> to learn more about how Elastic enhanced security at UC Davis, and to find out how the university plans to use security and machine learning to become an even greater institution of education and research.

blog-banner-elasticon-global-announcement.png

blog-thumb-elasticon-global-announcement.png

Why UC Davis chose Elastic to enhance its Security Operations Center

The South Dakota Bureau of Information and Telecommunications needed to monitor and alert on all of the endpoints associated with more than 9,000 systems, so they turned to Elastic Security for both endpoint logging and SIEM.

The South Dakota Bureau of Information and Telecommunications (BIT) provides quality customer services and partnerships to ensure South Dakota’s IT organization is responsive, reliable, and well-aligned to support the state government’s business needs.
The BIT believes that “People should be online, not waiting in line.”
The bureau’s goals for the state's 885,000 residents include:&nbsp;
<ul>
	<li aria-level="1">Delivering valuable services at economical costs</li>
	<li aria-level="1">Building and retaining a highly skilled workforce</li>
	<li aria-level="1">Providing reliable, secure, and modern infrastrastructure</li>
</ul>When it comes to meeting the goal of a secure government infrastructure, the BIT discovered that its legacy SIEM solution couldn’t adequately monitor and alert on all of the endpoints associated with more than 9,000 systems.&nbsp;The BIT chose Elastic because they required a better tool to mitigate threats — a cost-effective solution with enhanced visibility to promptly pinpoint endpoint security issues and vulnerabilities.
In addition, the new solution needed to provide visibility for endpoints within the state’s environment as well as for remote systems used by employees working outside the office.&nbsp;
“We needed faster incident response times because our SIEM at the time wasn't able to handle the load. And also coming back to the fiscal costs for bringing in all the endpoint logs was quite a bit of a challenge. So we needed to look for an alternative solution to get quicker response times when we're dealing with security incidents,” says Nicholas Penning, Security Technology Engineer for the BIT’s Security Operations Center.
<h2>Endpoint logging use cases&nbsp;</h2>With a mutate filter plugin on Logstash, engineers can deploy a tagging system so analysts can understand whether events are coming from within the environment or remotely.&nbsp;
In one example, the critical logs the BIT captures in Elasticsearch and monitors in Kibana are failed Windows logins. This practice could uncover somebody without the proper credentials trying to access the system through brute force.&nbsp;
For the second example, logs from links and attachments in Microsoft Outlook are also scrutinized to prevent malware seeping into the environment.
“That's a very huge use case today because we're always looking for those users that may be clicking on malicious links,” Penning says.
<h2>Harnessing Elastic Security for SIEM</h2>Still, there’s more to it than just discovering a malicious URL, Penning says. What happens after the discovery of a malicious link?
“It looks like nobody clicked on that link, or went to that malicious site when I searched it. But what about four hours from now? Or 10 minutes from now? Are you still continuing to search and look for those things?” Penning asks.&nbsp;
By harnessing the detection capabilities of Elastic, Penning answers his own questions in the affirmative. That’s because the BIT automatically creates detection rules for indicators of compromise, such as&nbsp;malicious URLs.&nbsp;
Anytime that detection rule for that malicious URL gets targeted, Kibana will send an alert: “Hey, you know that detection rule we just created 20 minutes ago,” Penning quips, “we actually just got a hit on it.”
Watch the <a href="/elasticon/archive/2020/global/how-south-dakota-bit-defends-against-cyber-threats">full presentation</a> to learn more about how South Dakota BIT deploys Elastic Security for SIEM to monitor and alert on more than 9,000 endpoints and ensure a secure infrastructure for the South Dakota state government.

Logstash

Telecommunication

Alerting

Monitoring

Endpoint security

How South Dakota Bureau of Information and Telecommunications deploys Elastic to secure endpoints

Learn how Dutch payments platform Adyen uses Elasticsearch to power payment search, monitoring, and log search.

At Adyen, we use Elasticsearch to power various parts of our payments platform. This includes payment search, monitoring, and log search. Let’s take a look at how we use Elastic for these different use cases and see how we capitalize on the power of Elasticsearch.&nbsp;
We recently did a talk about some of our Elasticsearch adventures at an Elastic meetup. <a href="https://www.youtube.com/watch?v=OCO7lgWVBQw">You can find a recording here</a>.
Before we look into our different use cases, it helps to get a high-level understanding of Adyen’s architecture.
At Adyen, we do many things, but the core of what we do is payment processing. Our payment platform is built for any business and every customer journey. We process payments across all sales channels.&nbsp;
Let’s say you want to buy a beer at the local pub (a merchant) and you want to pay with your credit card, which you received from a bank, called the issuing bank. Now when you pay for the beer, the pub wants to know if your account is valid and if you can make the payment. For this, the pub needs to contact the issuing bank.
This is where Adyen comes in.
When you provide your payment details — a credit card in this example — to the pub, the pub contacts Adyen with the payment&nbsp; details, and Adyen figures out the rest. This usually means sending the payment details to a third-party, or scheme, such as Mastercard or Visa, which forwards it to the issuer.
The Adyen flow looks like this:
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltd2439557133a7b4f/60a536586232565342cec03b/blog-adyen-1.png" data-sys-asset-uid="bltd2439557133a7b4f" alt="blog-adyen-1.png">
But why doesn’t the merchant contact the scheme directly?&nbsp;
Because the next shopper might not use a credit card but wants to pay by scanning a WeChat Pay QR code. The merchant does not want to build a separate integration for every way of paying, because it should also work for payment methods like credit/debit cards, SEPA, iDeal, Klarna, PayPal, and so on. This is especially true for international merchants, and this is where part of the power of Adyen lies.
Now imagine we do such a transaction not just once but hundreds of times per second. These are the principles that we follow that allow us to keep doing this efficiently and still be flexible:
<ul>
	<li aria-level="1">One platform worldwide</li>
	<li aria-level="1">A single application that receives payments</li>
	<li aria-level="1">Abstract away differences between payments to make it easier to handle the data later</li>
</ul>This is, of course, a very simplified view and explanation. In reality, there are hundreds of components and moving parts that work seamlessly together to make everything work.&nbsp;
Now that we have a basic understanding of the scale of the architecture, let’s look at how we use Elastic.
<h2>Payment search</h2>Customer Area is our merchant-facing web application where merchants can gain insights into their transactions and perform operational actions. In the Customer Area, there is a feature called payment search. It is the most used functionality in the Customer Area, and it allows merchants to search their payments by using an order reference, email address, payment method, and so on.
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt455889d71e752e6b/60a536816232565342cec03f/blog-adyen-2.png" data-sys-asset-uid="blt455889d71e752e6b" alt="blog-adyen-2.png">
We use Elasticsearch as a search engine in our application because it is built to provide simple REST APIs and has a distributed nature, speed, and scalability.&nbsp;
Adyen processes millions of payments every day. We want to provide a near real-time payment search functionality for merchants.&nbsp;
So, imagine that a shopper is purchasing a pair of shoes in a store. The credit-card payment for the shoes needs to be immediately visible to the merchant so they know the buyer’s purchase was successfully processed —&nbsp; usually within seconds.
In order to make this possible, we have to index all this data into the Elasticsearch cluster when the payment is successfully processed. For these purposes, we have an in-house streaming/consumer framework that helps us to make this happen in a scalable way. It is implemented to take the burden from accounting clusters and create a summary of the data in the streaming clusters. This data is fed to the consumer framework, which indexes the data into the payment Elasticsearch cluster.
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt9eee4002b17096be/60a5369ab80011658d7f55a2/blog-adyen-3.png" data-sys-asset-uid="blt9eee4002b17096be" alt="blog-adyen-3.png">
The consumer framework implements three parts of the indexing journey:&nbsp;
<ul>
	<li aria-level="1">The first part is filtering, which decides if the current stream item is of any interest to this consumer. This is used to filter out the payments which have a creation date before a certain date.&nbsp;</li>
	<li aria-level="1">The filtered payment goes to the consuming part, where we apply the business logic. In our use case, it converts a payment to a payment document and creates index/update requests depending on the status of the payment.&nbsp;</li>
	<li aria-level="1">After the consumer framework reaches a certain amount of consumed payment or a certain time, it goes to a persisting level where it performs the bulk operation with the bulk request. The bulk request consists of index/update requests that we consumed so far. After it gets a successful response from the Elasticsearch cluster, it saves the in-memory state of the consumer to db. In case some requests fail, the consumer framework includes them into the next persist and then saves the state. This helps us make sure we index all the data.&nbsp;</li>
</ul>This cycle in the consumer framework runs continuously, as long as we have new payments processed. It allows us to give our customers a near real-time, reliable, and fast payment search experience.
<h2>Platform monitoring</h2>The goal of the monitoring team at Adyen is to provide solutions that can monitor the whole platform for merchant-facing issues. This is already quite a challenging task, and on top of that, the solution must be flexible enough to be used by other teams, each with a very unique use case.&nbsp;
Plus, the alerts must occur on a (near) real-time basis. To summarize, monitoring aims to provide:
<ul>
	<li aria-level="1">Real-time alerts</li>
	<li aria-level="1">End-to-end monitoring</li>
	<li aria-level="1">Integrated, scalable solutions to all teams at Adyen</li>
</ul>To accomplish this, we leverage the Elastic Stack as the underlying basis to build the necessary tools. The monitoring solution looks upon custom-based analytical events. The analytic events are freely created by the programmers and are the main data source for all the configurable monitors.&nbsp;
These events exist in parallel and complement the logging infrastructure — and the data in databases. Next, the monitors accommodate the analytic events into multiple use cases, such as spikes, drops, underperforming transactions, anomaly detection, forecasting, and more. These monitors are responsible for creating alerts that are, finally, delivered to the right individuals (or teams).
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt70983e3bf84c81d6/60a5370b9748366cfa1543ce/blog-adyen-4.png" data-sys-asset-uid="blt70983e3bf84c81d6" alt="blog-adyen-4.png">
From a business perspective, the objectives were:
<ul>
	<li aria-level="1">Customizable monitors for predefined use cases.
	<ul>
		<li aria-level="2">Provide monitors for standard use cases (such as a drop in volume, or a spike in errors) that are easy to set up for anyone in the organization</li>
	</ul></li>
	<li aria-level="1">Out-of-the-box measurements.
	<ul>
		<li aria-level="2">Every new merchant, or host, is already capable of generating events from Day 1</li>
	</ul></li>
	<li aria-level="1">End-to-end alert lifecycle integration.
	<ul>
		<li aria-level="2">Use a single interface to create any monitor, tweak its settings, and receive the alerts</li>
	</ul></li>
</ul><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt1ef45c504b72fc20/60a5376fae6c8a6e290a629a/blog-adyen-5.png" data-sys-asset-uid="blt1ef45c504b72fc20" alt="blog-adyen-5.png">
From a development perspective, the objectives of monitoring were:
<ul> 
	<li aria-level="1">Real-time user-based analytical events</li>
	<ul>
		<li aria-level="2">Parallel and complementary with the logging structure</li>
		<li aria-level="2">Seamless ES template mapping to/from java objects</li>
	</ul>
	<li aria-level="1">Enabled/disable event materialization routes on top-level events.&nbsp;
	<ul>
		<li aria-level="2">Long-term storage</li>
	</ul></li>
</ul><h2>Keeping the lights on</h2>Apart from payment search and monitoring, we also use Elasticsearch for storing and retrieving logs, and we use Kibana for log search, which is integral for troubleshooting issues internally.&nbsp;
Our logging cluster was at 1 petabyte at one point. For payment search, we did a migration from Elasticsearch v1.7 to v7.6 with 12TB of data last year, and the flexibility of Elasticsearch made the migration smooth. So at Adyen, Elasticsearch will continue to be an integral part of keeping our lights on.

blog-banner-coin-stacks.jpg

Architecture

App search

blog-thumb-coin-stacks.jpg

Why Elasticsearch is an indispensable component of the Adyen stack

Since leveraging Elastic about two years ago for security operations, global law firm King & Wood Mallesons is now able to quickly pinpoint security events across the enterprise.

King & Wood Mallesons (KWM) is among the world’s most innovative law firms and is represented by 2,400 lawyers in 28 locations across the globe. The international law firm, based in Australia, helps clients flourish in Asian markets by helping them understand and navigate local challenges and by delivering solutions that provide clients with a competitive advantage.
The firm believes in taking a partnership approach with clients and in pushing boundaries to connect Asia to the world and the world to Asia.&nbsp;
To succeed in that mission, John Reeman, the Chief Information Security Officer for KWM, says the firm began leveraging Elastic about two years ago for security operations. The move has provided the firm with fresh insights to quickly pinpoint security events across the enterprise.
“We're using it predominantly for security operations, so looking at log sources, threat hunting, taking information in from endpoints, and a wide variety of critical applications to give us context about our environment so that we can spot and identify the bad guys and hopefully keep them out,” Reeman says.
<h2>Machine learning, maturing the security platform</h2>Reeman states the firm is working to refine and mature its security platform. That’s a constant battle businesses face when it comes to securing the enterprise as threat actors evolve.
“Am I collecting all of the right logs from every log source that I feel is critical? You're always going to be playing that game probably for a long time, because you never know what you don't know,” he says.
As part of that maturation process, and to gain even more insights into its security data, Reeman says KWM is experimenting with Elastic machine learning on domain name system (DNS) activity, user behavior, and other processes. However, understanding the benefits of machine learning takes time, Reeman cautions.&nbsp;
With machine learning, businesses must first acquire an understanding of a baseline for normal behavior. Once that is known, a clearer picture of anomalous behavior is more easily surfaced as time passes, Reeman says.
“I come from a digital forensics background and we always say, ‘you have to know normal to find evil,’” Reeman says.
<h2>Securing the endpoints</h2>Beyond securing the enterprise perimeter and applications, Reeman says the firm is adding log data into Elastic from endpoints scattered around the globe. Reeman says endpoint data, processed with machine learning, is important to capture because it's the “last bastion” of where security information resides.&nbsp;
“Pulling that information into Elastic,” he says, “was very important for us to be able to understand the environment and again, bring it back to understanding normal to find evil.”
Watch the <a href="/videos/customer-conversation-with-king-and-wood-mallesons">full presentation</a> to learn more about how King & Wood Mallesons harnesses Elastic to secure endpoints, push boundaries, and connect Asia to the world, and the world to Asia.

Machine learning

King & Wood Mallesons CISO relies on Elastic to “spot and identify” security threats

Learn how Shell embraces Elastic to gain insights into the future to predict equipment failure, leaks, and to increase uptime and comply with data retention regulations across the globe.

This post is a recap of a <a href="https://www.elastic.co/videos/customer-conversation-with-shell-international">presentation</a> given at<a href="https://www.elastic.co/elasticon/global/"> ElasticON 2020</a>. Interested in seeing more talks like this? Check out the<a href="https://www.elastic.co/elasticon/global/agenda"> conference archive</a>.
Shell International knows that it takes cutting-edge technology to thrive in the competitive, global energy industry. With projects around the world, in both renewable and non-renewable energy, Shell must always have insights into the future. From determining expected output to predicting equipment failures, there's no room for guessing in an industry where downtime is unacceptable.&nbsp;
This is why Shell is a part of the<a href="https://www.elastic.co/blog/elastic-has-joined-the-open-group-open-subsurface-data-universe-osdu-forum"> Open Subsurface Data Universe (OSDU)</a>, and why they use<a href="https://www.elastic.co/elasticsearch"> Elasticsearch</a> for the analysis of a range of geospatial, full-text, and numeric data that is critical in the energy space.
During his<a href="https://www.elastic.co/videos/customer-conversation-with-shell-international"> ElasticON Global presentation</a>, Johan Krebbers, general manager of Digital Emerging Technologies and VP of IT Innovation at Shell International, spoke to the importance of the flexibility that Elasticsearch provides the energy giant — including its ability to be cloud-native or on-prem, and its ability to be used with several hosting providers to enable Shell to comply with data retention regulations imposed by governments throughout the world.
What’s more, because of its capability to search and analyze so many different data types, Elasticsearch is at the heart of Shell’s observability, machine learning (ML), machine vision, and natural language processing solutions.
<h2>Predicting failure, increasing uptime </h2>According to Krebbers, the best time to replace equipment is before it fails. If you wait until it’s too late, the cascading effects can drive up repair costs, increase profits, and impact revenue and growth. By leveraging the real-time ingest, ML, and observability capabilities of Elastic, Shell is able to use predictive modeling to replace or repair machinery before it fails.&nbsp;
"We use ML for predictive maintenance of our facilities," says Krebbers, who is charged with bringing new technologies to Shell. "You collect the real-time data. You have the ML models. And then [you] start predicting when is a pump going to fail? When is a compressor going to fail? If you can predict failure, you can predict downtime. And downtime always costs you money. So you want to increase your uptime.”
Another way to keep overall costs down is to make sure resources aren't wasted. That’s why Shell uses a machine vision solution with Elasticsearch at its core to gain insights into potential leaks throughout their global infrastructure.&nbsp;
<h2>Detecting spills, leaks, and emissions with robots </h2>"Machine vision plays an important role in leak detection and emission detection," says Krebbers. "You have robots driving around with cameras collecting spillage, leakage, emissions. [Then] you bring it into a cloud environment, apply machine visions and start looking for videos of leakage and spillage. And when there's an issue, you can immediately raise that with appropriate staff."
Going beyond detecting leaks, spills, and emissions, Shell must always be on the hunt to harvest new energy sources.&nbsp;
As part of their mission to bring more resources to market, Shell employees are embracing and demanding natural language processing capabilities with Elasticsearch as the prime search engine. Many Shell experts, for example, are using natural language processing to quickly search through mountains of the company’s mission-critical subsurface data related to wells, development, and exploration to keep Shell competitive.
Watch the <a href="https://www.elastic.co/videos/customer-conversation-with-shell-international">full conversation with Shell International</a> to learn more about how Shell increases uptime, detects emissions, and surfaces exploration data faster with Elastic.

Energy & utilities

blog-thumb-elasticon-global-shell.png

Elastic powers Shell’s flexibility to thrive in the energy sector

Watch how Audi Business Innovation drives software development with Elasticsearch Service

This post is a recap of a <a href="/elasticon/archive/2020/global/observability-with-elastic-at-audi-business-innovation">community talk</a> given at <a href="https://www.elastic.co/elasticon/global/">ElasticON 2020</a>. Interested in seeing more talks like this? Check out the <a href="https://www.elastic.co/elasticon/global/agenda">conference archive</a>.
Today’s cars are computers on wheels, and they’re powered by software as much as they are by batteries or gasoline. When it comes to building software for Audi, Volskwagen, Porsche, Traton, and other brands, that’s a task assigned to Audi Business Innovation (ABI).&nbsp;
“Developers need the right tools in their hands that are easy to use,” says Stefan Teubner, an ABI team leader and DevOps engineer. Elasticsearch Service is one of those tools that allows developers to “focus on developing software” instead of mundane infrastructure tasks.
At the architectural heart of ABI’s digital assembly line for the development of software is Elasticsearch Service. It helps the automaker remain a top competitor in autonomous driving, electric vehicles, and in-car digital experiences.
With this continued focus on streamlining their digital processes, ABI stays true to their motto, “We move people and shape the future.”
<h2>Making a sale from the car dashboard </h2>Audi customers can purchase in-car digital experiences, such as voice recognition in certain vehicles, after they drive off the sales lot, explains Teubner. Customers can add these software experiences from their mobile phone, from an app in the car’s dashboard, or from the web.&nbsp;
To keep this customer experience up and running, ABI relies on Elasticsearch Service as a central logging service to store log files so software issues can quickly be detected and fixed before the customer experience is impacted.&nbsp;
“We wanted to provide a fast and easy-to-use service to store and analyze log files,” Teubner says. Kibana dashboards are great at searching and analyzing their 29 terabytes of log files and Elasticsearch Service comes with low operational costs as well as performs at a high level.
<h2>Elasticsearch Service, the reliable SaaS offering </h2>Initially, ABI had a single Elasticsearch cluster running on premises. But that wasn’t good enough. They needed a solution with higher accessibility, scalability, reliability, flexibility and security. After surveying the market, ABI decided on Elasticsearch Service, a “reliable SaaS offering,” he says.
Now, every application gets its own Elasticsearch cluster with Kibana instances. Elasticsearch Service, available on AWS, Google Cloud, and Microsoft Azure, offers Rest APIs making it easy to spin up clusters and Kibana instances, and to configure those instances.
“We are just using it. We don’t need to take care of scaling. It’s all automated,” Teubner says.&nbsp;
Watch the <a href="https://www.elastic.co/elasticon/archive/2020/global/observability-with-elastic-at-audi-business-innovation">video</a> to learn more about how Audi Business Innovation drives software development with Elasticsearch Service and helps them move people and shape the future.

blog-thumb-eon-global-audi.jpg

Audi Business Innovation drives software development with Elasticsearch Service

Learn why Cisco embraced Elasticsearch to build an IT infrastructure platform to manage and bring transparency to 1.5 million service contracts and 2 billion IT assets.

This post is a recap of a <a href="https://www.elastic.co/elasticon/archive/2020/global/elasticsearch-and-cisco-next-gen-customer-experience">presentation</a> given at<a href="https://www.elastic.co/elasticon/global/"> ElasticON 2020</a>. Interested in seeing more talks like this? Check out the<a href="https://www.elastic.co/elasticon/global/agenda"> conference archive</a>.
Network infrastructure is the engine that drives a company’s business. As companies scale, assets that compose this infrastructure become more complex to manage. That means there’s more hardware, more software, and more subscriptions and services that require tracking.
It’s critical that IT managers and network administrators have complete accounting of these assets. For Siva Pattabiraman, Technical Lead Engineer at Cisco, these assets represent the backbone that keeps the organization running without interruption.
 <h2>Introducing My Cisco Entitlements (MCE)</h2>According to Pattabiraman, companies must be able to understand whether assets are being properly optimized, whether there are too few or too many licenses and subscriptions, and whether assets are nearing expiration or approaching end of life.&nbsp;
Having that understanding helps cut down on wasteful spending, provides Cisco with an understanding of whether they are complying with license agreements, and gives them leverage at the negotiating table when it comes time to renew or acquire more assets.
“To put it simply, we didn’t have a single tool or platform to manage customers’ assets and their entitlements,” Pattabiraman says. “Without full visibility into what the customer owns and what they are using, organizations could face significant legal, financial, and operational issues.”
<blockquote>In response to this challenge, Cisco harnessed Elasticsearch to build a platform they call My Cisco Entitlements (MCE). It’s a fairly new, secure, user-friendly solution that provides a “holistic view” to manage about 1.5 million service contracts and 2 billion assets companywide.
</blockquote>“In summary, it provides end-to-end IT infrastructure transparency,” Pattabiraman says.
 <h2>Why entrust My Cisco Entitlements to Elastic?</h2>Cisco was already using Elasticsearch for other projects, including search on Cisco.com. So Pattabiraman knew it would work well for the MCE project. In addition, Elastic Support would be there to assist during the development phases, if needed.
To build the MCE tool, Cisco had to ingest asset and entitlement data into what he described as a “high-performance search index.” This includes data from contracts, devices, licenses, warranties, customers, resellers, distributors, and more.
Pattabiraman was confident in Elasticsearch’s ability to index and search quickly through these and other sources and render sub-second responses. “To achieve such a high performance index or a search engine, our primary component of technology stack had to be Elasticsearch,” he says.
 <h2>Minimizing risk&nbsp;</h2>Cisco uses Logstash, a component of the Elastic Stack, to ingest the data into MCE. Ingest rates can reach as high as 10,000 documents per second with 20-40 million change events per day.&nbsp;
“Moving to Logstash was a game changer for us because it helps you thread data between multiple sources very easily,” Pattabiraman says.
Now, as many as 15,000 partners, internal customers, and Cisco as a whole are embracing MCE to simplify access and management of assets.
“What did we really give back to business as a business value?” asks Pattabiraman. “My Cisco Entitlements, MCE, became the secure, one-stop platform where customers can gain insight into their business. They're able to better manage their products and services, and they are able to minimize their risk.”
Watch the <a href="https://www.elastic.co/elasticon/archive/2020/global/elasticsearch-and-cisco-next-gen-customer-experience">video</a> to learn more about why Cisco embraced Elasticsearch to build a platform to manage 1.5 million service contracts and 2 billion IT assets.

blog-thumb-elasticon-global-cisco.jpg

Why Cisco embraces Elastic for IT infrastructure transparency

The Department of the Armed Forces in France used Elastic Cloud to launch a new platform, in record time, to help in the fight against COVID-19 and any other pandemic.

WaKED-CO (Watch of Knowledge on Emergent Diseases COVID-19) is an initiative launched in record time — deployed just a month after developing a prototype — under the leadership of the health service within the Ministry of Armed Forces in France. The project had one core mission: to make it easier to research the literature around the COVID-19 crisis.
To achieve this mission, the initiative had two priorities: first, to be able to track the rapid evolution of the epidemic and the scientific progress around it, and second, to help stakeholders such as the authorities or the medical community make critical decisions about the crisis. For this project, several teams of data scientists from different government organizations worked together: SGA (General Secretariat for Administration), SSA (Armed Forces Health Service), CCIAT (Army Center for IT Management), and IRBA (Armed Forces Institute for Biomedical Research).
At the outset, the teams made an assessment of the situation and the underlying challenges:
<ul>
	<li>Resources are distributed and vary in quality</li>
	<li>Researching the academic literature is complex and time-consuming</li>
	<li>It is difficult to get a high-level view of COVID-19 research</li>
	<li>Publications are not exhaustive</li>
	<li>Researchers have limited time</li>
</ul>Right away, the Ministry of Armed Forces identified <a href="/cloud/">Elastic Cloud</a> as a way to tackle these challenges. The Ministry of Armed Forces was already using the Elastic Stack to provide <a href="/fr/customers/ministry-of-armed-services-france">a 360° view of each department’s data</a> in order to support decision making, allowing users to add, cross-reference, and finely analyze data. With <a href="/cloud/">Elastic Cloud</a>, the WaKED-CO team was able to remove limitations related to the maintenance and implementation of the solution, and focus instead on taking advantage of the features of the Elastic Stack:

The flexibility of Kibana, with numerous dashboard options (from a simple graphic to time series, word clouds, or detailed maps)
The powerful search capabilities of Elasticsearch paired with data science — a combination that makes Elastic a great fit for tackling the challenges of the project
<h2>Relevant Research, thanks to Elastic
</h2>The WaKED-CO team began by setting the functional objectives for WaKED-CO. They wanted to be able to create a list of all the scientific and clinical test publications on the platform and enrich them with additional information. In doing so, they aimed to make it easier to organize, sort, and gather information, as well as carry out scientific research and monitoring. The database needed to be usable in connection with artificial intelligence to help carry out studies on COVID-19. Lastly, it was essential for the platform to perform vigilant monitoring of the situation by surfacing relevant information for the user and by guiding the user’s research via a notification system.

<h3>Ingesting data into Elastic
</h3>The first stage of the project was to make numerous sources of assorted data available and thereby broaden the researchers’ range of analyses. This data comes from a wide range of peer-reviewed publications, the majority specializing in the medical field (e.g., MedWorm). This includes data from websites containing prepublished scientific articles as well as national and international databases (<a href="https://pubmed.ncbi.nlm.nih.gov">PubMed</a>, <a href="https://www.sciencedirect.com">ScienceDirect</a>, <a href="https://www.wiley.com/en-gb">Wiley</a>). To incorporate these disparate data sources, the team implemented an ETL (extract, transform, and load):

<ul>
	<li>Extract: Intake modules allowing the intake of various sources of data and either utilizing APIs or behaving like web crawlers (carrying out scraping) were developed in Python.
</li>
	<li>Transform: Once the data extraction has been carried out, the different sources go through a standardization process. This standardization is essential because it allows the data to be examined in a consistent manner.</li>
	<li>Load: Once the processing has been carried out, the data is loaded into an architecture that is distributed in terms of compute and storage, and the data is indexed to make it available via a search engine.</li>
</ul><h3>Data Processing with Elastic
</h3>The majority of the data is in the form of text, so the team implements a variety of natural language processing (NLP) techniques. Here is a non-exhaustive list of some of the various enrichments that have been implemented or planned:
<ul>
	<li>Detection of the author’s country</li>
	<li>Detection and extraction of named entities (pathologies, molecules, drugs, dosages) referenced in the publications</li>
	<li>Extraction of keywords, if not already provided by the authors</li>
	<li>Notification system based on either the appearance of new entities or their evolution</li>
	<li>Translation of publications into different languages in order to improve access to knowledge</li>
	<li>Execution of a chatbot based on knowledge extracted from publications</li>
</ul>Lastly, once the data is processed and the team has refined their Kibana visualizations, the provision is now dynamic and allows an inventory to be carried out in near real time, helping those actively involved in this health crisis to make decisions.
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blte602bbfa7f07b0dd/5fa42adb6f82405d9a4a9863/Waked-co-image.png" data-sys-asset-uid="blte602bbfa7f07b0dd" alt="" style="display: block; margin: auto;">
<h2>Supporting Public Services with Elastic
</h2>The platform now allows key participants in the current health crisis to carry out daily monitoring or receive notifications on trends around the crisis or any specific issues they’d like to track. WaKED-CO has been accessible since April 15 2020 on the French government’s publicly accessible <a href="https://covid-19.sante.gouv.fr/">COVID-19 platform</a>, in the scientific monitoring section. The platform is used by the IRBA and the crisis unit of the Department for Social Affairs and Health. The platform had on average 2,500 daily queries and included data on more than 1.2 million publications. Since then, the intake of data has continued on a daily basis and has not stopped growing.

The team now intends to continue to develop this platform, retaining the same objective to simplify the task for researchers and others involved in the health crisis. For instance, the team plans to use <a href="/what-is/kibana-canvas">Canvas</a> to highlight key figures on scientific progress (number of research projects, clinical trials, etc.). Beyond the COVID-19 crisis, the team hopes to use this platform as a crisis management tool to allow major participants to be better equipped should a similar situation ever occur again.
 
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt9b4f535a0c22ae04/5fa42c633cdbef7187ce45db/image2.png" data-sys-asset-uid="bltf47e834a633e2ad8" alt="Guillaume Vimont" width="148" height="160" style="float: left; width: 148px; height: 160px; margin: 0px 10px 10px 0px;">
Guillaume is responsible for the Data Science and Big Data center within the LABO BI & BIG DATA under the Department of the Armed Forces.

blog-banner-covid-19.png

blog-thumb-covid-19.png

French Ministry of Armed Forces picks Elastic Cloud to fight COVID-19 and future pandemics

Learn how Elastic InfoSec deploys, orchestrates, and keeps their clusters current with Elastic Cloud on Kubernetes (ECK) and Helm.

This post is part of a blog series highlighting how we embrace the solutions and features of the Elastic Stack to support our business and drive customer success.
The Elastic InfoSec Security Engineering team is responsible for deploying and managing InfoSec's infrastructure and tools. At Elastic, speed, scale, and relevance is our DNA and leveraging the power of the Elastic Stack is the heart of InfoSec. Our goal is not only to test new features as they are rolled out, but to use these capabilities to protect Elastic.
We trust and believe in our products to protect our brand. If you, too, use the Elastic Stack, you’re likely familiar with its rich features and power, which are deliberately released at a rapid cadence. The intent behind our frequent releases is to enable customers to solve issues and make life easier where possible, and what better way to do this than to consistently make new features readily available?
I know what you are thinking: in theory, remaining up-to-date sounds like an excellent idea, but frequently upgrading can be time consuming, can introduce new bugs, and can disrupt production systems. That is why here at Elastic, we put theory into practice. We would not expect you to keep up with our release cadence if we could not keep up with our own pace. So let us show you how we are harnessing <a href="https://www.elastic.co/downloads/elastic-cloud-kubernetes">Elastic Cloud on Kubernetes</a> (ECK) and Helm to deploy infrastructure and remain up-to-date. If you would like more information on how InfoSec leverages our technology to protect Elastic, check out “<a href="https://www.elastic.co/blog/securing-our-own-endpoints-with-elastic-security">Securing our endpoints with Elastic Security</a>.”
<h2>Why ECK?</h2>Our managed Elasticsearch offering — <a href="https://www.elastic.co/elasticsearch/service">Elasticsearch Service on Elastic Cloud</a> — meets all of our requirements around scalability, functionality, and capability. That said, why doesn’t InfoSec use Elasticsearch Service? If you think about how you would handle operational logs, the general best practice is to ship monitoring logs to a remote cluster because you want to ensure that data is available to you in the event the cluster you are monitoring becomes unavailable. For this reason, InfoSec wanted to operate independently from the Elasticsearch Service to ensure we are able to monitor our services if issues were to arise.
A couple years ago, SecEng began exploring using Kubernetes to deploy Elasticsearch clusters and when our internal teams developed ECK, it was a no brainer for us to test and provide feedback.
<h2>What is ECK?</h2><a href="https://www.elastic.co/blog/introducing-elastic-cloud-on-kubernetes-the-elasticsearch-operator-and-beyond">ECK</a> not only makes it easier to deploy Elasticsearch and Kibana, but it enables you to quickly upgrade, manage multiple clusters, perform backups, scale, and update clusters. Plus, it handles security by default. SecEng currently manages 6 production Elasticsearch clusters that are home to audit, monitoring, and various security logs. Our largest cluster is roughly 35 GKE nodes and growing as we collect additional data for better visibility.
As of today, we receive an average of 30,000 events per second (EPS), which is over 2 billion events per day or 6TB of data a day. Infosec is also leveraging cross-cluster search, which allows our Detection and Response teams to centralize reporting and alerting. With the latest ECK release, enabling this feature has never been easier. Below, we will show you how we have added this to our Helm chart, as well as other features that Elastic Infosec normally enables.
<h2>Getting started with ECK</h2>If you are new to ECK, please review the <a href="https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-quickstart.html">ECK quickstart documentation</a> to get started. If you want to a deploy a one-node Elasticsearch cluster, your manifest would look like this:
<pre class="prettyprint">apiVersion: elasticsearch.k8s.elastic.co/v1 
kind: Elasticsearrch 
metadata: 
 name: quickstart 
spec: 
 version: 7.8.0 
 nodeSets: 
 - name: default 
 count: 1 
 config: 
 node.master: true 
 node.data: true 
 node.ingest: true 
 node.store.allow_mmap: false
</pre>As you can see, you can easily configure the number of nodes and node types for your deployment. The Kibana manifest looks very similar:
<pre class="prettyprint">cat &lt;&lt;EOF | kubectl apply -f - 
apiVersion: kibana.k8s.elastic.co/v1 
kind: Kibana 
metadata: 
 name: 
spec: 
 version: 7.8.0 
 count: 1 
 elasticsearchRef: 
 name: quickstart 
EOF
</pre>Whether you want to deploy a one-node cluster or 100 nodes, you can configure the manifest to meet your needs. When we first started leveraging ECK, we would create a separate manifest per cluster. After deploying several clusters, we noticed several things:
<ol>
	<li>When creating a new cluster, we would manually copy the base Elasticsearch and Kibana manifest. Note: This made deployments prone to errors and more difficult to review in GitHub because it was being treated as new code.</li>
	<li>If there were any changes to the operator, we would have to update all of our manifest files as we upgraded.</li>
	<li>While our deployments consisted of the same services — Elasticsearch, Kibana, Logstash, Beats, beats-setup, NGINX-ingress — we needed to alter the specs for each cluster: resource sizes, SAML settings, secrets, storage, services and ports, cross-cluster search, etc. This resulted in a large amount of duplicate/repetitive code, making our deployments difficult to manage</li>
	<li>Wait… We still have to manage Logstash, Beats, NGINX — oh my?</li>
</ol>We concluded that we needed to abstract specific values relevant to each of the clusters and make it customizable. With this requirement in mind, we started experimenting with Helm, a package manager and templating engine for Kubernetes. We found that Helm allowed us to template and deploy our standard services and dependencies consistently across our clusters, while still allowing for flexibility.
Note: We already had pockets of Helm deployments in our environment so we expanded on its usage. You could achieve the same results with whatever package manager you would like to use: bring your own package manager.
<h2>ECK and Helm</h2>We decided to create Helm charts for all of the components that made up our stack and re-use charts where possible. By doing so, we significantly reduced the amount of code that needed to be modified and reviewed. We relied on the requirements.yaml to define our dependencies and the values.yaml to pass the appropriate values for our cluster. Using this approach, we were able to reuse a single ECK manifest to deploy Elasticsearch and Kibana. If modifications needed to be made, we would submit a PR against the manifest and then update each of the cluster’s dependencies. By default, here are the basic components that make up an Infosec cluster:
<h3>NGINX</h3>Route users to an OAuth proxy to check whether or not they are an Elastician. if true, route to the appropriate service — Elasticsearch or Kibana. We created a ConfigMap for the nginx.yml and abstracted custom values such as:
<ul>
	<li>DNS name</li>
	<li>Enable OAuth</li>
	<li>Load custom certificates</li>
	<li>Define backend services</li>
</ul>This is the values.yaml:
<pre class="prettyprint">nginx: 
hosts: 
 elasticsearch: 
 oauth: true 
 proxy: "https://oauth-url" 
 host: my-es-cluster.com 
 pki: 
 secretName: nginx-certificate 
 backend: 
 address: "infosec-demo-es-http:9200" 
 pki: 
 secretName: infosec-demo-http-certs-public 
 key: tls.crt 
 kibana: 
 oauth: true 
 proxy: "https://oauth-url" 
 host: my-kibana-instance.com 
 pki: 
 secretName: nginx-certificate 
 backend: 
 address: "infosec-demo-kb-http:5601" 
 pki: 
 secretName: infosec-demo-kb-http-certs-public 
 key: tls.crt
</pre><h3>Logstash</h3>This is our ingestion tier for logs. It Receives logs from Beats deployed at Elastic. We use the <a href="https://github.com/elastic/helm-charts/tree/master/logstash">official Elastic Helm chart for Logstash</a>.
 <h3>Beats</h3>We monitor our stack using Auditbeat, Filebeat, and Heartbeat. We use the <a href="https://github.com/elastic/helm-charts">official Elastic Helm chart for Filebeat</a>, but have created custom charts for Auditbeat and Heartbeat, since they are tailored for our use. ConfigMaps are created for both auditbeat.yml and heartbeat.yml anytime we add these charts as a dependency.
<h3>Beats-Setup</h3>We load Beats templates as we upgrade, so we don’t run into mapping conflicts. This is the values.yaml:
<pre class="prettyprint">beats-setup: 
 elasticsearchAddress: my-es-cluster-es-http:9220 
 kibanaAddress: my-es-cluster-kb-http:5601 
 clusterName: my-es-cluster 
 versions: [ 
 7.6.0, 7.6.1, 7.6.2, 
 7.7.0, 7.7.1, 
 7.8.0 
 ] 
 credentials: 
 # The secret name where the Elasticsearch credentials are stored 
 secretName: my-es-cluster-es-elastic-user 
 passwordKey: elastic 
 usernameKey: null
</pre><h3>job.yaml</h3>When the beats-setup job runs, it will automatically connect to Elasticsearch, use the *beats.yml, and load any dashboards/templates based on the version specified.
<pre class="prettyprint">apiVersion: batch/v1 
kind: Job 
metadata: 
 name: {{ $Release.Name }}-auditbeat-setup 
spec: 
 template: 
 spec: 
 restartPolicy: OnFailure 
 hostPID: true 
 containers: 
 {{- range $version := .Values.versions }} 
 - name: auditbeat-setup-{{ regexReplaceAllLiteral :\\." $version "-" }} 
 image: docker.elastic.co/beats/auditbeat:{{ $version }} 
 # Setup dashboards and templates. 
 command: ["auditbeat", "-e", "setup"] 
 securityContext: 
 capabilities: 
 add: ["AUDIT_CONTROL", "AUDIT_READ"] 
 resources: 
 limits: 
 memory: 200Mi 
 requests: 
 cpu: 100m 
 memory: 200Mi 
 volumeMounts: 
 - name: auditbeat-setup 
 mountPath: /usr/share/auditbeat/auditbeat.yml 
 subPath: auditbeat.yml 
 - name: elasticsearch-pki 
 mountPath: /usr/share/auditbeat/pki/elasticsearch.crt 
 subPath: elasticsearch.crt 
 - name: kibana-pki 
 mountPath: /usr/share/auditbeat/pki/kibana.crt 
 subPath: kibana.crt 
 env: 
 - name: ELASTIC_USERNAME 
 {{ if $Values.credentials.usernameKey -}} 
 valueFrom: { secretKeyRef: { name: {{ $Values.credentials.secretName }}, key: {{ $Values.credentials.usernameKey }} } } 
 {{- else -}} 
 value: elastic 
 {{- end }} 
 - name: ELASTIC_PASSWORD 
 valueFrom: { secretKeyRef: { name: {{ $Values.credentials.secretName }}, key: {{ $Values.credentials.passwordKey }} } } 
 {{- end }} 
 volumes: 
 - name: auditbeat-setup 
 configMap: 
 name: auditbeat-setup 
 items: 
 - key: auditbeat.yml 
 path: auditbeat.yml 
 - name: elasticsearch-pki 
 secret: 
 secretName: {{ .Values.clusterName }}-es-http-certs-public 
 items: 
 - key: tls.crt 
 path: elastic.crt 
 - name: kibana-pki 
 secret: 
 secretName: {{ .Values.clusterName }}-kb-http-certs-public 
 items: 
 - key: tls.crt 
 path: kibana.crt 
___
</pre> <h3>Elasticsearch / Kibana</h3>We created an ECK chart to abstract the following values:
<ul>
	<li>Resources: CPU, RAM and Java Heap, storage (Google SSD persistent storage)</li>
	<li>Enable SAML SSO</li>
	<li>Kube Secrets: Webhooks, certificates, and credentials</li>
	<li>Node roles: Master, data, ingest, machine-learning, Coordination. We also allow combined roles; although, we don’t have every possible combination, just those commonly used for our clusters</li>
	<li>IsBC: Is this a build candidate? InfoSec frequently deploys build candidates in production to validate them in real-world scenarios.</li>
	<li>Cross-cluster search: Enables and orchestrates cross-cluster search by just passing the client server’s CA. This automatically configures transport layer services.</li>
</ul><pre class="prettyprint">{{- if $config.ccs }} 
 xpack.security.transport.ssl.certificate_authorities: 
 {{- range $cert := $config.ccs.secretNames }} 
 - "/usr/share/elasticsearch/config/other/{{ $cert }}" 
 {{- end }} 
 {{- end }} 
 containers: 
 - name: elasticsearch 
 env: 
 - name: ES_JAVA_OPTS 
 value: {{ $config.java_heap }} 
 - name: NSS_SDB_USE_CACHE 
 value: "no" 
 resources: 
 requests: 
 memory: {{ $config.memory }} 
 cpu: {{ $config.cpu }} 
 limits: 
 memory: {{ $config.memory }} 
 # Loads remote certificates if Cross-Cluster Search is Enabled. 
 {{- if $config.ccs }} 
 volumeMounts: 
 - mountPath: "/usr/share/elasticsearch/config/other" 
 name: remote-certs 
 volumes: 
 - name: remote-certs 
 secret: 
 secretName: remote-certs 
 {{- end }}
</pre> <h2>Deploy a cluster using ECK and Helm</h2>Let’s walk through a deployment. In this example, we are going to deploy a two-node Elasticsearch cluster, Kibana, and Auditbeat. One node will have the master role and the appropriate resource specs. The other node will be a data and ingest node.
 <h3>values.yaml</h3><pre class="prettyprint">nginx: 
 hosts: 
 elasticsearch: 
 oauth: true 
 proxy: "https://oauth-url" 
 host: my-es-cluster.com 
 pki: 
 secretName: nginx-certificate 
 backend: 
 address: "infosec-demo-es-http:9200" 
 pki: 
 secretName: infosec-demo-http-certs-public 
 key: tls.crt 
 kibana: 
 oauth: true 
 proxy: "https://oauth-url" 
 host: my-kibana-instance.com 
 pki: 
 secretName: nginx-certificate 
 backend: 
 address: "infosec-demo-kb-http:5601" 
 pki: 
 secretName: infosec-demo-kb-http-certs-public 
 key: tls.crt 
eck: 
 clusterName: infosec-demo 
 version: 7.7.0 
 enableSecrets: false 
 secrets: [ 
 ] 
 elasticsearch: 
 nodes: 
 master: 
 isBC: false 
 # Enable Single Sign-on 
 enableSSO: false 
 role: master 
 count: 1 
 storage: 5Gi 
 memory: 8Gi 
 cpu: 2 
 java_heap: -Xms4g -Xmx4g 
 data: 
 isBC: false 
 # Enable Single Sign-on 
 enableSSO: false 
 role: data+ingest 
 count: 1 
 storage: 10Gi 
 spec: 
 memory: 8Gi 
 cpu: 2 
 java_heap: -Xms4g -Xmx4g 
 kibana: 
 count: 1 
 enableSSO: false 
 enableSecrets: false 
 secrets: [ 
 ] 
 isBC: false 
auditbeat: 
 version: 7.7.0
</pre><h3>requirements.yaml</h3><pre class="prettyprint">dependencies: 
 - name: nginx 
 repository: file://../../infosec-demo/charts/nginx 
 version: 0.1.0 
 - name: eck 
 repository: file://../../infosec-demo/charts/eck 
 version: 0.1.0 
 - name: auditbeat 
 repository: file://../../infosec-demo/charts/auditbeat 
 version: 0.1.0
</pre>With the values.yml and dependencies above, we are able to quickly and consistently deploy an Elasticsearch cluster that meets all of our standards into a new K8s cluster. This helps us to efficiently meet the needs of our internal customers when a dedicated cluster is required in a new location. With this as our foundation, we are able to quickly upgrade or scale our existing clusters by simply tweaking the values.yml for each cluster. For example, if we wanted to upgrade to a newer version and add additional nodes to the above cluster, we would simply make these changes in our values.yml and then deploy:
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltdd70cd163477c486/5fb805dd46f622769b5f0322/blog-infosec-eck-1.png" data-sys-asset-uid="bltdd70cd163477c486" alt="" style="display: block; margin: auto;">
By keeping these changes confined to the single values.yml file, it becomes simple for a teammate to quickly review the PR and validate the changes that were made prior to deployment.
<h2>Safety</h2>Deploying build candidates can be a bit scary, but the benefits they provide our team outweighs the bad. By deploying build candidates, we enable our product teams to test and build content using real-life data. We minimize upgrade risks by scheduling hourly snapshots using <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.10/snapshot-lifecycle-management.html">snapshot lifecycle management (SLM)</a>, creating alerts (pod-activity, system performance, search latency, Metricbeat, etc., -), and having the flexibility to quickly rebuild a new cluster with our charts.
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt48e64b7624a2ca3a/5fb805f04a4abb73ff7a0058/blog-infosec-eck-2.png" data-sys-asset-uid="blt48e64b7624a2ca3a" alt="" style="display: block; margin: auto; width: 350px;">
<h2>Conclusion</h2>ECK and Helm have been a game changer for InfoSec. We went from managing dependencies and struggling to keep up with the pace of security, product releases, and human error to focusing on making our services better. We have significantly reduced our upgrade time from weeks to upgrading our entire environment in under two hours. With this approach, we no longer have to sacrifice platform stability to enhance security visibility.
If you are new to Kubernetes, <a href="https://www.elastic.co/downloads/elastic-cloud-on-kubernetes">download ECK</a> and enjoy the journey. Take your time understanding the basics and incrementally create your templates. This has been an exciting six-month journey for InfoSec and I can promise you, there is light at the end of the tunnel. Deploying infrastructure and remaining up to date doesn’t have to be a small project anymore and you can enjoy the awesome features Elastic has to offer now — $helm upgrade!

How-to & features

Beats

Orchestration

Elastic on Elastic: How InfoSec deploys infrastructure and stays up-to-date with ECK

Learn how the Driver and Vehicle Licensing Agency (DVLA) uses the Elastic Stack to connect all the dots in their services architecture. And find out how Elastic Training helped them do it faster.

The core responsibility of the UK's Driver and Vehicle Licensing Agency (DVLA) is to maintain more than 48 million driver records, more than 40 million vehicle records, and to collect approximately £6 billion ($7.75 billion) a year in Vehicle Excise Duty.
The agency is at the forefront of public digital services, and has made significant progress in transforming its IT systems into new cloud-based platforms.
DVLA’s Cloud Engineering team is integral to the delivery of a cloud-first strategy, and the team started using the Elastic Stack to centralize logs. The goal is to offer to all internal customers the ability to measure and monitor everything using all DVLA logs and metrics, enabling the platform users to analyze and quantify the performance and usage of their apps on the shared platform. This Elastic-driven observability model has resulted in an enhanced experience for internal as well as external DVLA customers.
As the Cloud Infrastructure Department started expanding its use case, the agency decided to also invest more in Elastic Services to ensure DVLA teams could grow their skills.
We recently sat down with Matt Dendle, Engineering Team Lead, to discuss how <a href="/training">Elastic Training</a> has been able to support him in this initiative.
<h3>What was the rationale behind designing your new service architecture?</h3>Matt: We have been undergoing several phases of change over the past decade, and while we had organically grown into the cloud, we also had some technical debt challenges, as well as some system interoperability problems that hindered us when it came to moving at the pace we wanted.
We started by implementing various programs to increase cloud adoption so we could centralize everything in one place. That’s when we noticed that the systems were not connecting with each other very well, generating data silos and security concerns among the different teams of the DVLA.
To address this issue, we adopted a new approach for our target architecture: the Open Services Landscape (OSL). With OSL, our goal was to enable channels across the agency to evolve and be replaced in a flexible and timely manner thanks to a shared architecture and the continuous improvement of services.
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltfad34d7fba2d85a8/5f0ccde90395fe7f7bb11001/blog-dvla-osl.jpg" data-sys-asset-uid="bltfad34d7fba2d85a8" alt="" style="display: block; margin: auto;">
<figcaption>The DVLA’s Open Services Landscape vision
</figcaption><h3>How does the Elastic Stack fit into the Open Services Landscape?</h3>Matt: One of our key focuses of OSL was to be able to measure and monitor everything. That’s where the Elastic Stack came into play. We worked on logging all of the compute workload and storing it in a centralized way to get the most benefit out of it.
Using the Elastic Stack we’ve been able to get all of our logs into one place. Thanks to the <a href="/blog/introducing-the-elastic-common-schema">Elastic Common Schema</a> and rollover indices, we were also able to ingest the logs from many different sources at scale.
By doing this we were able to connect all the dots and identify things we wouldn’t have been able to see without Elastic, whether it’s within a single application or between different applications and services.
At the business level this means we can get a consolidated view on things such as the evolution of our web transactions, and we make decisions based on this data. Using Kibana as a self-service dashboard tool has also made it easier for other teams to also exploit and monitor their logs in an autonomous way.
Overall, the Elastic Stack has been easy to scale and aligns with our OSL strategy that, at the end of the day, improves the customer experience.
<h3>What prompted you to work with our Elastic Training teams?</h3>Matt: When I joined the DVLA last year, I already had some knowledge of the Elastic Stack. But my knowledge was outdated. Furthermore, one of our key investment areas is building critical skills by allocating significant time towards technical training. Given the space the Elastic Stack occupies in our OSL approach, it just made sense to invest in Elastic Training in order to scale up our skills on the Elastic Stack.
There’s also a lot of value in getting training directly from Elastic employees. You can be sure to get answers to the questions you have from the trainers and, if not, they can connect you with the right stakeholders at Elastic to address your questions.
<h3>What value did Elastic Training bring to DVLA?</h3>Matt: Right off the bat, in each of these training courses, we were able to talk in more detail about what we wanted to achieve, as well as discuss the best approach to tackle our challenges. We got the opportunity to collaborate in a productive way on the different ideas we had.
We enrolled in three different courses:
<ul>
	<li>30 employees took <a href="/training/elasticsearch-engineer-1">Elastic Engineer 1</a></li>
	<li>17 employees took <a href="/training/elasticsearch-engineer-2">Elastic Engineer 2</a></li>
	<li>9 employees finished the Kibana Data Analyst training course (Editor’s Note: This course has been updated, expanded, and re-released as<a href="/training/data-analysis-with-kibana"> Data Analysis with Kibana</a>)</li>
</ul>In addition, bringing Elastic’s trainers and consultants to our facility gave us the flexibility to enable our staff all at once. We were able to blend official training with the expertise of their consulting practice to create a highly engaging, highly productive learning experience.1
<h3>After engaging with Elastic&nbsp;Training, does DVLA now have the right skills?</h3>Matt: Absolutely. Our teams feel better equipped in getting their hands onto the Elastic Stack and have started building their own dashboards. The training courses helped them grasp the potential of the Elastic Stack and how to move forward.
The training courses we’ve followed are really both a confidence and career boost for us at the DVLA. We’re also working on getting some of our engineers&nbsp;<a href="/training/certification">Elastic Certified</a>, myself included.
We want to be considered as a center of excellence for cloud and this means sharing more of our knowledge of Elastic across the government. This would allow us to help expand the usage of the Elastic Stack beyond just the <a href="https://www.gov.uk/government/organisations/driver-and-vehicle-licensing-agency/about">DVLA</a>.
Following on from these training courses, we’re also thinking of expanding our usage of the Elastic Stack within the DVLA. We started getting more interested in Elastic Security and now we’re engaged in creating a proof of concept for <a href="https://www.elastic.co/siem">Elastic’s SIEM</a> to see if this could be a good tool for us.
<h3>What advice would you give others about investing in Elastic Training?</h3>Matt: The&nbsp;<a href="/training/schedule">instructor-led courses</a> provided an immersive, guided learning experience and were well worth the investment. Coupling this training with&nbsp;<a href="/training/catalog">on-demand courses</a> has been instrumental in developing my skills and helping me prepare for the&nbsp;<a href="/training/elastic-certified-engineer-exam">Elastic Certified Engineer exam</a>. At the end of the day, having training from Elastic will allow you to fully leverage the Elastic Stack.
To learn more about DVLA's use case, watch Matt’s <a href="/webinars/dvla-centralised-logging-in-a-changing-environment">presentation</a> at Elastic's UK Government Symposium.
Start building your Elastic Stack knowledge with our<a href="/training/free"> free, on-demand Fundamentals courses</a>. When you're ready to take your skills to the next level, our<a href="/training/schedule"> instructor-led virtual courses</a> provide the hands-on practice needed to prepare for&nbsp;<a href="/training/certification">Elastic certification</a>.
<hr>1 Delivered pre COVID-19

blog-banner-training-certification.png

blog-thumb-training-certification.png

Elastic Training helps UK Driver and Vehicle Licensing Agency better serve motorists

Learn how the Elastic Stack is used across diverse industries to empower customers, communities, and, even, interplanetary exploration.

What do an airline, the world’s largest retailer, the French government, Adobe, and NASA’s JPL have in common? They use the <a href="https://www.elastic.co/elastic-stack/?elektra=blog" target="_self">Elastic Stack</a> to empower customers, communities, and, even, interplanetary exploration.&nbsp; With the Elastic Stack’s ability to take data from any source and in any format, and then search, analyze, and visualize it in real time, organizations can act quickly to improve customer experience and power critical systems. The flexibility of Elastic as a platform means that the same underlying technology can serve a broad range of challenges.&nbsp; See how companies across the globe are using Elastic to... <h2><a href="https://www.elastic.co/elasticon/tour/2019/dallas/how-walmart-is-combating-fraud-and-saving-consumers-millions" target="_self">Combat fraud and save consumers millions</a></h2> Walmart’s Global Investigations team has used Elastic to ingest more than 4 billion metadata records over the past 5 years to combat fraud and protect customers. By ingesting IP address, point of sale, and other traffic data into Elasticsearch, the Walmart Global Risk Analysis team can identify instances of fraud in real time, especially gift card scams targeted at senior citizens. They even support the FBI, Secret Service, and Homeland Security to dismantle transnational criminal groups that scam consumers for millions of dollars each year. <h2><a href="https://www.elastic.co/blog/image-recognition-and-search-at-adobe-with-elasticsearch-and-sensei" target="_self">Search images with machine learning</a></h2> Adobe Sensei artificial intelligence and machine learning technology does the heavy lifting for training the image search models. To work alongside Sensei, Adobe created several Elasticsearch plugins — a similarity plugin and another for search ranking — to help drive image recognition capability and create an effective tool for users to find similar images. Together, they provide users with real-time search features such as face detection, object detection, face clustering, auto tagging, and named entity recognition. <h2><a href="https://www.elastic.co/blog/why-french-ministry-armed-forces-picked-elastic-cloud-to-fight-covid-19-future-pandemics" target="_self">Track the scientific research around Covid-19</a></h2> WaKED-CO (Watch of Knowledge on Emergent Diseases COVID-19) launched in record time on Elastic Cloud — deployed just a month after developing a prototype — with one core mission: to make it easier to research the literature around the COVID-19 crisis. The teams faced underlying challenges: resources were distributed, researching academic literature was time consuming, publications were not exhaustive, and researchers had limited time. By creating a list of all the scientific and clinical test publications on their platform and enriching them with additional information, they aimed to make it easier to organize, sort, and gather information, as well as carry out scientific research and monitoring. <h2><a href="https://www.elastic.co/customers/airbus" target="_self">Power near real-time access to thousands of critical files&nbsp;</a></h2> Airbus, one of the world’s largest airlines, customer service provides support to airlines and their maintenance partners, certification authorities, and the numerous suppliers involved in the manufacturing of an aircraft — totaling approximately 80,000 internal and external users. So they developed an internal application for digitizing the large volumes of assorted aircraft reference documents with the objective of satisfying 3,000 requests per minute in less than 2 seconds. Powered by the Elastic Stack, this new-generation technology allows Airbus to guarantee access rights to authorized users and monitor the platform's health. <h2><a href="https://www.elastic.co/elasticon/conf/2015/sf/unlocking-interplanetary-datasets-with-real-time-search" target="_self">Monitor and measure spacecraft instruments</a></h2> Elastic technology helps NASA’s JPL, a hub for spacecraft building, monitor metrics from the Mars Curiosity rover and even fly a helicopter on Mars. With an analytics system built on Elasticsearch, the JPL team can quickly spot patterns and anomalies in the telemetry transmitted by the Mars rover. Elastic also powers the JPL’s internal website search to bring more relevant results to users searching for spacecraft documents and designs. All of these challenges have been turned into opportunities thanks to a search platform — a centralized data store, search engine, and analytics solution that can ingest, store, search, and analyze data of any type, from any source, and provide results in milliseconds. Find out how more companies are putting Elastic to work to <a href="https://www.elastic.co/customers/" target="_self">solve data challenges</a>.&nbsp;

elastic-stack-blog-banner.png

Media & entertainment

Retail

Travel & transportation

Automotive & manufacturing

elastic-stack-blog-thumb.png

How versatile is the Elastic Stack? Ask Walmart, NASA, or Airbus.

Learn how Play used Elastic Observability to unify their telco data - transport, RAN, HW, planning, CP, PM, environment, physical structures, topology, subcontractors, and more - to predict and resolve issues faster.

The telecommunications world is in the middle of its fourth industrial revolution. Organisations are trying to bring out as many new services as possible to monetise their infrastructure, but despite their modern approach, they still own and maintain legacy — and most importantly — multi-vendor infrastructures. Due to complex organisational structures and decentralised management systems, most responsibilities are divided between multiple departments. This means that operational departments experience difficulties due to siloed information and lack of communication. Lack of access to data prevents teams from reacting to issues quickly, which can impact their relationship with subscribers. Many operators have already realised the need to pursue unification of operational data and improvement in issue detection at speed and scale. Otherwise customer attrition is unavoidable. In an environment like this, there is a very important need to bring everything into one unified picture. This centralisation of data is foundational to AIOps, and it enables telecommunication teams to remove access barriers and improve analysis speed within their stored data. <a href="https://www.elastic.co/observability" target="_self">Elastic Observability</a> can help telco companies do this by seamlessly fitting into an existing ecosystem and creating a unified view of their data. In this blog post, we'll take a look at how <a href="https://www.play.pl/nowy/" target="_self">Play</a> — the biggest mobile operator in Poland —&nbsp; did just this with the Elastic Stack. <h2>Playing with Elastic&nbsp;</h2>Play has successfully optimised its operational workforce with thorough planning and data-driven operations. They've created a transport network management model that is built on data normalisation and unification between vendors and network elements. This unification creates a great source of data for further analysis. Without advanced data analytics tools, Play had limited visibility into issues, which is very important for resolution. <img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltc0c63a3f5991c7c6/5fda6da81166ce7d2ed180c1/blog-play-telco-1.jpg" data-sys-asset-uid="bltc0c63a3f5991c7c6"/><figcaption>Play's transport network management model </figcaption>With this new model, it turned out that the core foundation was almost entirely built already. The only new additions were 1) the data ingest layer, which required slight reorganisation and denormalisation to make it ready for future, and 2) the unified datastore for analysis. With its lightning speed and nearly limitless scalability, Play decided on the Elastic Stack as the core for data analysis and the extremely important visualisation layer. Let's take a look at what Play has been able to achieve with this new, unified architecture. <h2>Radio line monitoring with Elastic</h2>Radio lines are necessary to provide communication over vast areas where fixed line connections are not viable from an economic perspective. They provide the link between core network elements and endpoints like eNB, NodeB and BTS. They are a crucial part of infrastructure which requires the shortest possible mean time to resolution (MTTR) to ensure network quality and availability. <img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt5ae2cc4f73aa74e5/5fda6dd8a7955d1a6a88ef6e/blog-play-telco-2.png" data-sys-asset-uid="blt5ae2cc4f73aa74e5"/>The Elastic Stack comes as an integrated suite of components, from data ingest through storage, processing and analysis to presentation and automated notification. Each stage of the data processing pipeline is designed to ensure the highest level of productivity. Let's take a look at the steps Play took to monitor their microwave links with the help of Elastic. <h3>Data ingest: How to make it simple</h3>One of the fastest ways to retrieve data from a complex system and ingest it into the Elastic Stack is through database integration. Teams usually have expertise in SQL, so it is fairly easy to write the necessary queries. Relational database management systems (RDBMS) also enable customers to create views on data and make enrichment (joins) on the SQL level. Play’s infrastructure (transmission infrastructure) data is stored in a RDBMS in curated form, so it was relatively straightforward to load it into the Elastic Stack. <img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltbd4f7aea05e6a1d0/5fda6e3402fd0c7d345f0921/blog-play-telco-3.png" data-sys-asset-uid="bltbd4f7aea05e6a1d0"/><h2>Data enrichment: Why is it so important?</h2>Most of the available systems provide a limited view, as they are specialised in one area. Today, system capability enables organisations to store much more data and, most importantly, process it faster. It is possible to merge multiple domains like planning, inventory management, performance management, fault management, configuration management, software management and environmental monitoring.&nbsp; Having all information in a single space enables entirely new analytical possibilities, leading to much faster issue resolution and new discoveries. Areas of interest that can be covered are:&nbsp; <ul><li>Understanding the influence of multiple factors on observed measurements<ul><li>The contribution of human factors to error rate</li><li>Understanding network element performance by element type, vendor, location, type of installation</li><li>The contribution of weather and environmental conditions to measurements</li></ul></li><li>Overview of the platform and surrounding systems and connections<ul><li>Much faster problem source identification&nbsp;</li></ul></li></ul>Another aspect of this attitude is that it helps to find hidden information in existing data sets that were previously unknown ("unknown unknowns"). A good example is calculating temperature changes in the time domain, which can give information about rapid changes that in turn can point to rainfall in certain areas. Rain has a high impact on radio transmission links, but it is a temporary phenomenon. From an operations point of view, this information should be filtered out so as not to disturb decisions about real and durable failures. On the other hand, being able to see link performance during rain is a good filter for optimisation analysis. This is alternative and quite precise (temperature taken at exact point of interest) information which can be a good filter for <a href="https://www.elastic.co/what-is/elasticsearch-machine-learning" target="_self">machine learning</a> processes. <img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt3799ddeff5941ffe/5fda6e4d396e65084a9e50d6/blog-play-telco-4.jpg" data-sys-asset-uid="blt3799ddeff5941ffe"/><figcaption>Temperature </figcaption><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blta8f8068800103d95/5fda6e6db47cdf7fc7d68644/blog-play-telco-5.jpg" data-sys-asset-uid="blta8f8068800103d95"/><figcaption>Temperature + Rain</figcaption><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt77c7462334fc53e5/5fda6e62396e65084a9e50da/blog-play-telco-6.jpg" data-sys-asset-uid="blt77c7462334fc53e5"/><figcaption>Rain </figcaption> <h3>Unified storage and processing</h3>Elasticsearch is the core of the Elastic Stack, and it's used to store and analyse data. It can store any type of data, making it the <a href="https://www.elastic.co/observability" target="_self">perfect observability tool</a>. It enables users to answer all possible questions based on all&nbsp;their data set: transport, RAN, HW, planning, CP, PM, environment, physical structures, topology subcontractors, and more. The Elastic Stack enables users to combine various types of data, including static and time series data. This, along with shared features for various types of data, provides a wide range of analysis and presentation possibilities. By applying this new paradigm, users will see much more and much faster, which affects the convenience and speed of troubleshooting. <h2>Data presentation: Where analysis meets visualisations</h2>Play’s transmission infrastructure is distributed across vast areas. Being able to automate failure detection processes accelerates problem resolution.&nbsp; <h3>Automated incident detection using machine learning</h3>Anomaly detection in the Elastic Stack provides unsupervised methods to find all data points that can be used to indicate a problem. Analysis is spread across all radio links (thousands analysed at the same time). No static rules have to be set — it just requires a simple job to be set for all entities at once, which only takes about 3 minutes. Further analysis can be done immediately after detection and notification through a link between anomalies and user-defined dashboards. This dashboard view is limited to time and entity (influencer) involved in the anomaly.&nbsp; The screenshot below shows an anomaly detection dashboard indicating significant anomalies in RSL (received signal level). RSL defines link quality; a signal that is too low causes communication problems.&nbsp; <img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt37406414b5768ca3/5fda6fbd8884c20ac3ca2b33/blog-play-telco-8.jpg" data-sys-asset-uid="blt37406414b5768ca3"/> <h2></h2><h3>Anomaly explanation with custom-built dashboards</h3>Anomalies can be explained further using a direct link between the anomaly detected and a dashboard. Most importantly, the dashboard shows the exact time and device that is experiencing a problem. The dashboards below show how such visualisations can present a very informative overview. They can present static inventory data such as: <ul><li>Radio line settings</li><li>Installation type: tower type, height</li><li>ODU and modem type</li><li>Date of installation&nbsp;</li><li>Geographic location</li></ul>As well as dynamic parameters such as: <ul><li>RSL</li><li>Calculated planned RSL</li><li>SNR</li><li>QOS</li><li>Modulation</li><li>Throughputs</li><li>Link capacity</li><li>Current link capacity calculated from modulation</li><li>Errors</li></ul><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltb767587328e252ee/5fda6fd9396e65084a9e50e2/blog-play-telco-9.jpg" data-sys-asset-uid="bltb767587328e252ee"/><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt7055ad1eb7d1f099/5fda7005b721aa1cd8dc0716/blog-play-telco-10.jpg" data-sys-asset-uid="blt7055ad1eb7d1f099"/><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltaa7d291cfe20761d/5fda7017b721aa1cd8dc071a/blog-play-telco-11.jpg" data-sys-asset-uid="bltaa7d291cfe20761d"/> <h3></h3><h3>Geo visualisations</h3>Elastic Maps enables users to overlay information such as wind, temperature, rain, storms and humidity. Being able to visualise this geospatial data is very useful for monitoring radio line behaviour: <img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltc6f6dd920da90efb/5fda70141166ce7d2ed180c9/blog-play-telco-12.jpg" data-sys-asset-uid="bltc6f6dd920da90efb"/> <h3></h3> <h2>Conclusion</h2>Play has used the Elastic Stack to significantly reduce the time required to set up and provision rules for problem detection. With <a href="https://www.elastic.co/observability" target="_self">Elastic</a>, Play can identify anomalies in their high-scale infrastructure with minimal effort, and&nbsp;visualisations are instantly available.&nbsp;This&nbsp;enables the operations team to identify, target and report problems in minutes.&nbsp; <hr/><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt7cd079707d2d1e3b/5f4fc584b5badb2465bb95c8/BartekPodles_.jpg" data-sys-asset-uid="blt7cd079707d2d1e3b" width="150"/>Bartłomiej Podleś is a Transmission Operations Manager at <a href="https://www.play.pl/nowy/" target="_self">Play</a>. He has used his 20 years of industry experience to set up RAN and Transmission operations team at Play from scratch. With expertise across optimal telco technological processes, including creating long term maintenance software, he is currently focused on cross-domain preventive maintenance solutions based on big data and machine learning for zero-touch/self-repairing processes for network maintenance. He is also an aficionado of vintage Japanese motorcycles!

blog-banner-play-telco.png

blog-thumb-play-telco.png

Play: Modernizing telecommunications with the Elastic Stack

Cerner was challenged by the massive amount of data that they log to monitor the health of their infrastructure. In response, Cerner chose Elasticsearch as their logging solution to help shore up gaps in data quality.

Cerner Corp. is a supplier of healthcare information technology systems, services, and devices. The company, with $5.7 billion in annual revenue, empowers people and communities to engage in their own care. A key aspect of the business is surfacing data to enable their clients to make informed decisions about their healthcare.&nbsp;The 29,000 Cerner employees in 30 countries are on a mission to shape the healthcare of tomorrow. They believe that their influence goes beyond healthcare, and impacts the world.Jim Avazpour, Director of Infrastructure, says Cerner was challenged by the massive amount of data that they log — network storage, virtualization, and supporting services —&nbsp;to monitor the health of its infrastructure. In response, Cerner chose Elasticsearch as their logging solution to help shore up gaps in data quality. They also cite scalability and resiliency, and cost-effectiveness, as key reasons for choosing Elastic.&nbsp;Elastic helps log and monitor the Cerner environment continuously. With data gaps filled, Cerner no longer misses critical alerts of performance violations in the company’s environment, Avazpour says.<h2>Lowering MTTK with Elastic machine learning&nbsp;</h2>With the introduction of Elastic, Cerner’s observability data pipeline now includes two data feeds.&nbsp;Data Feed 1 captures all the industry standard alerting messages, which are readable by people. Alerts in Data Feed 2, however, are generated for every violation. As soon as violations occur, alerts not immediately read by humans are generated and sent to Elasticsearch.“The amount of data that we collect on a daily basis from our devices in our infrastructure amounts to about 5 billion to 8 billion messages per day,” Avazpour says. “That's a very large amount of data that's flowing through our monitoring solution.”Using machine learning on Data Feed 2, Cerner is able to find anomalous behavior that was not detected on Data Feed 1. In the end, this two-feed configuration empowers engineers to prevent downtime that would hinder the customer experience.This two-feed approach has paved the way for dramatically lowering the team’s mean time to knowledge (MTTK) to find the root cause of issues and resolve them. Harnessing Logstash, Elasticsearch, and a home-brewed network crawler, Avazpour says Cerner has been able to reduce MTTK by 75%.“We've been able to use machine learning to be able to baseline and look for deviation from normal to look for chronic issues, look for anomalies in performance and then be able to generate an alert from that and then be able to notify our engineers to be able to troubleshoot and try to fix issues proactively before our clients were impacted,” Avazpour says.Watch the <a href="/elasticon/archive/2020/global/realizing-your-aiops-goals-with-machine-learning-in-elastic">full presentation</a> to learn more about how Cerner deploys the Elastic Stack and machine learning to drive a successful <a href="https://www.elastic.co/observability" target="_self">observability platform</a>. The video begins with talks by Elastic’s Mukesh Gadiya, Sr. Manager of Product Management, and Tom Grabowski, Principal Product Manager. They discuss the benefits of Elastic Observability with machine learning.

Healthcare

blog-thumb-cerner.png

Cerner depends on Elastic machine learning for a healthy infrastructure

At TierPoint, we’re constantly looking for new and even more effective ways to aggregate, process, and make decisions from massive amounts of data streaming in from diverse sources. That’s why we chose the Elastic Stack.

TierPoint is a leading provider of secure, connected data center and cloud solutions at the edge of the Internet with thousands of customers.&nbsp;At TierPoint, I’m responsible for maintenance and development of the information security program, which includes threat analytics, incident response, and digital forensics. We’re constantly looking for new and even more effective ways to aggregate, process, and make decisions from massive amounts of data streaming in from diverse sources. Nowhere are such efforts more important than in the realm of IT security.&nbsp;We wanted a strong analytics platform effective at helping us further enhance our robust IT security protocols. Among other requirements, the platform we sought needed to easily ingest, analyze, and enrich threat data from disparate inputs, like log events and emerging-threat feeds from various sources. This would allow us to make better-informed decisions even faster at a time when attacks are becoming more complex, more frequent, and sponsored by nation states and criminal organizations.&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;That’s why we chose the Elastic Stack, which helped us reduce investigation times in some cases by as much 400%. From a forensics perspective, speed and accuracy make all the difference.<h2>Hacking sophistication demands dynamic IT security systems</h2>The constant evolution of hacker sophistication requires a better understanding of “normal activity” to enable organizations to improve their detection of anomalies, shifts in the cyber threat landscape, as well as emerging threats. In other words, robust data analysis that enables the improved definition of a current state — evaluated against yesterday, last week, last quarter, and last year — equates to the quicker identification of events not seen before.We have embraced a tool that can analyze this disparate data and combine it with other sources. This platform centralizes information and better automates the process of updating key stakeholders on historical trends, shifting attack vectors, existing levels of protection, and real-time corrections required to enhance those levels of protection.&nbsp;We moved away from limited, pre-defined rule sets and expanded our ability to more quickly understand and respond to new types of data. With Elastic, TierPoint is achieving more centralized data analysis, in addition to getting a “single pane of glass” view of our security data with Kibana, regardless of its source.&nbsp;<h2>Leveraging Elastic beyond the security use case</h2>During our time harnessing Logstash to ingest security data into Elasticsearch, we realized the ease by which the Elastic Stack could provide additional analytics of data sources that extend beyond the capabilities of a traditional <a href="https://www.elastic.co/security/siem" target="_self">SIEM</a> platform. Elastic is currently helping us gain augmented insights on overall DNS traffic and flows across our internal networks, which is enabling us to identify ways in which those networks might be further optimized.&nbsp;This was a big lesson learned, and underscores that Elastic can be leveraged beyond information security to add more value to your business.&nbsp;Mark Sellers has more than 20 years of experience in the service provider space. He is a standing member of ISACA and ISSA International. As a vice president, Sellers manages the team of professionals responsible for maintenance and development of TierPoint’s information security program, including threat analytics, incident response, and digital forensics.

blog-banner-blue-circuit-locks.jpg

blog-thumb-blue-circuit-locks.jpg

Deploying Elastic to further strengthen IT security at TierPoint

A security analyst at Orange Business Services shares how they integrated Elastic into their existing SIEM, resulting in faster investigations and saving their engineers’ time

I’m a security analyst at Orange Business Services in Paris, and one of my current projects for the Orange Group is implementing a new SIEM based on the Elastic Stack. In this blog post, I’ll share why we chose Elastic and how we were able to integrate Elastic into our existing SIEM, resulting in faster investigations and saving our engineers’ time.
So follow along:
Orange Group is a multi-service network operator and digital service provider in 26 countries and serves about 253 million customers with 147,000 employees. Orange Business Services, the unit dedicated to businesses and organizations, focuses on digital services. From connectivity, the Internet of Things (IoT), and the cloud to artificial intelligence (AI), application development, and cybersecurity, Orange Business Services works with its clients to harness the true value of their data, helping them every step of the way.
<h2>Integrating the Elastic Stack into our SIEM strategy</h2>The Orange International Network Infrastructure (OINIS) department, which is in charge of building the Orange group’s international network, identified the need for a modern security monitoring solution capable of keeping up with the constant changes in the fast-paced world of security. As a result, two of our departments recently decided to collaborate on redesigning and modernizing the Orange Business Services SIEM infrastructure, which has been dubbed “EYES.” The key criteria for selecting our new solution were speed, simplicity of data integration, and better visualization tools.
Elastic had piqued our interest as early as 2018. We were finally convinced to take the plunge with Elastic after the ElasticON Tour Toronto and hearing the feedback from the security team at <a href="https://www.elastic.co/elasticon/tour/2018/toronto/security-events-logging-bell-canada">Bell Canada</a> (before Elastic introduced its own turnkey SIEM solution, which we will be rolling out soon). Finding themselves in a similar situation to our own, the team at Bell Canada had quickly taken the initiative of completely replacing their SIEM with the Elastic Stack, which was capable of ingesting all of their security logs, automatically detecting threats, and standardizing the format of their logs. We were also interested in Elastic’s machine learning features, which promised to help us modernize our SIEM approach.
We decided to implement the Elastic Stack for a functional proof of concept (POC) in less than three months. However, we took a slightly different approach than Bell Canada, in that our SIEM solution is still being used for its correlation engine. Following our three-month-long POC, we copied all our logs into the new infrastructure and enabled the Elastic Stack’s security features to protect access to our clusters. We started by learning to use the Elastic Stack ourselves; we were able to find all the necessary information to get started on the Elastic blog. Later on, we started working closely with the teams at Elastic in order to roll out new features. Elastic’s Support team has always proven to be very responsive, whether we came to them with application debugging questions or requests for advice on integration, or even our security architecture.
Our infrastructure evolved very quickly as a result:
<ul>
	<li aria-level="1">We separated the Dev, Preprod, and Prod environments, allowing us to quickly test the new features offered by Elastic and determine whether we wanted to go into production with them.</li>
	<li aria-level="1">Logstash also quickly went from a simple parsing tool to an essential tool for enriching our logs.&nbsp; <img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt9a37b648de42db12/60f0ab59b86fce146acbe04a/blog-orange-business-architecture-diagram.png" data-sys-asset-uid="blt9a37b648de42db12" alt="blog-orange-business-architecture-diagram.png" "=""></li><figcaption>Architecture diagram</figcaption>
	<li aria-level="1">We also applied the hot/warm/cold formats for managing the lifecycle of our indices, allowing us to improve both performance and security.&nbsp;</li>
	<li aria-level="1">Finally, we added a monitoring cluster to collect the metrics for the various components of our systems and keep a close eye on the performance of our clusters.</li>
</ul><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltefc777c7144bd385/60f0abcb487cea2ee9e6530a/blog-orange-business-server-distribution-diagram.png" style="background-color: initial;" data-sys-asset-uid="bltefc777c7144bd385" alt="blog-orange-business-server-distribution-diagram.png">
<figcaption>Server distribution diagram</figcaption>In these architectures, we use the products in the Elastic Stack (Elasticsearch, Logstash, Kibana) and have implemented Beats for a handful of very specific needs (application logs, NetFlow) that have not yet been standardized. We now have around 20 Elasticsearch nodes spread across two geographic sites. With all components included, this makes for a total of 80 servers, nearly 100 billion logs indexed, and several dozen terabytes of data.
<h2>A comprehensive overview of the speed and performance of Elastic Stack</h2>Following the integration of the Elastic Stack, we quickly saw an improvement in our SOC investigation abilities. Thanks to our new infrastructure, we can retrieve the results of our investigations much more quickly, and also benefit from all of the data regarding the security of our information systems (inventories, network mapping, management of computer equipment, alerts for external equipment, etc.) being centralized. This in turn gives us a greater degree of control over the ingestion of our logs, and we have implemented a dashboard that allows us to view the quality of our various pipelines so our SIEM engineers no longer have to waste time investigating whether a log format has been modified following the introduction of a new application or version.
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blta63bdb6b10784dd7/60f0ac2b487cea2ee9e6530e/blog-orange-business-ingestion-pipeline-performance.png" data-sys-asset-uid="blta63bdb6b10784dd7" alt="blog-orange-business-ingestion-pipeline-performance.png">
<figcaption>Dashboard displaying the performance of our ingestion pipelines</figcaption>We are now capable of detecting network anomalies more quickly and no longer have to query multiple databases to obtain a single piece of information. Since Orange Business Services has over 1,000 pieces of internal network equipment (LAN), the enrichment of the information taken from IPAM (IP address management) allows us to plot all of our equipment on a map and track the communications that have occurred.
Adopting the Elastic Stack also proved to be quick, as our users make use of Kibana on a daily basis to view their data, and Spaces also allows us to take that one step further by separating out the various workspaces according to their use case (for example, providing access to anonymized logs or masking IP fields).
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltd601ba3dbaa43cf4/60f0ac6136954912f039ac48/blog-orange-business-kibana-spaces.png" data-sys-asset-uid="bltd601ba3dbaa43cf4" alt="blog-orange-business-kibana-spaces.png">
<figcaption>Diagram of distribution of workspaces with Kibana Spaces</figcaption>We have also integrated the <a href="https://www.elastic.co/guide/en/ecs/current/index.html">Elastic Common Schema</a> (ECS). We realized the importance of having data that follows this model before being able to get started with machine learning and SIEM. Once ECS had been implemented, we were able to work on new security scenarios based on Elastic machine learning features. In the space of just a few weeks, this proved to be successful, allowing us to detect anomalies that were previously undetectable with a traditional correlation engine. The ease of creating machine learning jobs also made it easier for us to implement these “dynamic” rules.
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt01f6fbcc2b27ccaa/60f0aca836954912f039ac4c/blog-orange-business-siem-process.png" data-sys-asset-uid="blt01f6fbcc2b27ccaa" alt="blog-orange-business-siem-process.png">
<figcaption>Process followed by our SIEM engineers</figcaption>This is now the default process followed by our SIEM engineers. The SOC comes into play at stage 5, analyzing and making use of the alerts raised by the various machine learning jobs.
<h2>Next steps: Integrating Elastic Security</h2>Our first steps with the Elastic Stack proved to be successful, and we wish to continue improving our anomaly and intrusion detection techniques by implementing additional tools offered by Elastic. We intend to increase the number of servers and the types of logs collected with the Elastic Stack. We’re also migrating our cold nodes to a new infrastructure based on the Amazon S3 API, which will allow us to use the snapshot features of Kibana — allowing us to increase our data retention capacity without increasing our storage costs. What’s more, we will continue to improve the enrichment of our security logs so as to better identify all the individuals on our intranet and detect intruders. We are also currently developing a “control tower” interface to take our use of Kibana even further with <a href="https://www.elastic.co/blog/getting-started-with-canvas-in-kibana">Canvas</a>.
We would also like to improve our use of machine learning and make use of <a href="https://www.elastic.co/security">Elastic Security</a>. For one, we will be testing <a href="https://www.elastic.co/siem">Elastic Security for SIEM </a>— and, more specifically, the correlation rules — in an effort to improve the automation of threat detection, and we would also like to implement Elastic Security for <a href="https://www.elastic.co/endpoint-security/">Endpoint security</a>. Pairing the SIEM tool with endpoint security will allow us to improve our security orchestration, automation, and response (SOAR) capabilities, thus providing us with a welcome boost in productivity. Concentrating all our security tools into a single solution will also allow us to simplify investigations. We hope that the data generated by machine learning and endpoint security will supplement the data collected by our security equipment in a helpful way. Finally, this will strengthen the security perimeter monitored by EYES, particularly on our servers.
Jocelyn Renouf has been a security analyst at Orange Business Services in Paris for over two years and is currently focusing on implementing a new SIEM based on the Elastic Stack.

blog-banner-gears-close.jpg

blog-thumb-gears-close.jpg

How Orange Business Services is building a better SIEM with Elastic

The Elastic Stack is a far-reaching ecosystem with many interconnected parts, and the <a href="https://forge.puppet.com/elastic">official Puppet Modules</a> are no exception when it comes to the configuration management side of the house. Today, we want to highlight some developments taking place in the Elastic Puppet ecosystem to help spread awareness of new features and upcoming changes.<h3 id="modules-roadmap">Modules Roadmap</h3>First, some housekeeping items that pertain to all of our official modules.<h4 id="puppet-3">Puppet 3</h4>The most prominent upcoming change is the deprecation of Puppet 3 (if you’re skimming this post, don’t panic! You still have time). As noted in the upstream Puppet Labs page, version 3 met its end in December 2016, which means that it is not only deprecated, but end-of-life, and past the cutoff date for any future updates. As systems operators ourselves, we recognize that keeping stride with software versions is one of the challenges that comes with the discipline, particularly when it comes to large enterprises.With that in mind, we’ll be keeping version 3.8 of Puppet in our list of supported versions for a period to ensure everyone has a window of time to work in, for those cases where upgrading is a challenge. Nevertheless, the Puppet ecosystem is moving on from version 3, and we can only hold on to old versions for so long. It’s for this reason that we will be removing support for Puppet 3 with the 6.x release of the Elastic Stack, which grants many months of leeway for those operations engineers with a Puppet upgrade still on their plate. Version 6 of the&nbsp;Elastic Stack is likely to arrive in late 2017, and will be prefaced with additional announcements as its release draws closer.<h4 id="new-forge-namespace">New Forge Namespace</h4>What open source project doesn’t like changing names every so often? <a href="https://forge.puppet.com/modules?limit=25&sort_by=rank&module_groups=base%20pe_only&q=elastic">Our modules on the Puppet Forge</a> now properly live under the “elastic” namespace where all future module updates and release will land. While any references to old modules under the elasticsearch namespace will continue to work, the modules there are no longer updated. We suggest updating any automation or tooling that references released modules from the Forge to this new namespace (for example, to elastic/elasticsearch).<h3 id="official-puppet-kibana-module">Official Puppet Kibana Module</h3>One of the most common requests we hear from Puppet users is for other parts of our stack to ship official modules. At Elastic{ON} 2017 we released a beta version of <a href="https://forge.puppet.com/elastic/kibana">an official Puppet Kibana module</a>, which is now formally released at version 5.0 (major version synchronized with the rest of the Elastic Stack). The module supports everything you would expect from our Puppet modules, including Kibana plugin resources to make managing Kibana through Puppet a much easier experience.As always, we’re eager to hear from you regarding any bugs you've found or features you’d like to see and would encourage any bug reports or feature requests to be filed in the <a href="https://github.com/elastic/puppet-kibana">puppet-kibana GitHub repository</a>.<h3 id="elasticsearch-tasty-new-features">Elasticsearch: Tasty New Features</h3>The official Elasticsearch module hasn’t stood still either. In addition to the steady march of bugfixes, <a href="https://forge.puppet.com/elastic/elasticsearch">the official Elasticsearch module</a> supports a couple of useful new resource types: indices and pipelines.While the module already supported managing index templates (useful for auto-created indices from tools like Logstash or Beats), the elasticsearch::index type can help manage mappings and settings for existing indices in an Elasticsearch cluster. For example, to manage an Elasticsearch index called “logs” to retain only 1 replica per shard:<pre class="prettyprint">elasticsearch::index { 'logs': settings =&gt; { 'index' =&gt; { 'number_of_replicas' =&gt; 1 } } }</pre>Likewise, <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/pipeline.html">pipelines</a> are now supported as well. The ability to codify processing pipelines in Puppet can help centralize your workflow for processing logs and documents in Elasticsearch (see our <a href="https://www.elastic.co/blog/new-way-to-ingest-part-1">other blog posts about ingest node pipelines for more introductory content</a>). For example, to create a pipeline that will automatically parse dates, the following Puppet resource could be used:<pre class="prettyprint">elasticsearch::pipeline { ‘parsedate’: content =&gt; { 'description' =&gt; 'Parse the date field into @timestamp', 'processors' =&gt; [{ 'date' =&gt; { 'field' =&gt; 'date', 'formats' =&gt; ['dd/MM/yyyy hh:mm:ss'], 'timezone' =&gt; 'US/Pacific' } }] } }</pre><h3 id="logstash-all-aboard-for-version-5">Logstash: All Aboard for Version 5</h3>Version 5 of Logstash brought huge improvements to the handling of the Logstash system service, adding full support for systemd, among other things. In the past, the Puppet module was responsible for much more of the work around service handling, which was a common source of problems as the module became progressively more complex in order to handle various edge-cases across distributions. With Logstash itself now handling the service, the Puppet module is much simpler and more stable when it comes to service management.It's important to note, however, that the changes in service management were so comprehensive that it became impractical for the module to support Logstash 2.x and Logstash 5.x at the same time. Thus, module version 5 only supports Logstash 5. Of course, the legacy module <a href="https://forge.puppet.com/elasticsearch/logstash">elasticsearch/logstash</a> is still available if you need to run Logstash 2.It's now possible to fully <a href="https://github.com/elastic/puppet-logstash#logstash-config-files">manage Logstash settings, startup options, and JVM options with the module</a>. Like this:<pre class="prettyprint">class { 'logstash': settings =&gt; { 'pipeline' =&gt; { 'batch' =&gt; { 'size' =&gt; 25, 'delay =&gt; 5, } } } }</pre>Since the settings are all defined as standard arrays and hashes, they work well with Hiera too.Plugin support is now comprehensive, with support for remote locations like puppet:// and http:// URLs. The latter is essential for people choosing to install the new <a href="https://www.elastic.co/guide/en/logstash/master/configuring-logstash.html" target="_self">X-Pack for Logstash</a>.Finally, a lot of progress on the module is something you probably won't see, but hopefully you get to see the results. All new features, including those above, are now developed under testing from a <a href="https://github.com/elastic/puppet-logstash/blob/master/spec/acceptance/class_logstash_spec.rb">re-written test suite</a>. The tests are there to ensure everything in the module works reliably, across every supported operating system and Puppet version.

wooden-mannequin-bleedimage.jpg

wooden-mannequin-thumb.jpg

Puppet State of the Union: May 2017

When I added rivers way back in time in the early days of Elasticsearch, the idea was somewhat novel. One of the first tasks that users do when using Elasticsearch is to get data into Elasticsearch to make it searchable, so why not allow our community to write plugins that can be installed directly on an Elasticsearch cluster&nbsp;to pull data into Elasticsearch automatically.The first few rivers implementations were quite successful and very helpful. The CouchDB river was immensely simple and popular (thanks to CouchDB changes API). Others were popular as well, for example, the RabbitMQ one. They did start to slowly show the problematic nature of rivers as well.What was the problem we were witnessing? Cluster stability. You see, by their nature, rivers deal with external systems, and those external systems require external libraries to work with. Those are great to use, but they come with an overhead. Part of it is built in overhead, things like additional memory usage, more sockets, file descriptors and so on. Others, sadly, are bugs.Part of our efforts in the past couple of years has been to improve Elasticsearch resiliency, and we kept seeing, time and time again, that rivers are a big cause for cluster instability, due to their inherent notion of working with external systems and external libraries. When Found joined us a couple of months ago, we found that they see the same thing, with rivers plugins causing most of the cluster instabilities across the thousands of clusters under management.We knew it for some time, and in the spirit of helping users build more resilient systems, we decided to deprecate rivers and ask users to focus on&nbsp;getting data to Elasticsearch from "outside" the cluster. Rivers are deprecated from 1.5 moving forward. We will probably keep the infrastructure around in 2.0, and only remove them at a later version, to ease the migration.The ease of use in getting data into Elasticsearch is still important though, so where should you go from here?For more than a year, we've had official client libraries for Elasticsearch in most programming languages. It means that hooking into your application and getting data through an existing codebase should be relatively simple. This technique also allows to easily munge the data before it gets to Elasticsearch. A common example is an application that already used an ORM to map the domain model to a database, and hooking and indexing the domain model back to Elasticsearch tends to be simple to implement. Also, Logstash, or similar tools, can be used to ship data into Elasticsearch. For example, some of the rivers Elasticsearch came with are now implemented as Logstash plugins (like the <a href="https://github.com/logstash-plugins/logstash-input-couchdb_changes">CouchDB one</a>) in the forthcoming&nbsp;Logstash 1.5. I love the community and work that has gone into a vibrant set of river plugins, and the decision to deprecate rivers was not taken lightly. What should you do if you are a river plugin author? Since rivers are relatively self-sufficient, the code can be extracted into a common library that can be used to get data into Elasticsearch. Then, this library can be used in various different places. A simple "main class" can be written to allow to execute it as a standalone process. Another option is to move the plugin to be a Logstash input.&nbsp;Logstash inputs are very simple to write, and in 1.5 the Logstash team has <a href="https://www.elastic.co/guide/en/logstash/current/input-plugins.html" target="_self">made the process</a> of writing, maintaining, and discovering plugins super simple. It is a hard decision to take something away, especially with all the effort that has gone into writing those river plugins by the wonderful authors. I deeply apologize for it, and we would love to help out with any questions and ideas on how to move forward adapting them. The issue for it is <a href="https://github.com/elastic/elasticsearch/issues/10345">#10345</a>.

Deprecating Rivers

<img src="https://api.contentstack.io/v2/assets/575e4c8f9e7a83165490c1d0/download?uid=blt0a3be37aa959b31f?uid=blt0a3be37aa959b31f" data-sys-asset-uid="blt140adad6df1b0753" style="float: right;margin: 0 0 0 20px;text-align: right;"/>Since the recently released 1.0.0-beta2 version, Packetbeat can understand the MongoDB wire protocol. This means you can now use Packetbeat and Elasticsearch to monitor the performance of your MongoDB servers. You can play with a live demo of the associated Kibana dashboard <a href="http://demo.elastic.co/packetbeat/#/dashboard/MongoDB-performance" target="_blank">here</a>.The way Packetbeat works is that it captures the network traffic, correlates the request with the response, and inserts a document into Elasticsearch describing each MongoDB query or operation seen on the wire. You can then use Kibana to discover errors or slow queries and to visualize things like “Top slowest MongoDB queries”, “Response time percentiles for a particular collection”, or the “Number of writes with unacknowledged write concern”. But more about these visualizations later, let’s start by giving credit where credit is due.<h2>Based on a community contribution</h2>The beauty of having a system like Packetbeat be open source is that anyone can add support for the protocols they need. By making it easy to add new protocols and encouraging community contributions, we’re not only extending Packetbeat’s list of supported protocols faster, but we benefit from having experts in each technology providing insights. The open source nature also means that folks can implement support for niche or proprietary protocols.While we hoped that people will start creating their own protocol modules for Packetbeat, we didn’t expect it to happen this quickly. We don’t have yet a proper developer guide for new protocols and the APIs are scarcely documented and may be inconsistent here and there. But all of&nbsp;this didn’t stop <a href="https://github.com/albanm">Alban Mouton</a> to submit the pull request for adding MongoDB support just a few days after we announced that <a href="https://www.elastic.co/blog/welcome-packetbeat-tudor-monica">Packetbeat has joined Elastic</a>. His pull request was well documented, contained unit and integration tests and even a simple <a href="https://github.com/elastic/packetbeat/blob/8f42d0fe0c20d65c8f6965fef695fd807ef5a6c1/SUPPORT_PROTOCOL.md">guide</a> to future protocol module writers.As if he didn’t do enough already, we asked Alban to contribute to this blog post as well. He sent us the following quote:"I am a enthusiastic user of Packetbeat, along with the ELK stack. The company I work for develops and integrates multiple services communicating mostly over HTTP. Tracing activity and performance at the network level is great because it separates us from the varying implementations of these services and makes system-wide analysis easy. We are at the beginning of our experience with Packetbeat but I have high hopes for the future.""Recently our main database has been MongoDB and I was looking for an opportunity to take a peek at the Go language, so here we are ! Thanks a lot to the Packetbeat team, Tudor Golubenco in particular, for playing the Open Source game so well and for bringing this rough contribution to release level."Thank you Alban for your excellent work and for leading the way to more community contributions!<h2>Mongo’s wire protocol</h2>Let’s do a quick dive in the details of the MongoDB wire protocol, so you get a better understanding of what data Packetbeat captures.It is a fairly friendly protocol to a passive decoder like Packetbeat. Messages and individual fields have lengths, so it is easy to jump over the fields that we don’t understood or we don’t need. There is little contextual state so it is easy to understand what is happening even when capturing only part of the conversion. Also, because a single serialization protocol is used in all messages (<a href="http://bsonspec.org/">BSON</a> - binary JSON), it was easy for us to leverage an existing Go library to decode them. Namely, we use the BSON implementation from the <a href="http://labix.org/mgo">mgo </a>driver, written by Gustavo Niemeyer. One thing to note is that prior to version 2.6, the write operations didn’t actually require a response from the server. By default, however, the drivers called the GetLastError method to acknowledge that the write was successful.&nbsp;<img src="https://api.contentstack.io/v2/assets/575e4c8fd8edd48f76935e07/download?uid=bltc7e6aa0b9b7cb888?uid=bltc7e6aa0b9b7cb888" data-sys-asset-uid="bltc0cf17784a88a8cc" style="max-width: 100%;"/>This was <a href="https://www.mongodb.com/docs/legacy/?site=docs" target="_self">changed in version 2.6</a>, which introduced a new wire protocol for insert, update and delete commands so that a response is always sent from the server and it doesn’t require a second command. Currently Packetbeat only supports the newer version of the protocol, so write commands from versions prior to 2.6 are not captured. We (or you!) can implement this in the future if there is demand.<img src="https://api.contentstack.io/v2/assets/575e4c8e12798d8e381fd6a9/download?uid=blt3aa2e7c3b65a7a11?uid=blt3aa2e7c3b65a7a11" data-sys-asset-uid="blt119ef9b785d07514" style="float: right;margin: 0px;text-align: right;"/>Another interesting aspect is that in a MongoDB cluster the same wire protocol is used between the three server types: mongos, config server and data shards servers. The same protocol is also used for replication between shard nodes. This means you can use Packetbeat for advanced troubleshooting of what is going on inside the cluster!To capture the traffic inside the cluster, you need to install Packetbeat not only on the routing nodes but also on the shard nodes. You likely also need to add port 27019 to the list of MongoDB ports, because this is used by the configuration nodes.<h2>Kibana dashboards</h2>As usual with the protocols we support, we also provide a sample Kibana <a href="http://demo.elastic.co/packetbeat/#/dashboard/MongoDB-performance" target="_blank">dashboard</a> to highlight some of the visualizations you can build based on the data Packetbeat collects and to give you a good starting point for creating your own dashboard. Let’s see some of the visualizations made possible by this data.<h3>MongoDB errors</h3>Packetbeat inspects automatically the response payload to check if it contains error. If it finds any, it sets the status field &nbsp;to error Error. This example shows the number of MongoDB errors evolution sliced by both the collection name and operation type.<img src="https://api.contentstack.io/v2/assets/575e4c8f9e7a83165490c1ad/download?uid=bltc9525251c2a385a3?uid=bltc9525251c2a385a3" data-sys-asset-uid="blt87269016986f5928" style="max-width: 100%;"/><h3>Total time spent per collection</h3>This example shows the total amount of time spent while querying each collection. This view adds to, for example, a 99th percentile visualization of response times because a high total count could be reached if there are slow queries but also if there are many small queries. <img src="https://api.contentstack.io/v2/assets/575e4c8f3dc542cb38c082c8/download?uid=blt5c6b1c3e14b78a0f?uid=blt5c6b1c3e14b78a0f" data-sys-asset-uid="blte488011987bacbf7" style="max-width: 100%;"/><h3>Response times and count per collection</h3>This one plots the 99th response time percentile on the Y axis and the dot size represents the number of queries in the particular time bucket. Different colors represent different collections. Just another way of visualizing performance data which can be helpful in spotting trends. In this screenshot, for example, you can notice the cyclicality of the response times.<img src="https://api.contentstack.io/v2/assets/575e4c8fd8edd48f76935deb/download?uid=blte0f1674c1eb8f622?uid=blte0f1674c1eb8f622" data-sys-asset-uid="bltc3a043c0784f87ae" style="max-width: 100%;"/><h3>Number of writes with unacknowledged write concern</h3>By inspecting the request payload and looking for the writeConcern parameter of write operations, we can plot the number of them that have an unacknowledged write concern. As always with Packetbeat and Kibana, you can drill down from this graph and find these exact transactions and who creates them. <img src="https://api.contentstack.io/v2/assets/575e4c8f0342dfd738263158/download?uid=blt008d9a2e43355a76?uid=blt008d9a2e43355a76" data-sys-asset-uid="blt78196b2628f62bd3" style="max-width: 100%;"/><h3 dir="ltr">Top slowest MongoDB queries</h3>Let’s end with a fairly obvious but very useful one: top slowest MongoDB queries by their 99th percentile.<img src="https://api.contentstack.io/v2/assets/575e4c8f43e9adc538713b6c/download?uid=blt4613a8b34c7b13d3?uid=blt4613a8b34c7b13d3" data-sys-asset-uid="blt93d87fc0dee5c910" style="max-width: 100%;"/><h3>Getting started with Packetbeat for MongoDB</h3>If you didn’t try Packetbeat before, we recommend reading and following the <a href="https://www.elastic.co/guide/en/beats/packetbeat/current/packetbeat-getting-started.html">getting started</a> guide. By default, Packetbeat doesn’t index payload information to avoid capturing sensitive data by mistake. But you can easily enable payload indexing from the configuration file. Here is an example configuration section showing the main options:<pre class="prettyprint">protocols: mongodb: ports: [27017, 27019] send_request: true # index the request payload send_response: true # index the response payload max_docs: 10 # maximum number of documents to index per request/response max_doc_length: 1024 # maximum document size to index </pre>Image credit: <a href="https://www.flickr.com/photos/kumaravel/8282786757">green leaf</a>

leaf_large.jpg

Monitoring MongoDB with Packetbeat and Elasticsearch

race_finish.jpg

race_finish_small.jpg

Benchmarking REST client and transport client

It’s been 4 months since <a href="https://github.com/elasticsearch/elasticsearch-net/releases/tag/0.12.0.0">NEST 0.12.0</a> was released and a lot has happened since then. Elasticsearch released 1.0 and even version 1.1. With this release announcement, I’m pleased to say NEST has finally caught up and brings (almost) all the awesome that Elasticsearch 1.0 and 1.1 bring to the .NET world. Even more exciting – as of today, Elasticsearch.Net and NEST are both officially supported .Net clients for Elasticsearch!(For those just learning about Elasticsearch.Net and NEST, Elasticsearch.Net is the low level client and NEST is the high level client for users of .Net and Elasticsearch.)Before diving in all the technical details of the NEST 1.0 beta release, I would first like to express a sincere thank you to all of you who’ve reached out on Twitter, GitHub issues and via email. Your kind words made all the difference these past 4 months as I busily switched jobs, refactored the client and implemented 1.0/1.1 features. I am happy to say that I have formally joined Elasticsearch Inc and will now be fully dedicated to making Elasticsearch in the .NET world awesome!So, what does ‘beta’ mean? All tests are passing, but we are still waiting for at least one new feature to land. Additionally, beta means that some new features may change before general release. Most importantly, this beta period is intended to solicit as much feedback as possible from everyone on breaking changes/oversights/bugs.<h2>Breaking Changes</h2>With this 1.0 release, we will have many breaking changes. The 0.* NEST releases have been around since 2010 and the internals were showing their age. This 1.0 release represents an almost complete refactoring of NEST.A separate section that lists the breaking changes <a href="http://nest.azurewebsites.net/breaking-changes.html">can be found here</a>. If you find any breaking changes that are not documented, we’d love to hear about them in <a href="http://www.github.com/elasticsearch/elasticsearch-net/issues">elasticsearch-net’s github issues</a>.<h2>Not one but two clients</h2>NEST 0.12.0 introduced a separate, completely generated, client interface called IRawElasticClient which allowed you to build your own requests and responses without having to worry about building endpoints. Much of the work for this 1.0 beta release has been around refactoring this functionality out of the NEST assembly and making NEST’s strongly typed IElasticClient use IRawElasticClient internally.<h3>Elasticsearch.Net</h3>Elasticsearch.Net’s client is now called IElasticsearchClient and is almost completely generated <a href="https://github.com/elasticsearch/elasticsearch/tree/v1.1.0/rest-api-spec">from the client API specification</a>, exposing all the Elasticsearch 1.1 endpoints. It brings a client that’s well aligned in spirit and architecture with the other official Elasticsearch client libraries.Elasticsearch.Net leaves all the request and response mapping up to you although it comes with support for mapping responses into a dynamic container (a slightly modified DynamicDictionary from <a href="http://www.nancyfx.org">NancyFx</a>) out of the box.Elasticsearch.Net is very much intended to be used by other high level clients.Another big new feature is built-in <a href="http://nest.azurewebsites.net/elasticsearch-net/cluster-failover.html">cluster failover/connection pooling support</a> support. This allows the client to retry failed connections on different nodes and sniff the cluster for live nodes when a node fails or at certain configurable intervals.Elasticsearch is elastic, and the client should be, too.Elasticsearch 1.0.0-beta is a prerelease package so make sure you adjust your filter inside the package manager: <img src="https://api.contentstack.io/v2/assets/575e4c8e9e7a83165490c17c/download?uid=blt3d2d801570e82738?uid=blt3d2d801570e82738" data-sys-asset-uid="bltf13ab67aca83aee6"/>or pass the -PreRelease flag to nuget from the console:<blockquote><pre class="prettyprint noescape">PM&gt; Install-Package Elasticsearch.Net -PreRelease </pre></blockquote>Please read <a href="http://nest.azurewebsites.net/elasticsearch-net/quick-start.html">the new documentation for Elasticsearch.Net</a> to find out more.<h3>NEST</h3>NEST’s client classes have completely been rewritten and internally use and still expose the Elasticsearch.Net client. It benefits from it’s cluster failover support and it no longer has complicated internal path builders.NEST is highly opinionated on how you should build and consume Elasticsearch responses, and comes with a strongly typed Query DSL. NEST has really high coverage of mapped endpoints, and the unmapped API’s are all identified and documented on the <a href="https://github.com/elasticsearch/elasticsearch-net/issues?direction=desc&milestone=5&page=1&sort=updated&state=open">roadmap</a>.<blockquote><pre class="prettyprint noescape">PM&gt; Install-Package NEST -PreRelease </pre></blockquote><h2>Changelog</h2>This release consists of 300+ commits over a 4 month timespan so I will just list the highlights:<h3>Aggregations</h3>All of Elasticsearch’s (1.0 and 1.1!) aggregations have been mapped in NEST 1.0.<blockquote><pre class="prettyprint noescape">var results = this._client.Search&lt;ElasticsearchProject&gt;(s=&gt;s .Size(0) .Aggregations(a=&gt;a .Nested("contributors", n=&gt;n .Path(p=&gt;p.Contributors) .Aggregations(t=&gt;t .Average("avg_age", m=&gt;m .Field(p=&gt;p.Contributors.First().Age) ) ) ) ) ); var bucket = results.Aggs.Nested("contributors"); var averageAge = bucket.Average("avg_age"); </pre></blockquote><h3>Snapshot and Restore</h3>You can now script your create/delete/get repositories/snapshotting/restoring with NEST.<h3>New query/filter types</h3>The <a href="https://www.elastic.co/guide/en/elasticsearch/reference/2.4/query-dsl-common-terms-query.html" target="_self">common terms query</a> and simple_query_string have all been mapped, along with other new query and filter types.<h3>Lots of new API’s are mapped</h3>Count percolations, clearing scrolls, external versioning, suggests and more are now all mapped in NEST. The goal is to get as close to a 100% API coverage as possible before the final 1.0 release goes out. Just to reiterate: Elasticsearch.Net, the low level client, already has 100% API coverage but building and handling the responses is out of its scope.<h3>Conditionless() query construct</h3>NEST comes with a powerful feature called conditionless queries which greatly simplifies writing queries with optional parts.Consider this example:<blockquote><pre class="prettyprint noescape">.Query(q=&gt;q.Term("this_term_is_conditionless", "")) </pre></blockquote>NEST will remove the term query and default to not sending a query at all in the body (equivalent to match_all)But what if you want a different query to be run in instead of match_all? This is where the Conditionless() construct comes into play:<blockquote><pre class="prettyprint noescape">.Query(q=&gt;q .Conditionless(qs=&gt;qs .Query(qcq=&gt;qcq.Term("this_term_is_conditionless", "")) .Fallback(qcf=&gt;qcf.Term("name", "do_me_instead") ) ) </pre></blockquote>Now we’ll fallback to a term query on name instead. Note that fallbacks themselves can be conditionless and Conditionless() constructs can be nested inside fallbacks as well.Also if you want to disable conditionless queries you can specify .Strict() on individual parts in the query or globally on the search descriptor.<blockquote><pre class="prettyprint noescape">.Query(q=&gt;q.Strict().Term("this_term_is_conditionless", "")) </pre></blockquote>The previous example will throw an exception because the term misses a value. To enable conditionless only for certain parts in your query, you can also do:<blockquote><pre class="prettyprint noescape">.Strict() .Query(q=&gt; q.Term("must", "exist") &amp;&amp; q.Strict(false).Term("this_term_is_conditionless", "") ) </pre></blockquote>The previous example will just send a term query on the field must, and NEST is smart enough to infer the bool is no longer needed.Finally, if you really intend to send a conditionless query to Elasticsearch you can use .Verbatim()<blockquote><pre class="prettyprint noescape">.Query(q=&gt;q.Verbatim().Term("this_term_is_conditionless", "")) </pre></blockquote>The Strict() and Verbatim() constructs are not new but worth repeating in this context.<h2>Community</h2>As with every release the level of community engagement never ceases to amaze me. Whoever proclaimed open source in .Net is dead?A huge thank you goes out to the folks who submitted the following pull requests.<ul><li><a href="https://github.com/elasticsearch/elasticsearch-net/pull/412">#412</a> Add random_score support for FunctionScore(). ty <a href="http://github.com/vovikdrg">@vovikdrg</a>!</li><li><a href="https://github.com/elasticsearch/elasticsearch-net/pull/419">#419</a> Add script score support for FunctionScore(). ty <a href="http://github.com/V1tOr">@V1tOr</a>!</li><li><a href="https://github.com/elasticsearch/elasticsearch-net/pull/419">#419</a> Added support for index status API. ty <a href="http://github.com/richclement">@richclement</a>!</li><li><a href="https://github.com/elasticsearch/elasticsearch-net/pull/430">#430</a> Allow any object to be used for a partial update. ty <a href="http://github.com/Plasma">@Plasma</a>!</li><li><a href="https://github.com/elasticsearch/elasticsearch-net/pull/431">#431</a> Add Type as constructor parameter to CustomAnalyzer. ty <a href="http://github.com/sschlesier">@sschlesier</a>!</li><li><a href="https://github.com/elasticsearch/elasticsearch-net/pull/440">#440</a> Support dictionary key evaluation in expressions. ty <a href="http://github.com/azubanov">@azubanov</a>!</li><li><a href="https://github.com/elasticsearch/elasticsearch-net/pull/444">#444</a> Renamed QueryString() to Query() MatchQueryDescriptor. ty <a href="http://github.com/gmarz">@gmarz</a>!</li><li><a href="https://github.com/elasticsearch/elasticsearch-net/pull/445">#445</a> Renamed QueryString() to Query() MultiMatchQueryDescriptor. ty <a href="http://github.com/gmarz">@gmarz</a>!</li><li><a href="https://github.com/elasticsearch/elasticsearch-net/pull/446">#446</a> Renamed QueryString() to Query() TextQueryDescriptor. ty <a href="http://github.com/gmarz">@gmarz</a>!</li><li><a href="https://github.com/elasticsearch/elasticsearch-net/pull/447">#447</a> Added Params() support to script on RangeFacetDescriptor. ty <a href="http://github.com/angielamb">@angielamb</a>!</li><li><a href="https://github.com/elasticsearch/elasticsearch-net/pull/448">#448</a> Added filter option in function_score queries. ty <a href="http://github.com/gmarz">@gmarz</a>!</li><li><a href="https://github.com/elasticsearch/elasticsearch-net/pull/479">#479</a> Added detail response mapped to _explain. ty <a href="http://github.com/andreabalducci">@andreabalducci</a>!</li><li><a href="https://github.com/elasticsearch/elasticsearch-net/pull/488">#488</a> Added nested field attribute mapped back in. ty <a href="http://github.com/lukapor">@lukapor</a>!</li><li><a href="https://github.com/elasticsearch/elasticsearch-net/pull/490">#490</a> Added size parameter to CompletionSuggestDescriptor. ty <a href="http://github.com/bgiromini">@bgiromini</a>!</li><li><a href="https://github.com/elasticsearch/elasticsearch-net/pull/540">#540</a> Added missing sort options. ty <a href="http://github.com/Grastveit">@Grastveit</a>!</li><li><a href="https://github.com/elasticsearch/elasticsearch-net/pull/536">#536</a> Worlds smallest pull request (typo). ty <a href="http://github.com/ChrisMcKee">@ChrisMcKee</a>!</li><li><a href="https://github.com/elasticsearch/elasticsearch-net/pull/541">#541</a> FunctionScoreQueryDescriptor boost mode. ty <a href="http://github.com/azubanov">@azubanov</a>!</li><li><a href="https://github.com/elasticsearch/elasticsearch-net/pull/542">#542</a> Allow fields boosting when using expressions in multimatch query. ty <a href="http://github.com/azubanov">@azubanov</a>!</li><li><a href="https://github.com/elasticsearch/elasticsearch-net/pull/545">#545</a> Show requests/response as json string in debug mode when available. ty <a href="http://github.com/azubanov">@azubanov</a>!</li><li><a href="https://github.com/elasticsearch/elasticsearch-net/pull/548">#548</a> Create cache directory for code generation if it does not exist. ty <a href="http://github.com/gmarz">@gmarz</a>!</li><li><a href="https://github.com/elasticsearch/elasticsearch-net/pull/550">#550</a> Fixed typos, formatting, verbiage, naming conventions in the documentation. ty <a href="http://github.com/paigecook">@paigecook</a>!</li><li><a href="https://github.com/elasticsearch/elasticsearch-net/pull/563">#563</a> Added support for new text query types on the multi_match query. ty <a href="http://github.com/azubanov">@azubanov</a>!</li><li><a href="https://github.com/elasticsearch/elasticsearch-net/pull/565">#565</a> Thrift connection does not clean up properly after going into faulted state. ty <a href="http://github.com/danp60">@danp60</a>!</li></ul>In addition a huge thank you goes out to all the folks who gave continuous feedback on the state of master while it underwent major refactoring.A special shout out to <a href="http://github.com/synhershko">@synhershko</a> and <a href="http://github.com/icanhasjonas">@icanhasjonas</a> for opening <a href="https://github.com/elasticsearch/elasticsearch-net/pull/523">#522</a> and <a href="https://github.com/elasticsearch/elasticsearch-net/pull/537">#537</a>Seeing <a href="https://twitter.com/luigiberrettini/status/448413008604438528">tweets</a> like <a href="https://twitter.com/tonariman/status/451354457133969408">this</a> is awesome, thank you <a href="https://twitter.com/tonariman">tonariman</a> &amp; <a href="https://twitter.com/luigiberrettini">luigiberrettini</a>!<h2>Moving forward</h2>The goal is to come to a final Elasticsearch.Net and NEST 1.0 release as soon as possible and with your help we can!Please don’t forget to post your issues and feedback in <a href="http://www.github.com/elasticsearch/elasticsearch-net/issues">elasticsearch-net’s GitHub issues</a>!

Introducing Elasticsearch.Net and NEST 1.0.0-beta1

Ever-increasing quantities and varieties of data are being created and captured. Application, system and user behavior logs, network packets, sensor metrics... you name it. The availability of these data drives a diverse and vast set of use cases ranging from intrusion detection to user behavior analysis, network monitoring, machine monitoring, and remote sensor monitoring, just to name a few.<a href="/products/beats">Beats</a> are open source, purpose-built, lightweight, and efficient agents that acquire and feed data natively to Elasticsearch. Optionally, Beats can feed data to Logstash for further refinement before being forwarded to Elasticsearch.As part of the Elastic Stack, Beats is a <a href="https://en.wikipedia.org/wiki/Telemetry" target="_blank">telemetry</a> architecture, enabling the capture and transmission of measurements and other data from remote sources to Elasticsearch for analysis, aggregation, and search. Anywhere there is change, such as production and consumption, growth and demise, acceleration and deceleration, data has significance. GPS-driven tractors capture soil depths, soil fertility, and ground temperatures, all of which influence agricultural strategy. Ocean buoys record wave amplitudes that may help to warn of impending catastrophes or inform of welcome beach conditions. Automobiles capture thousands of metrics ranging from tire pressure to average speed to fuel consumption that facilitate an understanding of a vehicle's abilities and failures, as well as the owner's driving behaviors. Network sensors capture machine communication messages that help with understanding usage patterns that drive security and infrastructure policies.In the previous article <a href="/blog/running-elasticsearch-on-aws">Running Elasticsearch on AWS</a> we configured a three-node Elasticsearch cluster on AWS EC2. It was fairly straightforward and didn't take much time or effort. We will expand on the <a href="/blog/running-elasticsearch-on-aws">Running Elasticsearch on AWS</a> example and configure Beats to feed the Elasticsearch cluster running on AWS EC2 instances. This will demonstrate how simple it is to configure a centralized machine resource monitoring solution.First, we'll provide an overview of Beats, review available Beats and the environments in which they run - Part I. Then we'll do a step-by-step example installing, configuring, and verifying Beats output - Part II.<h2>Part I - Beats Overview</h2><a href="/products/beats">Beats</a> are purpose-built lightweight data shippers, or agents, that run on remote machines and feed Elasticsearch instances. Beats make it easy to get data into Elasticsearch. Beats are available on a number of operating systems such as Debian, Redhat, Linux and Mac.There are several available <a href="/products/beats">Beats</a> including Filebeat, Metricbeat, Packetbeat, Winlogbeat and Topbeat. Each Beat has a specific purpose or multiple purposes that are logically related, allowing each Beat to focus on its specific task and do it well.<a href="/products/beats/filebeat">Filebeat</a> tails logs and can ship data to Logstash for further refinement, or directly to Elasticsearch for analysis and search. Filebeat can be installed on any machine that has applications that generate log data, such as a database or application server.<a href="/downloads/beats/metricbeat">Metricbeat</a> (Alpha) captures operating system metrics as services such as Apache web server and Redis. Metricbeat does everything Topbeat does and much more in that it captures operating system metrics such as per-process CPU, memory, and storage use as well as common application messages, such as web servers. Metricbeat will replace Topbeat, so you might consider using Metricbeat for development.<a href="/products/beats/packetbeat">Packetbeat</a> captures web, database, and other network protocols, enabling Kibana real-time analytics. Packetbeat is extensible, enabling the addition of new protocols, metrics and analytics.<a href="/downloads/beats/winlogbeat">Winlogbeat</a> captures Windows event log system, application and security data, enabling monitoring of Windows machines.<a href="/products/beats/topbeat">Topbeat</a> captures per-process memory, CPU and disk usage statistics. Topbeat can be installed across a set of machines and indexed into a central Elasticsearch cluster, enabling centralized machine resource monitoring. Topbeat is being replaced by Metricbeat, currently in Alpha, and the change is discussed in this <a href="/blog/beats-5-0-0-alpha3-released">article</a>.If you have metric data not addressed by available Beats, there's also <a href="https://www.elastic.co/guide/en/beats/libbeat/current/beats-reference.html" target="_self">Libbeat</a>, a framework for developing new Beats.The <a href="https://www.elastic.co/guide/en/beats/libbeat/current/community-beats.html">Beats Community</a> is growing and developers are contributing everything from pingbeat, for capturing ping roundtrips, to redisbeat, for Redis monitoring, and to hsbeat, for capturing Java HotSpot VM performance counters.<h2>Part II - Configuring Beats</h2>In this example we will install <a href="https://www.elastic.co/products/beats/topbeat">Topbeat</a> and <a href="https://www.elastic.co/guide/en/beats/packetbeat/current/packetbeat-overview.html">Packetbeat</a> on a development machine, in this case a Macbook Pro, and feed an Elasticsearch cluster running on AWS. Beats can be installed on any machine running a supported operating system that has network connectivity with the Elasticsearch cluster.<h3>Configure Topbeat</h3><a href="/downloads/beats/topbeat">Download Topbeat</a> to a directory on the machine that will be running the Beats.Download and extract the Topbeat archive using your preferred method, for example:<pre class="prettyprint">curl -L -O &lt;a href="https://download.elastic.co/beats/topbeat/topbeat-1.2.3-darwin.tgz"&gt;https://download.elastic.co/beats/topbeat/topbeat-...&lt;/a&gt;</pre><pre class="prettyprint">tar xzvf topbeat-1.2.3-darwin.tgz </pre><img src="https://api.contentstack.io/v2/assets/57853a1ffc8134614f684651/download?uid=blt8fbd3baf06626f6d" data-sys-asset-uid="blt8fbd3baf06626f6d" alt="get_topbeat.png"/>This will create a folder named topbeat-1.2.3-darwin that contains topbeat, topbeat.yml and topbeat.template.yml.Locate and open topbeat.yml in your preferred editor. There is a hosts property that tells Topbeat where Elasticsearch is located. Locate the hosts parameter and update it to refer to the AWS public hostname, for example:<img src="https://api.contentstack.io/v2/assets/57853a36b4a7391d0c76deed/download?uid=blt1c456161684b4a17" data-sys-asset-uid="blt1c456161684b4a17" alt="topbeat_hosts.png"/>Next, find the template property, which defines the index template that defines the field mappings required by Topbeat. Topbeat will load the index template file topbeat.template.json to Elasticsearch. Uncomment the template property along with the name and path properties that immediately follow.<img src="https://api.contentstack.io/v2/assets/57853a41fc8134614f684670/download?uid=bltedc473e2c49ece65" data-sys-asset-uid="bltedc473e2c49ece65" alt="topbeat_template.png"/>Then, run Topbeat with the command:<pre class="prettyprint">sudo ./topbeat -e -c topbeat.yml </pre>To verify documents are being indexed, we'll use curl to make a search request:<pre class="prettyprint">curl -XGET 'http://ec2-50-17-114-78.compute-1.amazonaws.com:9200/topbeat-2016.07.07/_search?pretty' </pre><img src="https://api.contentstack.io/v2/assets/57853a3bb4a7391d0c76def4/download?uid=bltaf4915db04f8e94d" data-sys-asset-uid="bltaf4915db04f8e94d" alt="topbeat_request.png"/>The curl request has several components. The first is the AWS EC2 public hostname "ec2-50-17-114-78.compute-1.amazonaws.com". This should be changed to your EC2 instance's public hostname. The second component "topbeat-2016.07.06" is the index. Lastly, is the search request "_search" with an optional "pretty" argument to format the JSON response.If you want to dive deeper into Topbeat, a comprehensive <a href="https://www.elastic.co/guide/en/beats/topbeat/current/_overview.html">Topbeat Reference</a> is available and it covers topics such as <a href="https://www.elastic.co/guide/en/beats/topbeat/current/configuring-howto-topbeat.html">configuration</a> and <a href="https://www.elastic.co/guide/en/beats/topbeat/current/securing-topbeat.html">security</a>.<h3>Configure Packetbeat</h3><a href="/downloads/beats/packetbeat">Download Packetbeat</a> to a directory on the machine that will be running the Beats. Download and extract the Packetbeat archive using your preferred method, for example:<pre class="prettyprint">curl -L -O&lt;a href="https://download.elastic.co/beats/packetbeat/packetbeat-1.2.3-darwin.tgz"&gt;https://download.elastic.co/beats/packetbeat/packe...&lt;/a&gt;</pre><pre class="prettyprint">tar xzvf packetbeat-1.2.3-darwin.tgz </pre><img src="https://api.contentstack.io/v2/assets/57853a18fc8134614f68464b/download?uid=blt4cd347d705e292c7" data-sys-asset-uid="blt4cd347d705e292c7" alt="get_packetbeat.png"/>This will create a folder named packetbeat-1.2.3-darwin that contains three files: packetbeat, packetbeat.yml, and packetbeat.template.yml.Locate and open packetbeat.yml in your preferred editor. There is a hosts property that tells Packetbeat where Elasticsearch is located. Locate the hosts parameter and update it to refer to the AWS public hostname, for example:<img src="https://api.contentstack.io/v2/assets/57853a26fc8134614f684657/download?uid=blt738187754a29771b" data-sys-asset-uid="blt738187754a29771b" alt="packetbeat_hosts.png"/>Next, find the template property, which defines the index template that defines the field mappings required by Topbeat. Topbeat will load the index template file topbeat.template.json to Elasticsearch. Uncomment the template property along with the name and path properties that immediately follow.<img src="https://api.contentstack.io/v2/assets/57853a2cfc8134614f68465e/download?uid=bltbabfd10ee4a17877" data-sys-asset-uid="bltbabfd10ee4a17877" alt="packetbeat_template.png"/><pre class="prettyprint">sudo ./packetbeat -e -c packetbeat.yml </pre>To verify documents are being indexed, we'll use curl to make a search request:<pre class="prettyprint">curl -XGET 'http://ec2-50-17-114-78.compute-1.amazonaws.com:9200/packetbeat-2016.07.07/_search?pretty' </pre><img src="https://api.contentstack.io/v2/assets/57855336b4a7391d0c76e703/download?uid=blt6665d171b20e6547" data-sys-asset-uid="blt6665d171b20e6547" alt="packetbeat_request.png"/>If you want to dive deeper into Packetbeat, a comprehensive <a href="https://www.elastic.co/guide/en/beats/packetbeat/current/packetbeat-overview.html">Packetbeat Reference</a> is available and it covers topics such as <a href="https://www.elastic.co/guide/en/beats/packetbeat/current/configuring-howto-packetbeat.html">configuration</a> and <a href="https://www.elastic.co/guide/en/beats/packetbeat/current/securing-packetbeat.html">security</a>.<h2>Part III - Summary</h2>Installing, configuring and running Beats is very easy. Beats address a broad set of use cases. They eliminate the challenge of acquiring data and feeding Elasticsearch.&nbsp;Beats can feed any network accessible Elasticsearch cluster, whether it resides on Amazon AWS EC2, Microsoft Azure, dedicated iron and even <a href="https://www.elastic.co/cloud">Elastic Cloud</a>.In the next installment of the AWS blog series, we'll cover installing Kibana on AWS EC2 and visualize the Beats indexed in this example.

aws-beats-elasticsearch.png

aws-beats-elasticsearch-720x420.png

Using Beats with Amazon AWS

security-camera-arcsight-elasticserach-full-bleed.jpg

security_camera_full bleed.jpg

Integrating Elasticsearch with ArcSight SIEM - Part 5

<blockquote>When you help people build applications with Elasticsearch every day, you run into a lot of unique requirements and scenarios. All of us here at Elasticsearch have an ever-growing “bag of tricks”, and it only seems fair to share those tricks with you. We hope you find them interesting, and perhaps useful to your application.</blockquote><hr style="width:100%">A while ago on Twitter, someone was asking if aggregations could be used to categorize data based on irregular product codes. They were combining data from several legacy systems, so the product codes were erratic and not internally consistent.There were two classes of product codes:<ul><li>AB123: Two letters followed by three digits</li><li>A99999: One letter followed by five digits</li></ul>The goal was to determine how many of each type existed, and how many were missing certain meta-data tags. Many people are stumped by situations like this, since it is initially unclear how to categorize two groups of product codes that are only related to each other by the structural format of their ID.<h3>Option 1: Pre-parse with Grok and Logstash</h3>The best solution is to deal with inconsistencies like this at the input level. For example, you can build a <a href="http://logstash.net/docs/1.4.2/filters/grok">Grok filter in Logstash</a> to identify the two different patterns and tag them appropriately. Once tagged, it is trivial to do aggregations on the structured document data. If a new pattern occurs in your data that doesn’t match the pattern, Logstash will emit a _grokparsefailure tag.This has the benefit of improving search results, since only properly tagged documents will be visible to search. You can go back and fix/reindex the “broken” data at your lesiure and iteratively improve your search results through better-tagged data.<h3>Option 2: Regex to the rescue</h3>But things are not always this simple. What if you just indexed 10TB of data and realized you forgot to include appropriate pre-parsing? Re-indexing would be a pain, or potentially impossible. We need a solution that operates on your existing data.The key is to use a regular expression and filter buckets. A <a href="/guide/en/elasticsearch/reference/current/search-aggregations-bucket-filter-aggregation.html">filter bucket</a> will hold all documents matching its filtering criteria. If we place a <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-regexp-query.html" target="_self">regexp filter</a> inside the bucket, we can find all product IDs matching a certain pattern.Once documents have been sorted into one of the filter buckets, we can apply other bucketing and metrics to derive statistics.Let’s take a look at a very simple example. First we index some data with mixed product codes:<pre class="prettyprint lang-js">PUT /test/data/_bulk {"index":{}} {"product_code" : "AB123"} {"index":{}} {"product_code" : "XY345"} {"index":{}} {"product_code" : "AZ987"} {"index":{}} {"product_code" : "ZZ192"} {"index":{}} {"product_code" : "A99999"} {"index":{}} {"product_code" : "A12345"} {"index":{}} {"product_code" : "A98765"} {"index":{}} {"some_other_field" : "xyz"} {"index":{}} {"some_other_field" : "123"} </pre>Then we can run a very simple aggregation to sort out the various codes:<pre class="prettyprint lang-js">GET /test/data/_search?search_type=count { "aggs" :{ "total_count" : { "global" : {} }, "XX999" : { "filter" : { "regexp":{ "product_code" : "[a-z]{2}[0-9]{3}" } } }, "X99999" : { "filter" : { "regexp":{ "product_code" : "[a-z]{1}[0-9]{5}" } } }, "no_format" : { "missing" : { "field" : "product_code" } } } } </pre>This query will give a document count for each product code matching the two regex patterns. The missing bucket, which will get <a href="https://github.com/elasticsearch/elasticsearch/issues/5659">a nice performance boost in Elasticsearch v1.3.0</a>, will show all documents that don’t have any product code at all. From this base, it is easy to add extra metrics, such as the average price of each product, or the average number sold each day for the last month, etc.<h3>The takeaway tip</h3>The key to this tip is the filter bucket. This bucket accepts any Elasticsearch filter, which means you can construct the same kind of complex filtering operations which you already use in search requests. And because these are filters, they enjoy all the performance benefits inherent to filters.Starting in Elasticsearch version 1.3.0, you will also have access to the <a href="https://github.com/elasticsearch/elasticsearch/issues/6118">filters bucket</a> (note the plural). Although functionally equivalent to using multiple filter buckets, the filters simplifies the syntax for applying multiple filters at the same time.So next time you need to aggregate some statistics, but are stumped by the irregular or inconsistent nature of your data, remember the filter bucket (and the upcoming filters bucket).

Quick Tips: regex filter buckets

<div class="main">Building a highly available product requires some creative thinking. High availability is measured in two aspects when talking about distributed products. The first is the ability to survive partial cluster failure gracefully (and scale to new nodes when added), and the second is handling complete cluster failure or shutdown (and bringing it back up again).The following covers how a distributed system can handle these two problems, especially in cloud environments, and what ElasticSearch does in such cases.<h2>Partial Cluster Failure</h2>Partial cluster failure means that one or more nodes failed within the cluster. Support in a distributed system for partial cluster failure means (surprise surprise) not losing data when such events happen. It gets even more important when working in the cloud, since the control one has over the servers (for good or bad) is out of our hands. Moreover, with cloud providers innovating with cool features like Amazon’s <a href="http://aws.amazon.com/ec2/spot-instances/">spot instances</a> (the ability to bid for instances), we cannot control when our nodes operate. They basically come and go as they wish.This is a question of state, and where the state is stored. I will cover the case when the state is stored locally to the node, which means that if you lose your node, you lose the data stored with the node. It is important to understand that almost any system will work much faster with local state (local file system, â€œlocalâ€ memory) than with a shared storage solution (like NFS). Also, when more than one node needs to access shared state (like NFS), usually locking needs to take place, which hinders the scalability of the system.The most common solution to partial cluster failure is to replicate data between nodes. The ability to control the number of replicas in the cluster allows one to provide a higher degree of availability (this also relates to total cluster failure, see below). The idea of replication is to have one node that perform the operation on itself, and then to replicate the data or operation to its replicas.In the case of ElasticSearch, an index is broken down into shards, and each shard can have zero or more replicas. Both are configurable when creating an index. Here is an example of creating an index with 3 shards, each with 2 replicas (note, this is configurable on a per index basis):<pre class="dontquote prettyprint">&lt;span class="pln"&gt;$ curl &lt;/span&gt;&lt;span class="pun"&gt;-&lt;/span&gt;&lt;span class="pln"&gt;XPUT http&lt;/span&gt;&lt;span class="pun"&gt;:&lt;/span&gt;&lt;span class="com"&gt;//localhost:9200/twitter/ -d '&lt;/span&gt;&lt;span class="pln"&gt; index &lt;/span&gt;&lt;span class="pun"&gt;:&lt;/span&gt;&lt;span class="pln"&gt; number_of_shards &lt;/span&gt;&lt;span class="pun"&gt;:&lt;/span&gt;&lt;span class="pln"&gt; &lt;/span&gt;&lt;span class="lit"&gt;3&lt;/span&gt;&lt;span class="pln"&gt; number_of_replicas &lt;/span&gt;&lt;span class="pun"&gt;:&lt;/span&gt;&lt;span class="pln"&gt; &lt;/span&gt;&lt;span class="lit"&gt;2&lt;/span&gt;&lt;span class="pln"&gt; &lt;/span&gt;&lt;span class="str"&gt;'&lt;/span&gt;</pre>When an operation is presented to a node, it is routed automatically to the primary shard within its replication group, which performs the operation and then replicates it to all its backups. Replication to the backup is done in parallel using non-blocking IO, which basically means that more replicas mainly just means more traffic on the network.Replicas accept read operations as well, meaning that the more replicas you have, the more highly available your index is, but also the more scalable it is in terms of search and get operations!Once we work under the assumption that node level state is volatile, we can store the index itself in a fast medium, perhaps a local file system (even SSD), JVM heap memory, or even non-JVM (or native) memory.The choice of where to store an index for ElasticSearch is configurable on the index level. Here is an example that indicates that the â€œtwitterâ€ index be stored in main memory:<pre class="dontquote prettyprint">&lt;span class="pln"&gt;$ curl &lt;/span&gt;&lt;span class="pun"&gt;-&lt;/span&gt;&lt;span class="pln"&gt;XPUT http&lt;/span&gt;&lt;span class="pun"&gt;:&lt;/span&gt;&lt;span class="com"&gt;//localhost:9200/twitter/ -d '&lt;/span&gt;&lt;span class="pln"&gt; index &lt;/span&gt;&lt;span class="pun"&gt;:&lt;/span&gt;&lt;span class="pln"&gt; store&lt;/span&gt;&lt;span class="pun"&gt;:&lt;/span&gt;&lt;span class="pln"&gt; type &lt;/span&gt;&lt;span class="pun"&gt;:&lt;/span&gt;&lt;span class="pln"&gt; memory &lt;/span&gt;&lt;span class="str"&gt;'&lt;/span&gt;</pre>But if node storage is considered transient, what do we do regarding long term persistency of the index? The answer is the next section.<h2>Full Cluster Failure / Long Term Persistency</h2>Handling full cluster failure or shutdown means that both the state of the cluster (i.e. which indices exist, what are the their settings, what are their mapping definitions, etc.) and the content of each index must be stored persistently elsewhere.A solution for this is similar to <a href="http://www.apple.com/macosx/what-is-macosx/time-machine.html">Time Machine</a>, we need to persist the state into a long term storage. This solution is similar to what data grids have called <a href="http://www.infoq.com/articles/write-behind-caching">Write Behind</a>.The idea is that short term high availability is maintained by replication, while long term persistency is done by asynchronously writing deltas of the state into long term (persistent) storage. The benefit is that real time operations are not affected by this process.So, what happens when we restart a cluster? When it first starts up, it queries the long term storage for the cluster state (indices created, settings, mappings, etc.) and established them locally (creating indices, creating mapping). Each time a shard is first instantiated within its shard replication group, it will also recover its state from the long term persistency.ElasticSearch provides this capability with a module called <a href="/guide/reference/modules/gateway/">Gateway</a>. For example, to have the state stored in a shared file system, each node needs to start with a configuration of:<pre class="dontquote prettyprint">&lt;span class="pln"&gt;gateway &lt;/span&gt;&lt;span class="pun"&gt;:&lt;/span&gt;&lt;span class="pln"&gt; type &lt;/span&gt;&lt;span class="pun"&gt;:&lt;/span&gt;&lt;span class="pln"&gt; fs fs &lt;/span&gt;&lt;span class="pun"&gt;:&lt;/span&gt;&lt;span class="pln"&gt; location &lt;/span&gt;&lt;span class="pun"&gt;:&lt;/span&gt;&lt;span class="pln"&gt; &lt;/span&gt;&lt;span class="str"&gt;/shared/&lt;/span&gt;&lt;span class="pln"&gt;fs&lt;/span&gt;&lt;span class="pun"&gt;/&lt;/span&gt;</pre>This can also be started right from the command line using (only on unix):<pre class="dontquote prettyprint">&lt;span class="pln"&gt;$ bin&lt;/span&gt;&lt;span class="pun"&gt;/&lt;/span&gt;&lt;span class="pln"&gt;elasticsearch &lt;/span&gt;&lt;span class="pun"&gt;-&lt;/span&gt;&lt;span class="pln"&gt;f &lt;/span&gt;&lt;span class="pun"&gt;-&lt;/span&gt;&lt;span class="typ"&gt;Des&lt;/span&gt;&lt;span class="pun"&gt;.&lt;/span&gt;&lt;span class="pln"&gt;gateway&lt;/span&gt;&lt;span class="pun"&gt;.&lt;/span&gt;&lt;span class="pln"&gt;type&lt;/span&gt;&lt;span class="pun"&gt;=&lt;/span&gt;&lt;span class="pln"&gt;fs &lt;/span&gt;&lt;span class="pun"&gt;-&lt;/span&gt;&lt;span class="typ"&gt;Des&lt;/span&gt;&lt;span class="pun"&gt;.&lt;/span&gt;&lt;span class="pln"&gt;gateway&lt;/span&gt;&lt;span class="pun"&gt;.&lt;/span&gt;&lt;span class="pln"&gt;fs&lt;/span&gt;&lt;span class="pun"&gt;.&lt;/span&gt;&lt;span class="pln"&gt;location&lt;/span&gt;&lt;span class="pun"&gt;=&lt;/span&gt;&lt;span class="str"&gt;/shared/&lt;/span&gt;&lt;span class="pln"&gt;fs&lt;/span&gt;&lt;span class="pun"&gt;/&lt;/span&gt;</pre>The above means that the cluster state will be stored on a shared file system, and each index created will automatically store its state on a shared file system as well. But, since each index has its own <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-gateway.html" target="_self">index gateway</a>, we can get clever and decide on a per index level if it should be stored using a gateway or not. For example, the following example disables long term storage of an index:<pre class="dontquote prettyprint">&lt;span class="pln"&gt;$ curl &lt;/span&gt;&lt;span class="pun"&gt;-&lt;/span&gt;&lt;span class="pln"&gt;XPUT http&lt;/span&gt;&lt;span class="pun"&gt;:&lt;/span&gt;&lt;span class="com"&gt;//localhost:9200/my_special_volatile_index/ -d '&lt;/span&gt;&lt;span class="pln"&gt; index &lt;/span&gt;&lt;span class="pun"&gt;:&lt;/span&gt;&lt;span class="pln"&gt; gateway&lt;/span&gt;&lt;span class="pun"&gt;:&lt;/span&gt;&lt;span class="pln"&gt; type &lt;/span&gt;&lt;span class="pun"&gt;:&lt;/span&gt;&lt;span class="pln"&gt; none &lt;/span&gt;&lt;span class="str"&gt;'&lt;/span&gt;</pre>As you can see, this is really powerful. More over, this is the perfect solution for cloud environments. In the cloud, storing long term state is usually done using what the cloud provider provides. For example, with Amazon, its <a href="http://aws.amazon.com/ebs/">EBS</a> or <a href="http://aws.amazon.com/s3/">S3</a>. Right out of the (current version) box, you can use Amazon EBS for long term storage in ElasticSearch (since its a mounted shared file system). In the future, there will also be a gateway module to store the state directly on Amazon S3 and others.<h2>Final Words</h2>This is only the tip of the iceberg of how ElasticSearch replication, recovery and gateway works, and the future will just increase the iceberg (right back at you, global warming!). For example, what I am playing with now is with the idea of treating indices state as a version control, being able to tag them, and recover back specific tags into new indices or current ones. I hope that at least in terms of handling cluster failures (both partial and total), ElasticSearch makes more sense now.-shay.banon</div>

Search Engine Time Machine

This is the final post in a three-part series about Instant Aggregations. Read how it all started in <a href="https://www.elastic.co/blog/the-tale-of-caching-and-why-it-matters">The Tale of Caching and Why It Matters</a> from Simon Willnauer and Shay Banon and the meaty middle detailed in <a href="https://www.elastic.co/blog/the-great-query-refactoring-thou-shalt-only-parse-once"> </a><a href="https://www.elastic.co/blog/the-great-query-refactoring-thou-shalt-only-parse-once">The Great Query Refactoring: Thou Shalt Only Parse Once</a>. Enjoy the trilogy! In 1.4.0 Elasticsearch gained a shard level ‘Request Cache’ which caches the result of the query phase on each shard keyed on the search request itself. Until 5.0 this feature was disabled by default since it wasn’t as useful as it could be for most of the computationally heavy use cases. The cache was intended to make searches faster, especially for aggregations. One problem was that ordering in JSON is not deterministic so although two requests may be logically the same, when rendered to JSON strings they may not be equal. The other main problem is that these computationally heavy use cases tend to use time windows relative to the current time so subsequent requests will tend to have slightly different time ranges. Enabling it would likely be a waste memory for most of the users &nbsp;since they would rarely ever get a cache hit. So why would we add such a feature?Here at Elastics, we try to follow the rule of “progress over perfection”. Even if we can’t utilize it’s full potential we get a step closer to our goals. Now after 2 years of engineering effort and <a href="/blog/the-great-query-refactoring-thou-shalt-only-parse-once">refactoring the entire search request</a> representation, we can finally take full advantage of it.From 5.0 the request cache will be enabled by default for all requests with `size:0`. The request cache is most useful for analytics use cases and generally analytics use cases use search requests with `size: 0`.So how did the changes made since 1.4.0 enable us to better use this feature?The search request cache gives massive improvements in search performance when running the same search multiple times on a static index. However, search requests are rarely exactly the same, especially in the time series use case.Lets look at some of the characteristics of typical search for time-series use cases:<ul><li>Queries have a &nbsp;time range component often relative to current time (e.g. from now-7d to now)</li><li>Search requests span &nbsp;across multiple indices</li><li>Indexing happens on the latest index, all other indices are static</li><li>Same search requests are run multiple times with different time ranges</li></ul>The above are common characteristics of search requests for a lot of time series use cases including common patterns of usage for Kibana. A lot of Kibana dashboards contain quite a few visualisations a lot of which are complex and can take a number of seconds for the dashboard request to be executed by Elasticsearch. Because search requests are rarely the same, the search request cache can’t be used effectively to improve search performance for these use cases.Or can it?How can we make the search request cache work when the time range is always moving? Let’s solve this by considering an example. Imagine we have the details for every residential house sale in the UK since 1995. We can index this into 21 yearly indices (e.g. house-prices-1995, house-prices-1996, …, house-prices-2016). For the purposes of this explanation let’s make each index only have 1 shard (though the same idea can easily be extended to multiple shard indices without modification) Now we can run queries against the indices by using requests like this:<pre>``` GET house-prices-*/_search { "query": { "range": { "date": { "gte": "2013-09-01", "lt": "2016-03-01" } } } } ``` </pre>The figure below shows three of these such queries overlaid on the yearly indices. These queries are typical for dashboard style use cases in that they differ only by small amounts (imagine a refreshing dashboard)<img src="https://api.contentstack.io/v2/assets/57d6e0bf39cce73c4a091058/download?uid=blt1a2b92ef4097f1ff" height="325" width="850" style="width: 850px; height: 325px;"/>A couple of things stand out from this diagram:<ul><li>All three queries match no documents in house-prices-2012 (highlighted red on the diagram)</li><li>All three queries will match all documents in house-prices-2014 and house-prices-2015 (highlighted green on the diagram)</li></ul>So the only indices which affect the matching documents between the three queries are house-prices-2013 and house-prices-2016.If we could rewrite the queries on each shard based on the range of values present in that index we could make the queries able to actually utilize the request cache. Below is the diagram with the same three queries rewritten to make them more cachable in the search request cache:<img src="https://api.contentstack.io/v2/assets/57d6e1d039cce73c4a0910ad/download?uid=blte0f8b865a8d465c8" height="325" width="850" style="width: 850px; height: 325px;"/>You can see that in the diagram above, the first query is rewritten and run as a `match_none` query on the `house-prices-2012` indexes shard and a `match_all` query on the `house-prices-2014` and `house-prices-2015` indices shards. This means that when the second and third query is run it can use the cached results of the search on the `house-prices-2012`, `house-prices-2014` and `house-prices-2015` indices shards instead of actually running a search on those shards. So for the second and third queries we only need to actually search the shards on the `house-prices-2013` and `house-prices-2016` indices.Also, even for the first query, running a `match_none` query instead of a range query will be faster since it will not need to try to lookup the date range in the index.So how does this work in practice?This feature not only relies on the search request cache mentioned at the beginning of this post, but it is also heavily based on the query refactoring that we described in a recent <a href="/blog/the-great-query-refactoring-thou-shalt-only-parse-once">blog post</a>. The reason for this is that in order to add the rewrite logic described above we need objects that represent the query and the search request that we can actually rewrite.When the search request is received by the shard, we rewrite the entire request including `query` and `post_filter` QueryBuilder objects. Each type of query has it’s own implementation of QueryBuilder which will rewrite according to it’s own rules.Compound queries, queries that themselves contain queries (such as `bool` and `constant_score`) will call rewrite on the queries they wrap. Most leaf queries, queries that do not contain other queries, currently do not contain any rewrite logic so just return themselves (to indicate they did not change).The range query however, will check the minimum and maximum value of the field (we will call this the field range) and compare this to the range it contains. There are three cases that we care about when evaluating the rewrite for the range:<ul><li>The query range and the field range do not overlap at all - In this case we can rewrite this range as a `match_none` query so we return a `MatchNoneQueryBuilder`</li><li>The query range contains the entire field range - In this case all documents which contain a value for this field will match the range so we can rewrite the range query as an unbounded range query (not a match_all query) and return it</li><li>The query range partially overlaps the field range. There’s not much we can do to help in this case so we don’t rewrite and return the original range query.</li></ul>Now that the search request has been rewritten we can check the cache with the rewritten version and either retrieve the cached result or execute the search and add the result to the cache, keyed on the rewritten search request. This has a nice side effect since the rewritten queries are “normalized” search requests that are semantically equivalent but differ in the order of keys in the json will now also produce cache hits.Why not use `match_all` ?In the case that the query range contains the entire field range you might think we could rewrite the `range` query as a `match_all` query. We can’t do this because we can’t assume that all documents have a value for the field. If we rewrite to a `match_all` query we would incorrectly match documents that have no value for the field. For this reason we instead rewrite the range to an unbounded range query (effectively [* TO *]) which still means the query is much more useful to the search request cache.Is this all worth the trouble?To answer this question we’ll go back to our yearly house price indices. This dataset contains 21 yearly indices containing 21,304,688 residential UK house sales from 1995 to 2016. I ran this fairly unscientific test on my Macbook Pro. Each index has a single shard. I created a Kibana dashboard showing 14 different visualisations relevant to the data mixing simple and complex visualisations. The top of the dashboard looks like the following:<img src="https://api.contentstack.io/v2/assets/57e02db9acc34b3254b8af0b/download?uid=bltf740e6e1aec8ca2b" height="469" width="850" style="width: 850px; height: 469px;"/>With the request cache disabled, requests to elasticsearch for the dashboard to refresh takes 12s-14s. When the request cache is enabled this drops to ~100ms. The same request is now 100x faster!But this is on completely static data so what happens when we are indexing data?To test this I deleted the data from 2010 to 2016 and loaded it in date order whilst refreshing the dashboard:<img src="https://api.contentstack.io/v2/assets/57d6e20f39cce73c4a0910cb/download?uid=blt7d49d71a4510bee5" height="502" width="850" style="width: 850px; height: 502px;"/>Now the request to Elasticsearch for the dashboard refresh takes 60-200ms. So the query is still at least 50x faster than our original un-cached query! The refresh time varies depending on how much data is present in the changing index since the search on the latest index does not hit the cache. This is because as new data is indexed, the cache on the shards of the latest index is constantly being invalidated.The Instant Aggregations feature enables much better caching of search requests by rewriting the query based on the data present on the shard. As of 5.0.0 the query is only rewritten for date range queries, but with this infrastructure in place it opens the door to the potential of rewriting other types of query to improve the request cache utilization. We even have the potential to rewrite aggregations to make them more cacheable! We have shown in this post that the potential performance improvements when we use this technique can be huge, so watch this space for more improvements in the future.

instant-aggregations-elasticsearch-colin-goodheart-smithe-request-cache-full-bleed.jpg

instant-aggregations-elasticsearch-rewriting-queries-thumbnail.jpeg

Instant Aggregations: Rewriting Queries for Fun and Profit

full-bleed-2k.jpg

thumb.jpeg

How to Develop Your Own Security Extensions and Custom Realms for Elasticsearch

<h2>First, some context</h2><a href="https://www.elastic.co/blog/hot-warm-architecture-in-elasticsearch-5-x">Hot-Warm</a> architectures are often used when we want to get the most out of our hardware. It is particularly useful when we have time-based data, like <a href="https://www.elastic.co/blog/observability-with-the-elastic-stack">logs, metrics, and APM data</a>. Most of these setups rely on the fact that this data is read-only (after ingest) and that indices can be time(or size)-based. So they can be easily deleted based on our desired retention period. In this architecture, we categorize Elasticsearch nodes into two types: 'hot' and 'warm'.Hot nodes hold the most recent data and thus handle all indexing load. Since recent data is usually the most frequently queried, these nodes will be the most powerful in our cluster: fast storage, high memory and CPU. But that extra power gets expensive, so it doesn’t make sense to store older data that isn’t queried as often on a hot node.On the other hand, warm nodes will be the ones dedicated to long-term storage in a more cost-efficient way. Data on the warm nodes is not as likely to be queried often and data within the cluster will move from hot to warm nodes based on our planned retention (achieved through <a href="https://www.elastic.co/guide/en/elasticsearch/reference/6.6/shard-allocation-filtering.html">shard allocation filtering</a>), while still being available online for queries.Starting with Elastic Stack 6.3, we've been building new features to enhance hot-warm architectures and simplify working with time-based data.<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blte8ad3502d7fba4c1/5c98d0a5c9ab24634dcd6d86/hot-warm-features-frozen-indices-transp.png" data-sys-asset-uid="blte8ad3502d7fba4c1" alt="hot-warm-features-frozen-indices-transp.png" width="550" style="margin: auto;display: block;"/><a href="https://www.elastic.co/blog/data-rollups-in-elasticsearch-you-know-for-saving-space">Data rollups</a> were first introduced in version 6.3, to save storage. In time-series data, we want fine-grained detail for the most recent data. But it is very unlikely that we will need the same for historical data, where we will typically look at datasets as a whole. And this is where rollups came in, since starting at version 6.5 we can <a href="https://www.elastic.co/blog/how-to-create-manage-and-visualize-elasticsearch-rollup-data-in-kibana">create, manage and visualize rollup data in Kibana</a>.Shortly after, we added <a href="https://www.elastic.co/guide/en/elasticsearch/reference/6.6/modules-snapshots.html#_source_only_repository">source-only snapshots</a>. This minimal snapshots will provide a significant reduction of snapshot's storage, with the tradeoff of having to reindex data if we want to restore and query. This has been available since version 6.5.In version 6.6, we released two powerful features, Index Lifecycle Management (ILM) and Frozen Indices.<a href="https://www.elastic.co/guide/en/elasticsearch/reference/6.6/index-lifecycle-management.html">ILM</a> provides the means to automate your indices management over time. It simplifies moving indices from hot to warm, allows deletion when indices are too old, or automates force merging indices down to one segment.And for the rest of this blog, we’ll talk about frozen indices.<h2>Why freeze an index?</h2>One of the biggest pain points with “old” data is that, regardless of age, indices still have a significant memory footprint. Even if we place them on cold nodes, they still use heap.A possible solution could be to close the index. If we close an index, it won’t require memory, but we will need to re-open it to run a search. Reopening indices will incur an operational cost and also require the heap it was using before being closed.On each node, there is a memory (heap) to storage ratio that will limit the amount of storage available per node. It may vary from as low as 1:8 (memory:data) for memory intensive scenarios, to something close to 1:100 for less demanding memory use cases.This is where frozen indices come in. What if we could have indices that are still open — keeping them searchable — but do not occupy heap? We could add more storage to data nodes that hold frozen indices, and break the 1:100 ratio, understanding the tradeoff that searches might be slower.When we freeze an index, it becomes read-only and its transient data structures are dropped from memory. In turn, when we run a query on frozen indices, we will have to load the data structures to memory. Searching a frozen index doesn’t have to be slow. Lucene heavily depends on the filesystem cache which might have enough capacity to retain significant portions of your index in memory. In such a case searches are comparable in speed per shard. Yet, a frozen index is still throttled such that only one frozen shard is executing per node at the same time. This aspect might slow down searches compared to unfrozen indices.<h2>How freezing works</h2>Frozen indices are searched through a dedicated, <a href="https://www.elastic.co/guide/en/elasticsearch/reference/6.6/modules-threadpool.html#search-throttled">searched-throttled</a> threadpool. This by default uses a single thread, to ensure that frozen indices are loaded into memory one at a time. If concurrent searches are happening, they will get in the queue to add additional protections to prevent nodes from running out of memory.So, in a hot-warm architecture, we will now be able to transition indices from hot to warm, and then be able to freeze them before archiving or deleting them, allowing us to reduce our hardware requirements.Before frozen indices, to reduce infrastructure cost we had to snapshot and archive our data, which adds a significant operational cost. We would have to restore data if we needed to search again. Now, we can keep our historical data available for search, without a significant memory overhead. And if we need to write again on an already froze index, we can just <a href="https://www.elastic.co/guide/en/elasticsearch/reference/6.6/unfreeze-index-api.html">unfreeze it</a>.<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blta19e364f7d3f763a/5c98d09ec9ab24634dcd6d80/index-states-transp.png" data-sys-asset-uid="blta19e364f7d3f763a" alt="index-states-transp.png" width="400" style="margin: auto;display: block;"/><h2>How to freeze an Elasticsearch index</h2>Frozen indices are easy to implement in your cluster, so let’s get started on how to use the <a href="https://www.elastic.co/guide/en/elasticsearch/reference/6.6/freeze-index-api.html">Freeze index API</a> and how to <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/sql-index-frozen.html" target="_self">search on frozen indices</a>.First, we'll start by creating some sample data on a test index.<pre class="prettyprint">POST /sampledata/_doc { "name": "Jane", "lastname": "Doe" } POST /sampledata/_doc { "name": "John", "lastname": "Doe" } </pre>And then check that our data has been ingested. This should return two hits:<pre class="prettyprint">GET /sampledata/_search </pre>As a best practice, before you freeze an index it’s recommended to first run a <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-forcemerge.html">force_merge</a>. This will ensure that each shard has only a single segment on disk. It will also provide much better compression and simplifies the data structures we will need when running an aggregation or a sorted search request on the frozen index. Running searches on a frozen index with multiple segments can have a significant performance overhead up to multiple orders of magnitude.<pre class="prettyprint">POST /sampledata/_forcemerge?max_num_segments=1 </pre>The next step is to just invoke a freeze on our index via de Freeze index API endpoint.<pre class="prettyprint">POST /sampledata/_freeze </pre><h2>Searching frozen indices</h2>Now that it’s frozen, you’ll see that regular searches won’t work. The reason for this is that, to limit memory consumption per node, frozen indices are&nbsp;<a href="https://www.elastic.co/guide/en/elasticsearch/reference/6.6/searching_a_frozen_index.html">throttled</a>. &nbsp;Since we could target a frozen index by mistake, we’ll prevent accidental slowdowns by specifically adding ignore_throttled=false to the request.<pre class="prettyprint">GET /sampledata/_search?ignore_throttled=false { "query": { "match": { "name": "jane" } } } </pre>Now we can check the status of our new index, by running the following request:<pre class="prettyprint">GET _cat/indices/sampledata?v&amp;h=health,status,index,pri,rep,docs.count,store.size </pre>This will return a result similar to the following, with the index status being ‘open’:<pre class="prettyprint">health status index pri rep docs.count store.size green open sampledata 5 1 2 17.8kb </pre>As mentioned above, we must protect the cluster from running out of memory, thus there is a limit in the number of frozen indices we can concurrently load for search on a node. The number of threads in the search-throttled threadpool defaults to 1, with a default queue of 100. This means that if we run more than one request, they will be queued up to a hundred. We can monitor the&nbsp;<a href="https://www.elastic.co/guide/en/elasticsearch/reference/6.6/cat-thread-pool.html#cat-thread-pool">threadpool status</a>, to check queues and rejections, with the following request:<pre class="prettyprint">GET _cat/thread_pool/search_throttled?v&amp;h=node_name,name,active,rejected,queue,completed&amp;s=node_name </pre>Which should return a response similar to:<pre class="prettyprint">node_name name active rejected queue completed instance-0000000000 search_throttled 0 0 0 25 instance-0000000001 search_throttled 0 0 0 22 instance-0000000002 search_throttled 0 0 0 0 </pre>Frozen indices might be slower, but they can be pre-filtered in a very efficient manner. It is also recommended to set the request parameter pre_filter_shard_size to 1.<pre class="prettyprint">GET /sampledata/_search?ignore_throttled=false&amp;pre_filter_shard_size=1 { "query": { "match": { "name": "jane" } } } </pre>This will not add a significant overhead to the query and will allow us to take advantage of the usual scenario. For example, when searching on a date range on time-series indices, not all shards will match.<h2>How to write to a frozen Elasticsearch index</h2>What will happen if we try to write on an already frozen index? Let’s go for it and find out.<pre class="prettyprint">POST /sampledata/_doc { "name": "Janie", "lastname": "Doe" } </pre>What happened? Frozen indices are read-only, so writing is blocked. We can check these in the index settings:<pre class="prettyprint">GET /sampledata/_settings?flat_settings=true </pre>Which will return:<pre class="prettyprint">{ "sampledata" : { "settings" : { "index.blocks.write" : "true", "index.frozen" : "true", .... } } } </pre>We have to use the&nbsp;<a href="https://www.elastic.co/guide/en/elasticsearch/reference/6.6/unfreeze-index-api.html">Unfreeze index API</a>, invoking the unfreeze endpoint on the index.<pre class="prettyprint">POST /sampledata/_unfreeze </pre>And now we'll be able to create a third document and search for it.<pre class="prettyprint">POST /sampledata/_doc { "name": "Janie", "lastname": "Doe" } GET /sampledata/_search { "query": { "match": { "name": "janie" } } } </pre>Unfreezing should be done only under exceptional situations. And remember to always run a `force_merge` before freezing the index again to ensure optimal performance.<h2>Using frozen indices in Kibana</h2>To begin with, we will need to load some sample data, like the <a href="https://www.elastic.co/guide/en/kibana/6.6/tutorial-sample-data.html">sample flight data</a>.Click on the “Add” button for Sample flight data.<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt142abc81bf2e4ac0/5c98d104632578d5055bb49f/kibana-load-data.png" alt="kibana-load-data.png" width="700" style="margin: auto;display: block;" data-sys-asset-uid="blt142abc81bf2e4ac0"/>We should now be able to see the loaded data by clicking the “View data” button. The dashboard will be similar to this one.<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt70ea061b7b398da9/5c98d0ff026179ca592ee205/flight-dashboard.png" width="100%" data-sys-asset-uid="blt70ea061b7b398da9" alt="flight-dashboard.png" style="margin: auto;display: block;"/>Now we can test freezing the index:<pre class="prettyprint">POST /kibana_sample_data_flights/_forcemerge?max_num_segments=1 POST /kibana_sample_data_flights/_freeze </pre>And if we go back to our dashboard, we’ll notice that the data has apparently “disappeared”.<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt0a68db6e72e372e7/5c98d0f9acf2956c56cb1953/empty-flight-dashboard.png" data-sys-asset-uid="blt0a68db6e72e372e7" alt="empty-flight-dashboard.png" width="100%" style="margin: auto;display: block;"/>We need to tell Kibana to allow searches on frozen indices, which is disabled by default.Go to Kibana Management, choose Advanced Settings. In the Search section, you will find that “Search in frozen indices” is disabled. Toggle to enable and save the changes.<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltf5f1d58c21e9ba65/5c98d0f249a201165fca1028/frozen-indices-kibana-settings.png" data-sys-asset-uid="bltf5f1d58c21e9ba65" alt="frozen-indices-kibana-settings.png" width="100%" style="margin: auto;display: block;"/>And the flight's dashboard will show the data again.<h2>Wrapping up</h2>Frozen Indices are a very powerful tool in hot-warm architectures. They enable a more cost-effective solution for increased retention while retaining online search. I recommend that you test your search latency with your hardware and data, to come with the right sizing and search latency for your frozen indices.Check out the <a href="https://www.elastic.co/guide/en/elasticsearch/reference/6.6/frozen-indices.html">Elasticsearch documentation</a> to learn more about the <a href="https://www.elastic.co/guide/en/elasticsearch/reference/6.6/freeze-index-api.html">Freeze index API</a>. And as always, if you have any questions, reach out on our <a href="https://discuss.elastic.co/">Discuss</a> forums. Happy freezing!

frozen-indices-freeze-index-api.jpg

frozen-indices-freeze-index-api-small.jpg

Creating frozen indices with the Elasticsearch Freeze index API

At Elastic, we are big fans of providing self-service tools to successfully deploy and maintain our software in production. One of the most popular features of Elasticsearch is the APIs used to retrieve it's operational data. While Logstash was always easy to get started with, it was a bit of a black box when it came to managing it in production. Administrators had to rely on external monitoring systems to make sure Logstash was functioning without any issues. A core theme in 5.0 release was to make it easier to manage Logstash. In this post, I'll show you how to monitor your Logstash filters using the inbuilt APIs and the <a href="https://www.elastic.co/products">Elastic Stack</a> itself.&nbsp;
<h2>What is my ingestion rate?</h2>
If you have deployed Logstash as part of a high-volume data ingestion pipeline, one of the most important metrics to monitor is “how fast is Logstash ingesting data”. Previously, to answer this question, you had to install the <a href="https://www.elastic.co/guide/en/logstash/current/plugins-filters-metrics.html">metrics filter</a> to generate the ingestion stats. Starting <a href="https://www.elastic.co/blog/logstash-5-2-0-released">with 5.2</a>, we've made it really easy to monitor Logstash using the Elastic Stack itself in addition to the <a href="https://www.elastic.co/guide/en/logstash/8.13/configuring-logstash.html" target="_self">native APIs</a>&nbsp;which were introduced in Logstash 5.0. The <a href="https://www.elastic.co/downloads/x-pack">x-pack plugin</a> for Elastic Stack bundles an agent that you can install on all your production Logstash nodes. It also ships with charts for the most important metrics information that let you visualize Logstash monitoring data out of the box. The monitoring feature in x-pack is under the <a href="https://www.elastic.co/subscriptions">Basic license</a> which means you get all this awesomeness for free. The APIs mentioned here are all part of the open source Logstash distribution.&nbsp;
Let's get started: 
<ul>
 <li>Install x-pack on your production Logstash nodes. This assumes you've already configured your production cluster for monitoring.</li>
</ul><pre class="prettyprint">cd $LS_HOME bin/logstash-plugin install x-pack </pre>
This plugin should be installed on all Logstash nodes you wish to monitor. The x-pack plugin extends your Logstash to periodically ship all the metrics it collects to your production Elasticsearch cluster. Once installed, you can configure the shipping agent using the logstash.yml file. 
<ul>
 <li>Configure your Elasticsearch end-point.</li>
</ul>
xpack.monitoring.elasticsearch.url: "http://my-es-prod-node:9200"
We strongly recommend that you enable security for your production Elasticsearch clusters. Once you've installed x-pack, you can use the extensive <a href="https://www.elastic.co/products/x-pack/security">security features</a> that ships with it. 
<ul>
 <li>Restart Logstash and navigate to http://kibana-url:5600</li>
</ul>
Pick the Logstash you want to get detailed monitoring info:
<img src="https://api.contentstack.io/v2/uploads/588febd1c387e5ea18b34892/download?uid=blt166210739a9e46e6" data-sys-asset-uid="blt166210739a9e46e6" alt="nodes_list.png" width="568" height="151" style="width: 568;height: 151;" />
<img src="https://api.contentstack.io/v2/uploads/588febf1f6ea3cf85f2c933c/download?uid=blt90f828912b55c071" data-sys-asset-uid="blt90f828912b55c071" alt="events_rate.png" width="575" height="311" style="width: 575;height: 311;" />
<h2>What are my slowest filters?</h2>
Filters are the workhorses of Logstash, which means they can be the slowest stage of your pipeline, compared to the inputs and outputs stage. The performance of a filter is also dependent on the type of data it processes. For example, the grok filter, a very popular one, uses a regular expression engine to extract fields. And certain types of event data can throw your grok filter into an expensive backtracking loop. Before we attempt to tweak the filters for performance, the first step if to actually figure out which one of them is the slowest: <pre class="prettyprint">curl localhost:9600/_node/stats/pipeline?pretty </pre><pre class="prettyprint">... "filters":[ { "id":"2178468f89ab43c6f32e934d3cb587e3c850d93a-4", "events":{ "duration_in_millis":24005, "in":49482, "out":49480 }, "name":"geoip" }, { "id":"2178468f89ab43c6f32e934d3cb587e3c850d93a-2", "events":{ "duration_in_millis":84193, "in":49483, "out":49482 }, "matches":29400, "failures":20083, "patterns_per_field":{ "message":1 }, "name":"grok" }, { "id":"2178468f89ab43c6f32e934d3cb587e3c850d93a-5", "events":{ "duration_in_millis":44353, "in":49480, "out":49477 }, "name":"useragent" } ... </pre>
For all the plugins in our pipeline, this API provides a per-pipeline break down of event stats. One particular field of interest is the duration_in_millis field which reports the total amount of time spent in a filter for all the events that passed through it. If you divide duration_in_millis by events.in, you should get the average time events have spent in this filter. From the response above, it looks like the grok filter is the slowest, followed by the useragent filter. Zooming in on the grok filters, we can also see that there are quite a bit of grok match failures. It is plausible that the regular expression engine tried parsing the pattern(s), and may have gone into a backtracking loop wasting a lot of CPU cycles, contributing to the overall slowness.
<img src="https://api.contentstack.io/v2/uploads/588fed461dd6e41e6ff76159/download?uid=blt6092a17c7ccb1cba" data-sys-asset-uid="blt6092a17c7ccb1cba" alt="tip.png" style="background-color: initial;float: left;margin: 0px;text-align: left;" />&nbsp; An important tip to our readers — if you see the response above, the ID field contains a randomly UUID which identifies each of the filters. You can also, and its highly recommended that you set the ID field in your config file to label plugins appropriately. This can help a lot when&nbsp;debugging a pipeline with multiple grok filters. For example, in your config file :<pre class="prettyprint">filter { grok { id =&gt; "apache_common_log_praser" pattern =&gt; .. } grok { id =&gt; "java_exception_logs_parser" pattern =&gt; .. } } </pre>
Now that we've identified the grok filter as the slowest, we can tweak the regular expression pattern to optimize the performance. We have <a href="https://www.elastic.co/blog/do-you-grok-grok">previously posted</a> about various ways to tame your Grok filter, which is a highly recommended read.
If you head over to the monitoring UI, the events latency graph shows you aggregated information on how long it takes on average for events to be processed in your pipeline. 
<img src="https://api.contentstack.io/v2/uploads/588fec3b647b87d73bfa45bd/download?uid=blt7676c01a7e3493a7" data-sys-asset-uid="blt7676c01a7e3493a7" alt="Screen Shot 2017-01-29 at 08.37.54.png" width="426" height="141" style="width: 426;height: 141;" />
One of my favorite parts of shipping monitoring data to Elasticsearch is the access to historical information. As you iterate over your configs, you can validate from these graphs if your changes made a difference to the overall pipeline latency. This historical information can also be used to spot performance regressions over time in your production setting. You are no longer flying in the dark! 
<h2>Slowlog</h2>
In our previous section, we showed you how to identify the slow sections of the pipeline. But, this doesn't pinpoint which events may have contributed to the slowdown. Enter the <a href="https://www.elastic.co/guide/en/logstash/current/logging.html#_slowlog">slowlog feature</a>. Slowlog adds the ability to log when a specific event takes an abnormal amount of time to make its way through the pipeline. You can define the criteria when you want the slowlog to be triggered. Just like normal application logs, you can find slow-logs in your —path.logs directory. In your logstash.yml file, add this setting: <pre class="prettyprint">slowlog.threshold.warn: 2s </pre>
When events take more than 2 seconds to be processed, you should see: <pre class="prettyprint">[2016-10-27T13:48:19,308][WARN ][slowlog.logstash.filters.grok] event processing time {:took_in_millis=&gt; 2130, :event=&gt;"{\"sequence\":1,\"@timestamp\":\"2016-10-27T20:48:18.150Z\",\"@version\":\"1\",\"host\":\"elasticbox-2.local\",\"mes sage: \"217.69.133.237 - - [11/Sep/2014:21:58:29 +0000] "GET /scripts/netcat-webserver HTTP/1.1" 304 - "-" "Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; +http://go.mail.ru/help/robots)""\"}\"} </pre>
<h2>Hot Spots</h2>
If the CPU on your Logstash instance is running red hot, you can issue a command like top -H to analyze what parts of the Logstash pipeline is consuming the most CPU. Logstash threads are labeled based on the stage and plugin name, so from the output of top, one can quickly find out which parts are CPU intensive. For example, [main]&gt;worker1 denotes a filter + output thread, while [main]&lt;generator denotes a generator input. 
<img src="https://api.contentstack.io/v2/uploads/588fed9e2886d7f63f63b5d7/download?uid=blt1f9f15aed960df49" data-sys-asset-uid="blt1f9f15aed960df49" alt="top_output.png" width="396" height="96" style="width: 396;height: 96;" />
While this is super useful, it would be nice to zoom into those threads to find the expensive parts. We have an API just for that — <a href="https://www.elastic.co/guide/en/logstash/current/hot-threads-api.html">the hot threads API</a> provides a thread dump or a snapshot of the java threads sorted by the time spent. <pre class="prettyprint">curl localhost:9600/_node/hot_threads?human ::: {} Hot threads at 2017-01-29T04:18:55+00:00, busiestThreads=3: ================================================================================-------------------------------------------------------------------------------- 64.07 % of cpu usage, state: runnable, thread name: '[main]&gt;worker0' org.jruby.RubyRegexp$SearchMatchTask.run(RubyRegexp.java:273) org.jruby.RubyThread.executeBlockingTask(RubyThread.java:1066) org.jruby.RubyRegexp.matcherSearch(RubyRegexp.java:235) org.jruby.RubyRegexp.search19(RubyRegexp.java:1780) org.jruby.RubyRegexp.matchPos(RubyRegexp.java:1720) org.jruby.RubyRegexp.match19Common(RubyRegexp.java:1701) org.jruby.RubyRegexp.match_m19(RubyRegexp.java:1680) org.jruby.RubyRegexp$INVOKER$i$match_m19.call(RubyRegexp$INVOKER$i$match_m19.gen) org.jruby.internal.runtime.methods.JavaMethod$JavaMethodOneOrNBlock.call(JavaMethod.java:350) org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168) </pre>
In the above example response, we see again that the regex engine used by the Grok filter is the busiest thread. You could also use this API to find out if your Logstash is stuck performing I/O operations. 
<h2>Conclusion</h2>
In this post, we introduced you to our new monitoring UI feature and showed a few ways to identify slow parts of Logstash. We are not done yet — our roadmap has exciting monitoring features that will make Logstash and your pipelines even more of a breeze to manage. This post is part of a series on monitoring, so stay tuned for our next one and all our product updates.

events_rate.png

Monitoring Logstash Filters: X-ray glasses included

44281324-e7a8c880-a203-11e8-8b54-3a15c15f34a3.jpg

44281325-e7a8c880-a203-11e8-9fed-61b9ee218354.jpg

Elasticsearch Audit Trail and Log File Filter Policies Explained

This blog post outlines the upgrade path that you can use to upgrade your Elasticsearch 1.7 cluster hosted on Elasticsearch Service to a supported version. Newer versions of Elasticsearch include an incredible amount of features like <a href="/products/stack/canvas">Canvas</a>, <a href="/products/beats">Beats</a>, <a href="/solutions/apm">APM</a> and much more. Keeping a cluster updated to the latest version also provides other benefits, such as access to the latest security patches and performance improvements.
As per our <a href="/support/eol">product end-of-life policy</a>, Elasticsearch 1.7 reached end of life (EOL) 26 months ago. In late 2018, we began an email campaign informing customers that <a href="/cloud/elasticsearch-service">Elasticsearch Service</a> will cease operations for Elasticsearch 1.7 clusters on April 8, 2019. Although Elasticsearch Service makes upgrading to the next major version very easy via the console, unfortunately this upgrade path isn’t available for extremely old versions of Elasticsearch.
At a minimum, you need to use the steps in this blog post to upgrade to version 5.6.x, but we recommend that you also upgrade to Elasticsearch 6.x to avoid having to deal with an EOL version again in the near future. You should stay on version 5.6.x only if you cannot switch to a <a href="/blog/removal-of-mapping-types-elasticsearch">single mapping type</a> for indices easily and need more time before upgrading to 6.x.
<h2>How to upgrade your Elasticsearch 1.7 cluster</h2><ol>
	<li>Log into the <a href="http://cloud.elastic.co">Elasticsearch Service console</a>.</li>
	<li>Create a new 5.6.x deployment using the latest available version.</li>
	<li>Index your data into the new deployment:</li>
	<ul>
		<li style="list-style-type:disc;">If you have access to the source, index all of your data directly into the new deployment from the source application. </li>
		<li style="list-style-type:disc;">Alternatively, if you do not have access to the data source, you can use <a href="/guide/en/elasticsearch/reference/6.6/docs-reindex.html">reindex from remote</a> from your new 5.6.x deployment, where the source is set to your existing Elasticsearch 1.7 deployment. As described below, ensure that breaking changes, like mapping updates for string data types, are addressed prior to reindexing. Please also note, it would be advisable to create these indices with the appropriate mappings beforehand. 
		<pre class="prettyprint">POST _reindex
{
 "source": {
 "remote": {
 "host": "https://SOURCE_1_7_CLUSTER_URL:9243",
 "username": "SOURCE_1_7_CLUSTER_USERNAME",
 "password": "SOURCE_1_7_CLUSTER_PASSWORD"
 },
 "index": "INDEX_NAME",
 "query": {
 "match_all": {
 }
 }
 },
 "dest": {
 "index": "INDEX_NAME"
 }
}
 </pre> Important: Be sure to replace <code>SOURCE_1_7_CLUSTER_URL</code>, <code>SOURCE_1_7_CLUSTER_USERNAME</code>, <code>SOURCE_1_7_CLUSTER_PASSWORD</code>, and <code>INDEX_NAME</code> with your own information.
		</li>
	</ul>
	<li>Upgrade to 6.6.x via the Elasticsearch Service console using the Upgrade Assistant. </li>
</ol>There are some breaking changes to be aware of, though this list is not exhaustive. Please reference the breaking changes for <a href="/guide/en/elasticsearch/reference/5.6/breaking-changes-5.0.html">5.x</a> or <a href="/guide/en/elasticsearch/reference/current/breaking-changes-6.0.html">6.x</a>, depending on which version you are upgrading to:
<ul>
	<li>When upgrading from 1.7 to 5.6/6.x, the ‘string’ data type was used for all text strings, with <code>"type": "string", "index": "analyzed"</code> for full text search capable fields and <code>"type": "string", "index": "not_analyzed"</code> for fields that required an exact value. To learn more, see <a href="/blog/strings-are-dead-long-live-strings">Strings are dead, long live strings!</a> This behavior has been changed and simplified:</li>
	<ul>
		<li><code>"type": "string", "index": "analyzed"</code> now is just <a href="/guide/en/elasticsearch/reference/6.6/text.html"><code>"type": "text"</code></a></li>
		<li><code>"type": "string", "index": "not_analyzed"</code> now is just <a href="/guide/en/elasticsearch/reference/6.6/keyword.html"><code>"type": "keyword"</code></a> </li>
	</ul>
	<li>A full list of the breaking changes introduced with each major and minor update can be found in our documentation for each version: <a href="/guide/en/elasticsearch/reference/2.4/breaking-changes-2.0.html">2.x</a>, <a href="/guide/en/elasticsearch/reference/5.6/breaking-changes-5.0.html">5.x</a>, <a href="/guide/en/elasticsearch/reference/6.6/breaking-changes-6.0.html">6.x</a>. The breaking changes for minor version are listed in the documentation menu.</li>
</ul>If none of the options in this blog post work for you, you can also consider moving your data off Elasticsearch Service entirely and run Elasticsearch on your own. To get your data off Elasticsearch Service, snapshot your Elasticsearch 1.7 cluster to a <a href="/guide/en/cloud/current/ec-custom-repository.html">custom repository</a> and then restore the snapshot to a self-managed Elasticsearch 1.7 cluster. Note that this version of Elasticsearch is known to have security issues, is no longer being patched, and is not recommended.
We’ve added a huge amount of new functionality to the Elastic Stack since Elasticsearch 1.7. If you haven't seen them yet, explore the <a href="/products">new features</a> that have been released in the past few years, or even join the <a href="/v7">7.0 Elastic Pioneer Program</a> to test out features that are just around the corner.

blog-banner-release-elasticsearch.png

blog-thumb-release-elasticsearch.png

Elastic Support Alert: Upgrading Your Elasticsearch 1.7 Cluster on Elastic Cloud

Elastic Cloud and Meltdown

<html><head></head><body>Quite some time ago, with the GA release of Elasticsearch 5.0 we released an <a href="/guide/en/elasticsearch/reference/current/indices-shrink-index.html">API</a> that allows to shrink an index into a new index with fewer shards than the original index. The reasoning behind adding this functionality was to provide a tool to keep the number of shards in a cluster at bay. Indices are commonly created with a large number of shards to maximise indexing throughput but later once theses indices are rolled over into a daily or hourly index, the number of shards should be reduced again to maximise resource utilization of the cluster. While this was a step in the right direction, we were still missing the ability to go in reverse: to increase the number of shards in an index if the index has become too small for the number of documents in it.
<h2>We had a story for this, we thought. </h2>For the entire history of Elasticsearch the answer to the question of how to split an index was to just add another index and grow your shard count that way. Under the hood Elasticsearch just operates on a set of shards which allows you to add indices after the fact and search across them. This sounds like a great plan for use-cases where documents are largely immutable like metrics and logging, but not as great when &#xA0;you have to update documents, as is the case for use cases like full-text search. For this you had to go and figure out in what index the document was added and update it there or you delete the document everywhere and add it to the latest index. All of these options have downsides and can cause temporary duplicates or missing documents which are not always acceptable.
After having some deep technical conversations with some users on a trip to Munich I was convinced we have to do something about this. But some of the options scared me a fair bit. For instance splitting an index or a shard while the index is actively indexing sounds very appealing in the first place but when you start thinking about the implications and how it would work under the hood, the approach doesn&#x2019;t sound all that great anymore. Aside from all the internal changes of how we do consistent hashing, how the number of shards are immutable in an index and how we can make this all work in a backwards compatible way, the performance implications of resharding while a cluster is rebalancing or loses a node had so many edge cases that this feature would likely take many engineering iterations to get right.
Luckily we already had implemented the shrink API that was basically doing the same thing in reverse with a lot of interesting properties. Shrink works by creating a copy of the index that recovers from the original index but uses fewer shards. Wait, a copy? That sounds very expensive copying all the data. To prevent this we make use of the properties lucene provides us by never touching a file more than once. Given that we can take point-in-time snapshots of the source index and use hard-links on the filesystem level, the shrink operation is almost instant and doesn&#x2019;t require any additional space. All properties of Elasticsearch indices (such as the fixed number of shards) are maintained during this time. If any issues occur during the shrink, the shrink can be aborted and the target index deleted.
This sounds great, let&#x2019;s see if we can do the same thing for splitting an index as well. And in fact it worked like a charm. The <a href="/guide/en/elasticsearch/reference/current/indices-split-index.html">split</a> API uses the same mechanics as the shrink API. Once an index is split, Elasticsearch allocates the shards of the target index alongside of the source index and recovers from the point-in-time snapshot of the underlying Lucene index. The only significant difference is that we need to do a single pass through the index on disk to mark all documents as deleted that don&#x2019;t belong in the target shard. This means we have an additional space requirement of 1 bit per document while all other data files are reused via hard-links. Just like the shrink process this is a reasonably fast operation that typically happens within a couple of seconds and can be monitored with the <a href="/guide/en/elasticsearch/reference/current/indices-recovery.html">recovery API</a>.
Unfortunately you can&#x2019;t rename an index in Elasticsearch today. To make efficient use of this API, the usage of <a href="/guide/en/elasticsearch/reference/current/indices-aliases.html">index aliases</a> is highly recommended. If an index needs to be resized, users should split or shrink the index, flip an alias to point to the new index and delete the old index in a single atomic step, using the <a href="/guide/en/elasticsearch/reference/6.2/indices-aliases.html">update-aliases API</a>.
<h2>It&#x2019;s all about trade-offs.</h2>That is likely the story of every software engineers life. In this case we made the trade-off that indexing has to stop for a short amount of time until the index is resized since source indices must be read-only. But as mentioned above these operations are considered fast, and given that this provides transactional behavior with the least amount of surprise to the user we are following our consistent strategy to go with the safe option rather than the fancy one. For details on split factors and how to pick the number of target shards, please see the official <a href="/guide/en/elasticsearch/reference/current/indices-split-index.html">documentation</a> for the specific version of Elasticsearch you are using.
</body></html>

Elasticsearch Platform

ELK Stack

Elastic Cloud

Observability

Security

Search

By industry

By solution

Customer spotlight

Developers

Connect

Learn

Help

See what's happening at Elastic

Follow us

About us

Join us

Press

Partners

Trust & Security

Investor relations

EXCELLENCE AWARDS