# ELK Stack

{% embed url="<https://youtu.be/j9A5dNqCb9Q>" %}

Once configured, Scoutflo can:

* Query **Elasticsearch indices** for relevant logs around incident timelines
* Correlate **log patterns** with Kubernetes events, Sentry errors, and cloud changes
* Use log excerpts as **evidence** in root‑cause analysis

***

### ✅ Pre requisites

Before setting up the ELK integration, ensure you have:

* An **Elasticsearch cluster** (cloud‑hosted or self‑hosted) with API access
* **Administrative access** to create **API keys** in Elasticsearch
* Access to **Scoutflo’s Integrations** tab in the platform UI
* For **self‑hosted** setups:
  * The **Elasticsearch hostname / endpoint** (for example, `https://es.yourdomain.com:9200`)
* Familiarity with your **index patterns** and **retention** settings

If you don’t have these, coordinate with your **SRE / Platform** team before proceeding.

***

### 🧩 Setup Guide

To connect ELK to Scoutflo, you’ll:

1. Create an **API key** in Elasticsearch
2. Configure the **connection** in Scoutflo
3. (Optional) Configure **self‑hosted** Elasticsearch

This enables Scoutflo’s AI engine to search your logs as part of incident investigations.

***

### Step 1: Create an API key in Elasticsearch

1. Log in to your **Elasticsearch cluster** management interface.
2. Navigate to **Management → Security Settings → API Keys**.
3. Click **Create API Key**.
4. Configure the API key:
   * **Name**: Something descriptive, for example:\
     `Scoutflo AI SRE` or `Scoutflo Logs Integration`
   * **Role**: Use a role with **read access** to the relevant indices.\
     Common options:
     * `logstash_reader` (if using Logstash indices)
     * A custom role scoped to your log indices
     * `elastic` superuser (if no restrictions)
5. Click **Create API key**.
6. **Copy the generated token** immediately (it’s shown only once).

Store this token securely, for example:

* In the Scoutflo Integrations UI (recommended)
* As an environment variable: `SCOUTFLO_ELK_API_TOKEN`

{% hint style="warning" %}
Treat this API key like any other secret. Do not commit it to Git or share it in plain text.
{% endhint %}

***

### Step 2: Configure ELK in Scoutflo

In the Scoutflo platform UI:

1. Navigate to **Settings → Integrations → ELK**.
2. Fill in the configuration fields:

| Field                 | Description                             | Example                               |
| --------------------- | --------------------------------------- | ------------------------------------- |
| **Elasticsearch URL** | Full URL to your Elasticsearch endpoint | `https://es.yourdomain.com:9200`      |
| **API Key**           | Token from Step 1                       | `eyJ2ZXIiOiI5LjAuMCIsImFkciI6W119...` |
| **Index pattern**     | Default indices to search (optional)    | `logs-*, app-logs-*`                  |

3. Click **Test connection** to validate:
   * Elasticsearch endpoint is reachable
   * API key authenticates successfully
   * Scoutflo can list indices or run a simple search
4. If the test succeeds, click **Save** to activate the integration.

***

### Step 3: Configure self‑hosted Elasticsearch (optional)

The setup above already works for **self‑hosted** Elasticsearch. Just make sure:

1. The **Elasticsearch URL** includes:
   * Correct protocol (`https://` recommended)
   * Correct hostname / port (`es.yourdomain.com:9200`)
2. The API key has **read access** to your log indices.
3. The Elasticsearch cluster is **reachable** from Scoutflo:
   * Check DNS resolution
   * Verify firewall rules (ports 9200 or 443)
   * Confirm TLS certificate is valid and trusted

If using a **proxy** or **VPN**, ensure Scoutflo’s outbound traffic can reach your Elasticsearch endpoint.

***

### ✅ Post‑setup validation

After completing the steps above, do these quick checks:

1. **Test the connection in Scoutflo**
   * In **Integrations → ELK**, run **Test connection**.
   * Confirm the status shows **Connected**.
2. **Verify index access**
   * From Scoutflo, list available indices or run a simple search.
   * Confirm your expected log indices are visible.
3. **Check log flow in investigations**
   * Trigger an incident or investigation that should generate logs.
   * Confirm that relevant log snippets appear in the Scoutflo investigation timeline.

***

### ⚠️ Most common failure points

* **Invalid API key:**\
  Double‑check that the key is copied exactly (no extra spaces or missing characters). Regenerate in Elasticsearch if issues persist.
* **Wrong or unreachable URL:**\
  Verify the Elasticsearch URL is correct and accessible from Scoutflo. Check DNS, ports (9200 or 443), and firewall rules.
* **Insufficient permissions:**\
  Confirm the API key role has **read access** to your log indices. Test with a broader role temporarily if needed.
* **No logs in investigations:**\
  Check that your index pattern matches the actual indices Scoutflo should search. Verify that logs are being generated around incident timestamps.