Send Security Command Center data to Elastic Stack

This page explains how to automatically send Security Command Center findings, assets, and security sources to Elastic Stack without using a Docker container. It also describes how to manage the exported data. Elastic Stack is a security information and event management (SIEM) platform that ingests data from one or more sources and lets security teams manage responses to incidents and perform real-time analytics. The Elastic Stack configuration discussed in this guide includes four components:

Filebeat: a lightweight agent installed on edge hosts, such as virtual machines (VM), that can be configured to collect and forward data
Logstash: a transformation service that ingests data, maps it into required fields, and forwards the results to Elasticsearch
Elasticsearch: a search database engine that stores data
Kibana: powers dashboards that let you visualize and analyze data

Upgrade to the latest release

To upgrade to the latest release, you must deploy a Docker container image that includes the GoApp module. For more information, see Exporting assets and findings with Docker and Elastic Stack.

Prerequisites and system cleanup

Before installing the new Docker container, clean up your legacy integration and establish required permissions and Pub/Sub topics:

Delete the following files and directories:
- /etc/systemd/system/go_script.service
- The GoApp directory
- Logstash configurations
- logstash2.service
- filebeat.service
Optional: To avoid issues when importing the new dashboards, remove the existing dashboards from Kibana:
1. Open the Kibana application.
2. In the navigation menu, go to Stack Management, and then click Saved Objects.
3. Search for Google SCC.
4. Select all the dashboards that you want to remove.
5. Click Delete.
Add the Logs Configuration Writer (roles/logging.configWriter) role to the service account.
Create a Pub/Sub topic for your audit logs.
Optionally, if you are installing the Docker container in another cloud, configure workload identity federation instead of using service account keys. You must create short-lived service account credentials and download the credential configuration file.

Configure the Elastic data stream and Docker container

Complete the steps in Download the GoApp module.
Complete the steps in Install the Docker container.
Complete the steps in Update permissions for audit logs.

Visualization and dashboard setup

Import all the dashboards, as described in Import Kibana dashboards.

Use the instructions in Exporting assets and findings with Docker and Elastic Stack to administer your SIEM integration.

Manage service and logs

This section explains how to view GoApp module logs and make changes to the module's configuration.

This section applies only to the GoApp module that you installed from the GoogleSCCElasticIntegration installation package that was made available in February 2022. For up-to-date information, see Upgrade to the latest release.

Check the status of the service:
```
  systemctl | grep go_script
```
Check the current working logs, which contain information on execution failures and other service information:
```
 sudo journalctl -f -u go_script.service
```
Check historical and current working logs:
```
  sudo journalctl -u go_script.service
```
To troubleshoot or check the logs of go_script.service:
```
  cat go.log
```

Configure Elastic Stack applications

This section explains how to configure Elastic Stack applications to ingest Security Command Center data. The instructions assume you properly installed and enabled Elastic Stack, and that you have root privileges in the application environment.

View Logstash service logs

To view current logs, run the following command:

    sudo journalctl -f -u logstash2.service

To view historical logs, run the following command:

    sudo journalctl -u logstash2.service

Set up Filebeat

View Filebeat service logs

To view current logs, run the following command:

    sudo journalctl -f -u filebeat.service

To view historical logs, run the following command:

    sudo journalctl -u filebeat.service

View Kibana dashboards

You can use custom dashboards in Elastic Stack to visualize and analyze your findings, assets, and security sources. The dashboards display critical findings and help your security team prioritize fixes.

Overview

The Overview dashboard contains a series of charts that displays the total number of findings in your organization by severity level, category, and state. Findings are compiled from Security Command Center's built-in services—Security Health Analytics, Web Security Scanner, Event Threat Detection, and Container Threat Detection—and any integrated services you enable.

Additional charts show which categories, projects, and assets are generating the most findings.

Assets

The Assets dashboard displays tables that show your Google Cloud assets. The tables show asset owners, asset counts by resource type and projects, and your most recently added and updated assets.

You can filter asset data by time range, resource name, resource type, owner, and project, and quickly drill down to findings for specific assets. If you click an asset name, you are redirected to Security Command Center's Assets page in the Google Cloud console and shown details for the selected asset.

Findings

The Findings dashboard includes a table showing your most recent findings. You can filter the data by resource name, category, and severity.

Table columns include finding name, in the format of organizations/<var>ORGANIZATION_ID</var>/sources/<var>SOURCE_ID</var>/findings/<var>FINDING_ID</var>, category, resource name, event time, create time, parent name, parent URI, and security marks. The format of parent URI matches finding name. If you click a finding name, you are redirected to Security Command Center's Findings page in the Google Cloud console and shown details for the selected finding.

Sources

The Sources dashboard shows the total number of findings and security sources, the number of findings by source name, and a table of all your security sources. Table columns include name, display name, and description.

Edit dashboards

To customize the data that is displayed in your Kibana dashboards, you can add or remove columns.

Add columns

Navigate to a dashboard.
Click Edit, and then click Edit visualization.
Under Add sub-bucket, select Split rows.
In the list, select Aggregation.
In the Descending drop-down menu, select ascending or descending. In the Size field, enter the maximum number of rows for the table.
Select the column you want to add.
Save the changes.

Remove columns

Navigate to the dashboard.
Click Edit.
To hide a column, next to the column name, click the visibility, or eye, icon.
To remove a column, next to the column name, click the X, or delete, icon.

Uninstall the component

Uninstall the GoApp module and Elastic Stack applications when you no longer want to retrieve Security Command Center data for Elastic Stack.

Uninstall the GoApp module

Delete go_script.service from /etc/systemd/system/.
Remove feeds for assets and IAM policies.
Remove Pub/Sub for assets, IAM policies, and findings.
Delete the working directory.

Uninstall Logstash

Delete Logstash configurations.
Delete logstash2.service.

Uninstall Filebeat

Delete Filebeat configurations.
Delete filebeat.service.

What's next

Upgrade to the latest version to integrate Security Command Center with Elastic Stack.
Learn more about setting up finding notifications in Security Command Center.
Read about filtering finding notifications in Security Command Center.