Aquileo | AWS Kinesis Enhancements by kyleknighted · Pull Request #279 · Graylog2/graylog-plugin-integrations

kyleknighted · 2019-09-26T17:24:20Z

Overview

This PR makes the following changes for the graylog-plugin-integrations AWS Kinesis/CloudWatch input:

Improve authorization support: Previously, only an explicit AWS Key/Secret could be used to authorize the input. Now, the user can pick between using a key/secret and automatic AWS authorization, which uses the DefaultAWSCredentialsProviderChain, which allows many more flexible authentication options.

See #264 for more info.

As part of this change, the Assume Role ARN option was also moved to the first page (previously was tucked-away on a subsequent page, which was too late, since many AWS SDK calls are initiate starting from the first page):

Add ability to specify VPC endpoints for all AWS API services used (CloudWatch, DynamoDB, IAM, and Kinesis). Fields for this are now present on the first page of the setup, since the endpoints must be used throughout the setup wizard. See Support Interface VPC Endpoints for Kinesis Data Streams #271 for more info.

Lots of code was changed to implement these, because many API calls support this functionality, and almost all of them had to be updated :)

Closes #264, #271

danotorrey · 2019-09-30T14:14:55Z

I am planning to integrate the backend changes into this PR too. I'll start work on that tomorrow or Wednesday.

danotorrey · 2019-10-16T21:03:31Z

@kyleknighted I have begun working on the backend for this. Sorry for the delay!

danotorrey · 2019-10-16T21:49:37Z

@kyleknighted I noticed this error in the browser when running this branch:

client.js:196 /Users/danieltorrey/workspace/graylog/master/graylog-project-repos/graylog-plugin-integrations/src/web/aws/common/awsAuth.jsModule not found: Error: Can't resolve './awsKeySecret' in '/Users/danieltorrey/workspace/graylog/master/graylog-project-repos/graylog-plugin-integrations/src/web/aws/common'

danotorrey · 2019-10-28T19:13:02Z

@kyleknighted I have begun working on the backend for this again. I'l keep you updated on my progress.

The initial policy needs to be fully resolved before attempting to assume a role. This is because the initial role must have the sts:AssumeRole permission to assume a role. This also allows any authentication scenario to support assuming a role. This matches the auth logic previously supported in the original AWS plugin.

Allow nullable assume role ARN as it will not always be specified.

danotorrey · 2019-10-30T22:11:42Z

@kyleknighted Initial testing looks good so far. Noticed one possible issue:

Looks like the Assume Role ARN is not being passed in the Streams request (specified in input field, but is not included in the post request JSON payload):

lingpri · 2019-12-11T17:52:08Z

@danotorrey @kyleknighted @ceruleancee Great work ! If you need any help with any specific task Please keep me posted. I had a chance to look at creating the kinesis input with Claudia's help and we are able to see the log messages. Also, we weren't sure how to test the VPC endpoints(Dynamo DB API Endpoint override ). Is there a Swagger documentation for these end points, that we can access. Thanks.

danotorrey · 2019-12-16T10:36:51Z

@lingpri @ceruleancee I investigated and found that there are valid use cases to allow an override endpoint for the IamClient AWS API URL. I feel that we should provide endpoint overrides for all AWS SDK services that we communicate to (just as the PR is currently coded to do).

IAM endpoint override might be needed if the AWS FIPS endpoints are required by the user (https://aws.amazon.com/compliance/fips/)
Other vendors that integrate with AWS allow IAM endpoint overrides. For example https://www.terraform.io/docs/providers/aws/guides/custom-service-endpoints.html#iam

Please also note that for the security concern of allowing IAM overrides, we are also adding URL white list capability, which allows the user to control which URLs are allowed to be used within Graylog. I will add support for this in a later PR.

danotorrey · 2019-12-16T10:37:21Z

@ceruleancee @lingpri This PR is ready to be merged unless you have any other concerns.

lingpri

Great Work . Thank you very much for your patience. I enjoyed reviewing the PR .

cobaltclaudia · 2019-12-10T17:42:11Z

 import com.google.auto.value.AutoValue;
 import org.graylog.autovalue.WithBeanGetter;

+import javax.annotation.Nullable;


delete unneeded imports
import com.fasterxml.jackson.annotation.JsonAutoDetect;
import com.fasterxml.jackson.annotation.JsonProperty;
import javax.annotation.Nullable;

* Moving CloudWatch and StepAuthorize * Removing unnecessary AuthProvider * cleaner dir paths * cleanup * Render different auth options * ARN field * Swapping fields * Adding Automatic content * cleanup * Remember auth type * Add auth provider using latest SDK version * Add cross account role tester * Resolve the initial policy before attempting to assume a role The initial policy needs to be fully resolved before attempting to assume a role. This is because the initial role must have the sts:AssumeRole permission to assume a role. This also allows any authentication scenario to support assuming a role. This matches the auth logic previously supported in the original AWS plugin. * Use AWSAuthProvider with assume role support for all AWS requests * Flip incorrect logic for when credentials and auto auth are used * Fix failing unit tests Allow nullable assume role ARN as it will not always be specified. * Updating Auth experience * validation and review: * Passing arn with intial streams request * Fix invalid role name * Every auth has arn * update note * Fix incorrect role ARN * Updating button implementation * Review value of undefined * Cleanup auth provider formatting * Remove cross account role test * More auth provider formatting adjustments * Add request support for AWS SDK client endpoint overrides * Specify endpoints at runtime * Add endpoint config fields * Migrate to centralized AWS client initializer Centralized initializer appropriately specifies general properties such as region, auth provider, and override endpoint (if specified). * Migrate to centralized AWS client initializer (part 2) Centralized initializer appropriately specifies general properties such as region, auth provider, and override endpoint (if specified). * Specify SDK override endpoint for each Graylog AWS API endpoint * Consolidate passed AWSRequest arguments Passing individual arguments is verbose and can lead to transposition errors. * Centralize logic for initializing and building AWS SDK clients * Cleanup * Undo unintended Palo Alto formatting changes * Undo unintended Palo Alto formatting changes * Key and secret are not required * Validate override endpoint URIs to avoid silent failure * Update to latest AWS SDK and Kinesis client * Adding VPC Endpoint UI * remove icon size sm * fixed review value of undefined * Validate endpoint URLs * Shared section title styles * Formatting cleanup * Rework API value classes to inherit common properties This removes the many key, secret, arn, and endpoints fields from all request classes. * Migrate the generic AWSRequestImpl to use common fields VIA interface * Add back integrations.protobuf.version variable * Improve roleSessionName logic Do not include "NONE" when access key is not present. * Comment cleanup * Add back test credentials * Cleanup, finish implementing ignored `testCreateNewKinesisStream` test * Fix failing test * Add exception object to error message * Save endpoints at end of wizard setup * Do not save a title, which is deleted anyway when saving the input * Update placeholder for endpoints * Adjust text * Adjust endpoint placeholder * Fix failing test * Add URL validation unit test

kyleknighted added 11 commits September 25, 2019 14:21

Moving CloudWatch and StepAuthorize

2fbed66

Removing unnecessary AuthProvider

7f95b6b

cleaner dir paths

443f536

cleanup

f1eec61

Render different auth options

1a829b3

ARN field

7f8e4bf

Swapping fields

b73ab5c

Adding Automatic content

7bca631

cleanup

2c5ffb6

Remember auth type

8500b36

conflix

53f61d4

danotorrey closed this Oct 16, 2019

danotorrey reopened this Oct 16, 2019

danotorrey self-assigned this Oct 16, 2019

Add auth provider using latest SDK version

cbe3eb6

Dan Torrey and others added 7 commits October 28, 2019 16:52

Add cross account role tester

d597c3f

Use AWSAuthProvider with assume role support for all AWS requests

31b59a8

Flip incorrect logic for when credentials and auto auth are used

4a29d63

Fix failing unit tests

a3667bc

Allow nullable assume role ARN as it will not always be specified.

Updating Auth experience

205c5e4

validation and review:

90fbff1

kyleknighted and others added 3 commits October 31, 2019 07:14

Passing arn with intial streams request

c2fefae

Fix invalid role name

d9cea47

Every auth has arn

1a82439

danotorrey requested review from alex-konn and cobaltclaudia December 11, 2019 11:53

Dan Torrey added 6 commits December 11, 2019 14:29

Save endpoints at end of wizard setup

115fd40

Do not save a title, which is deleted anyway when saving the input

628f89c

Update placeholder for endpoints

8724b18

Adjust text

0e4810b

Adjust endpoint placeholder

921c4bf

Fix failing test

28e69b3

lingpri closed this Dec 11, 2019

lingpri reopened this Dec 11, 2019

lingpri reviewed Dec 11, 2019

View reviewed changes

Comment thread src/main/java/org/graylog/integrations/aws/transports/KinesisTransport.java

lingpri reviewed Dec 11, 2019

View reviewed changes

Comment thread src/main/java/org/graylog/integrations/aws/transports/KinesisTransport.java

lingpri reviewed Dec 11, 2019

View reviewed changes

Comment thread src/main/java/org/graylog/integrations/aws/transports/KinesisTransport.java

Add URL validation unit test

8f341e6

danotorrey requested a review from lingpri December 16, 2019 10:38

lingpri reviewed Dec 16, 2019

View reviewed changes

Comment thread src/main/java/org/graylog/integrations/aws/AWSClientBuilderUtil.java

lingpri reviewed Dec 16, 2019

View reviewed changes

Comment thread src/main/java/org/graylog/integrations/aws/inputs/AWSInput.java

lingpri reviewed Dec 16, 2019

View reviewed changes

Comment thread src/main/java/org/graylog/integrations/aws/AWSClientBuilderUtil.java

lingpri reviewed Dec 17, 2019

View reviewed changes

Comment thread src/web/aws/authentication/AWSCustomEndpoints.jsx

Merge branch 'master' into aws_multi_auth

91e8e11

danotorrey mentioned this pull request Dec 17, 2019

Retry when rate limit exception occurs for getStreams #342

Merged

lingpri approved these changes Dec 17, 2019

View reviewed changes

cobaltclaudia approved these changes Dec 17, 2019

View reviewed changes

cobaltclaudia merged commit c3301d8 into master Dec 17, 2019

cobaltclaudia deleted the aws_multi_auth branch December 17, 2019 15:52

bernd mentioned this pull request Jan 10, 2020

Support Interface VPC Endpoints for Kinesis Data Streams #271

Closed

Conversation

kyleknighted commented Sep 26, 2019 • edited by danotorrey Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

Uh oh!

danotorrey commented Sep 30, 2019

Uh oh!

danotorrey commented Oct 16, 2019

Uh oh!

danotorrey commented Oct 16, 2019

Uh oh!

danotorrey commented Oct 28, 2019

Uh oh!

danotorrey commented Oct 30, 2019

Uh oh!

lingpri commented Dec 11, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

danotorrey commented Dec 16, 2019

Uh oh!

danotorrey commented Dec 16, 2019

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lingpri left a comment

Choose a reason for hiding this comment

Uh oh!

cobaltclaudia Dec 10, 2019

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

kyleknighted commented Sep 26, 2019 •
edited by danotorrey

Loading

lingpri commented Dec 11, 2019 •
edited

Loading