CodeBuild - multiple input artifacts in AWS Pipeline

Question

I am building an AWS CodePipeline which depends on two source repositories. If the 'CodeBuild' step is specified with multiple Input artifacts, then you get the error: .. declares 2 input artifacts which is more than the maximum count. Documentation has a strict requirement for CodeBuild to have only a single input artifact.

Lambdas, on another hand, do not have this limitation. I've been looking for a way how a Lambda could "merge" two artifacts into a single one which I can pass to the CodeBuild then. Unfortunately, information on this subject is quite limited.

Does anyone have a working CodePipeline Lambda which would merge two Artifacts into one?

Note: Documentation about How to access artifacts.

score 6 · Answer 1 · answered Jan 08 '18 at 18:19

We've had exactly the same problem - our build stage needs to use information from two separate locations.

To solve this in a generic manner, I've created two general purpose Lambdas - one that will merge artifacts, and one that will extract specific files from an artifact. In both cases a new artifact is created that can be passed on to a subsequent CodePipeline stage.

See https://github.com/tooltwist/codepipeline-artifact-munge

If it does not meet your exact requirement, it should be easy enough to tweak to do what you need.

Awesome work, Philip, thanks for sharing this Lambda. – romaninsh Jan 26 '18 at 13:55 — romaninsh, Jan 26 '18 at 13:55

score 4 · Answer 2 · edited Jan 26 '18 at 15:06

4

I have actually solved the problem by cloning repository directly in codebuild:

Pass GitHub token:

      - { Name: GITHUB_TOKEN, Value: {Ref: GitHubToken } }
      - { Name: GITHUB_BRANCH, Value: {Ref: GitHubBranch } }

# Execute git clone in install step:

      - git clone --single-branch --depth=1 -b $GITHUB_BRANCH https://gitorgname:$GITHUB_TOKEN@github.com/gitorgname/reponame.git  src/reponame

I found that this solution is easier to implement, does not rely on Lambdas and even though performs two pulls, works quite reliably.

edited Jan 26 '18 at 15:06

Dan Cornilescu

6,730
2
19
44

answered Jan 26 '18 at 13:51

romaninsh

370
2
11

1

Where exactly do you put the - { Name ... } ? – Alex R Jun 03 '18 at 04:20
I create CodeBuild through cloudformation, but that is an environment variable which you can specify through console also. – romaninsh Jun 22 '18 at 14:12

score 3 · Accepted Answer · answered Sep 07 '18 at 09:59

As of Sep 4 2018 Amazon added support for multiple input sources:

https://aws.amazon.com/about-aws/whats-new/2018/08/aws-codebuild-adds-ability-to-create-build-projects-with-multiple-input-sources-and-output-artifacts/

From documentation:

{
"name": "sample-project",
"source": {
  "type": "S3",
  "location": "bucket/sample.zip"
},
"secondarySources": [
  {
    "type": "CODECOMMIT",
    "location": "https://git-codecommit.us-west-2.amazonaws.com/v1/repos/repo"
    "sourceIdentifier": "source1"
  },
  {
    "type": "GITHUB",
    "location": "https://github.com/awslabs/aws-codebuild-jenkins-plugin"
    "sourceIdentifier": "source2"
  }
],

version: 0.2

phases:
  build:
    commands:
      - cd $CODEBUILD_SRC_DIR_source1
      - touch file1
      - cd $CODEBUILD_SRC_DIR_source2
      - touch file2

artifacts:
  secondary-artifacts:
    artifact1:
      base-directory: $CODEBUILD_SRC_DIR_source1
      files:
        - file1
    artifact2:
      base-directory: $CODEBUILD_SRC_DIR_source2
      files:
        - file2

score -1 · Answer 4 · answered Jun 15 '18 at 05:35

-1

You can review CodeBuild. The BuildScpec file can be put in the CodeBuild project.

answered Jun 15 '18 at 05:35

darangom

1

4

I fail to see how this answer the question. – Tensibai Jun 15 '18 at 08:16

CodeBuild - multiple input artifacts in AWS Pipeline

4 Answers4