· engineering · 1 min read
Triggering Gitlab pipelines from AWS
Automatically trigger a GitLab pipeline when content changes in an S3 bucket
Recently, while implementing a Gatsby site that renders content from a JSON file stored in an S3 bucket, I faced the challenge of needing to trigger a GitLab pipeline whenever this content changed.
The solution
The solution appears straightforward: we need a deploy token, which can be obtained and used by following these steps:
- In your GitLab project, navigate to Settings > CI/CD.
- Go to ‘Pipeline Trigger Tokens’ and create a new token.
- Follow the instructions to configure your webhook.
Challenges Encountered.
Despite following the instructions, I consistently received a 404 error when attempting to use the webhook, though using the POST method worked correctly.
Debouncing the pipeline
Given that the content in the bucket is dynamic and files are often updated in quick succession, it is necessary to debounce the pipeline to avoid redundant builds:
- When a file is changed, a ‘rebuild required’ marker is also added to the root directory.
- Using a scheduled EventBridge rule, we configure a Lambda function to run at set intervals (e.g., hourly).
- This Lambda function checks for the presence of the ‘rebuild required’ marker, and if found, it triggers the deployment pipeline.
- Finally, the GitLab pipeline is configured to delete the marker once the build completes.
About James Babington
A cloud architect and engineer with a wealth of experience across AWS, web development, and security, James enjoys writing about the technical challenges and solutions he's encountered, but most of all he loves it when a plan comes together and it all just works.
No comments yet. Be the first to comment!