Self-Hosting a Terraform Provider Registry with AWS S3 and CloudFront
Sometimes you need to host your own Terraform Provider, and want to do it privately, without the Terraform Registry. This can be done via an S3 bucket and CloudFront, as the API just needs to respond with JSON, which can easily be done with some static JSON files. (This could most likely be done with Azure Blob Storage or GCP Cloud Storage Buckets as well).
We’ll go through:
- The Terraform to create the S3 bucket, CloudFront distributions, ACM Certificate and Route53 records.
- Build an example Terraform Provider with GoReleaser.
- Package our build to work as a registry and push it to AWS S3.
- How to consume the Terraform Provider from the Private Terraform Registry.
Code referenced in this article can be found in this GitHub repository. I started with a fork of an example provider and added what’s needed to create a registry with Terraform, and build + push the provider to it:
Create a Git Tag
Run git tag -f v0.0.1 in the root of the repository to create a Git tag on the current commit checked out that you would like to build the provider off of. GoReleaser will use the version of this tag to create a a build.
GPG code signing
Create a GPG key to used for code-signing by Go Releaser with: gpg --gen-key
This command will ask you for a name for the key, an eMail address to associate with the key, and a password for the key.
Export the key to a file as “armored ascii”, we will need this later when packaging the provider and pushing it to S3.
gpg --armor --export --output key.txt "{Key ID or email address}"
Set an environment variable for the GPG Fingerprint in CLI, which will be part of the output when you create the key. This will be used by Go Releaser.
export GPG_FINGERPRINT="[~40 character fingerprint]"
.goreleaser.yaml
Create a .goreleaser.yaml file in the root of your repository with the following content. This file uses the GPG_FINGERPRINT environment variable which is how GoReleaser knows which key to sign it with.
---
builds:
- env:
- CGO_ENABLED=0
mod_timestamp: "{{ .CommitTimestamp }}"
flags:
- -trimpath
ldflags:
- "-s -w -X main.version={{.Version}} -X main.commit={{.Commit}}"
goos:
- freebsd
- windows
- linux
- darwin
goarch:
- amd64
- "386"
- arm
- arm64
ignore:
- goos: darwin
goarch: "386"
binary: "{{ .ProjectName }}_v{{ .Version }}"
archives:
- format: zip
name_template: "{{ .ProjectName }}_{{ .Version }}_{{ .Os }}_{{ .Arch }}"
checksum:
name_template: "{{ .ProjectName }}_{{ .Version }}_SHA256SUMS"
algorithm: sha256
signs:
- artifacts: checksum
args:
- "--batch"
- "--local-user"
- "{{ .Env.GPG_FINGERPRINT }}"
- "--output"
- "${signature}"
- "--detach-sign"
- "${artifact}"Build locally with goreleaser release --skip-publish , which will create a dist directory. Go Releaser will ask you for the GPG key password while building, so that it can do the code-signing step properly.
Note: You don’t have to skip the publish step, set a GitHub token in the CLI for Go Releaser to also push the build files to GitHub as part of a release.
Creating AWS Resources
We’ll create the following AWS resources to host the Terraform provider:
- S3 bucket to store the provider build files.
- CloudFront distribution to add a domain name.
- ACM Certificate for SSL on the CF distribution.
- Route53 records (assumed that host zone already exists).
Change the local values to fit your needs, and run terraform init and terraform apply to create the resources. Make sure you have AWS credentials set either in a ~/.aws directory, or the current terminal session via environment variables. The bucket is set to public so that all files can be reached through the CloudFront distribution.
provider "aws" {
region = "us-east-1"
}
locals {
name = "TF Provider"
domain_name = "cjscloud.city"
subdomain_name = "tfp.${local.domain_name}"
environment = "default"
}
# S3 bucket
resource "aws_s3_bucket" "provider" {
bucket = local.subdomain_name
tags = {
Name = local.name
Environment = local.environment
}
}
resource "aws_s3_bucket_public_access_block" "provider" {
bucket = aws_s3_bucket.provider.id
block_public_acls = false
block_public_policy = false
ignore_public_acls = false
restrict_public_buckets = false
}
resource "aws_s3_bucket_policy" "provider" {
bucket = aws_s3_bucket.provider.id
policy = data.aws_iam_policy_document.provider.json
}
data "aws_iam_policy_document" "provider" {
statement {
principals {
type = "*"
identifiers = ["*"]
}
actions = [
"s3:GetObject",
]
resources = [
"${aws_s3_bucket.provider.arn}/*",
]
}
}
# ACM Certificate
resource "aws_acm_certificate" "provider" {
domain_name = local.subdomain_name
validation_method = "DNS"
tags = {
Name = local.name
Environment = local.environment
}
lifecycle {
create_before_destroy = true
}
}
# Route53 Records
data "aws_route53_zone" "provider" {
name = local.domain_name
private_zone = false
}
resource "aws_route53_record" "provider" {
for_each = {
for dvo in aws_acm_certificate.provider.domain_validation_options : dvo.domain_name => {
name = dvo.resource_record_name
record = dvo.resource_record_value
type = dvo.resource_record_type
}
}
allow_overwrite = true
name = each.value.name
records = [each.value.record]
ttl = 60
type = each.value.type
zone_id = data.aws_route53_zone.provider.zone_id
}
resource "aws_route53_record" "www" {
zone_id = data.aws_route53_zone.provider.zone_id
name = local.subdomain_name
type = "CNAME"
ttl = 300
records = [aws_cloudfront_distribution.provider.domain_name]
}
# Cert DNS validation, depended on by CloudFront
resource "aws_acm_certificate_validation" "provider" {
certificate_arn = aws_acm_certificate.provider.arn
validation_record_fqdns = [for record in aws_route53_record.provider : record.fqdn]
}
# Cloudfront
resource "aws_cloudfront_distribution" "provider" {
origin {
domain_name = aws_s3_bucket.provider.bucket_regional_domain_name
origin_id = aws_s3_bucket.provider.bucket_domain_name
}
enabled = true
is_ipv6_enabled = true
comment = "Terraform Provider Registry"
default_root_object = "index.html"
aliases = [local.subdomain_name]
default_cache_behavior {
allowed_methods = ["GET", "HEAD", "OPTIONS"]
cached_methods = ["GET", "HEAD"]
target_origin_id = aws_s3_bucket.provider.bucket_domain_name
forwarded_values {
query_string = false
cookies {
forward = "none"
}
}
viewer_protocol_policy = "allow-all"
min_ttl = 0
default_ttl = 3600
max_ttl = 86400
}
price_class = "PriceClass_100"
restrictions {
geo_restriction {
# restriction_type = "whitelist"
# locations = ["NZ"]
restriction_type = "none"
}
}
tags = {
Name = local.name
Environment = local.environment
}
viewer_certificate {
cloudfront_default_certificate = false
acm_certificate_arn = aws_acm_certificate.provider.arn
ssl_support_method = "sni-only"
}
}How to package the Terraform provider
The following Go code can package the build in the dist/ directory and create the necessary files for a Terraform provider registry hosted as static files. In the example repo, this script is under a folder named tfpp/ and by default assumes there is a dist/ folder and key.txt file in the parent directory, but different paths can be provided via command line flags.
An example of calling this would be like:
go run main.go \
-ns=cjh-cloud \
-d=tfp.cjscloud.city \
-gf=$GPG_FINGERPRINT \
-v=0.0.1main.go
Here’s the Go script that will package a Terraform provider build to act as a Terraform Registry, hosted statically. This is in the GitHub repo mentioned at the start as in the tfpp/ (Terraform Provider Packager) directory.
package main
import (
"bufio"
"flag"
"fmt"
"io"
"log"
"os"
"strings"
)
func main() {
log.Println("📦 Packaging Terraform Provider for private registry...")
namespace := flag.String("ns", "", "Namespace for the Terraform registry.") // No default
domain := flag.String("d", "", "Private Terraform registry domain.") // No default
providerName := flag.String("p", "mock", "Name of the Terraform provider.")
distPath := flag.String("dp", "../dist", "Path to Go Releaser build files.")
repoName := flag.String("r", "terraform-provider-mock", "Name of the provider repository used in Go Releaser build name.")
version := flag.String("v", "0.0.1", "Semantic version of build.")
gpgFingerprint := flag.String("gf", "", "GPG Fingerprint of key used by Go Releaser") // No default
gpgPubKeyFile := flag.String("gk", "../key.txt", "Path to GPG Public Key in ASCII Armor format.")
flag.Parse()
*distPath = *distPath + "/" // If the path already ends in "/", it shouldn't matter
// Create release dir - only the contents of this need to be uploaded to S3
err := createDir("release")
if err != nil {
log.Printf("Error creating 'release' dir: %s", err)
}
// Create .wellKnown directory and terraform.json file
err = wellKnown()
if err != nil {
log.Printf("Error creating '.wellKnown' dir: %s", err)
}
// Create v1 directory
err = provider(*namespace, *providerName, *distPath, *repoName, *version, *gpgFingerprint, *gpgPubKeyFile, *domain)
if err != nil {
log.Printf("Error creating 'v1' dir: %s", err)
}
log.Println("📦 Packaged Terraform Provider for private registry.")
}
// This establishes the "API" as a TF provider by responding with the correct JSON payload, by using static files
func wellKnown() (error) {
log.Println("* Creating .well-known directory")
err := createDir("release/.well-known")
if err != nil {
return err
}
terraformJson := []byte(`{"providers.v1": "/v1/providers/"}`)
log.Println(" - Writing to .well-known/terraform.json file")
err = writeFile("release/.well-known/terraform.json", terraformJson)
if err != nil {
return err
}
return nil
}
// provider is the Terraform name
// repoName is the Repository name
func provider(namespace, provider, distPath, repoName, version, gpgFingerprint, gpgPubKeyFile, domain string) (error) {
// Path to semantic version dir
versionPath := providerDirs(namespace, provider, version)
// Files to create under v1/providers/[namespace]/[provider_name]
createVersionsFile(namespace, provider, distPath, repoName, version) // Creates version file one above download, which is why downloadPath isn't used
// Files/Directories to create under v1/providers/[namespace]/[provider_name]/[version]
copyShaFiles(versionPath, distPath, repoName, version)
downloadPath, err := createDownloadsDir(versionPath)
if err != nil {
return err
}
// Create darwin, freebsd, linux, windows dirs
createTargetDirs(*downloadPath)
// Copy all zips
copyBuildZips(*downloadPath, distPath, repoName, version)
// Create all individual files for build targets and each architecture for the build targets
createArchitectureFiles(namespace, provider, distPath, repoName, version, gpgFingerprint, gpgPubKeyFile, domain)
return nil
}
// Create the directories with a path format v1/providers/[namespace]/[provider_name]/[version]
func providerDirs(namespace, repoName, version string) string {
log.Println("* Creating release/v1/providers/[namespace]/[repo]/[version] directories")
providerPathArr := [6]string{"release", "v1", "providers", namespace, repoName, version}
var currentPath string
for _, v := range providerPathArr {
currentPath = currentPath + v + "/"
createDir(currentPath)
}
return currentPath
}
// Create the versions file under v1/providers/[namespace]/[provider_name]
func createVersionsFile(namespace, provider, distPath, repoName, version string) (error) {
log.Println("* Writing to release/v1/providers/[namespace]/[repo]/versions file")
versionPath := fmt.Sprintf("release/v1/providers/%s/%s/versions", namespace, provider)
shaSumContents, err := getShaSumContents(distPath, repoName, version)
if err != nil {
return err
}
// Build the versions file...
platforms := ""
for _, line := range shaSumContents {
fileName := line[1] // zip file name
// get os and arch from filename
removeFileExtension := strings.Split(fileName, ".zip")
fileNameSplit := strings.Split(removeFileExtension[0], "_")
// Get build target and architecture from the zip file name
target := fileNameSplit[2]
arch := fileNameSplit[3]
platforms += "{"
platforms += fmt.Sprintf(`"os": "%s",`, target)
platforms += fmt.Sprintf(`"arch": "%s"`, arch)
platforms += "}"
platforms += ","
}
platforms = strings.TrimRight(platforms, ",") // remove trailing comma, json does not allow
var versions = []byte(fmt.Sprintf(`
{
"versions": [
{
"version": "%s",
"protocols": [
"4.0",
"5.1"
],
"platform": [
%s
]
}
]
}
`, version, platforms))
writeFile(versionPath, versions)
return nil
}
func copyShaFiles(destPath, srcPath, repoName, version string) {
log.Printf("* Copying SHA files in %s directory", srcPath)
// Copy files from srcPath
shaSum := repoName + "_" + version + "_SHA256SUMS"
shaSumPath := srcPath + "/" + shaSum
// _SHA256SUMS file
_, err := copyFile(shaSumPath, destPath + shaSum)
if err != nil {
log.Println(err)
}
// _SHA256SUMS.sig file
_, err = copyFile(shaSumPath + ".sig", destPath + shaSum + ".sig")
if err != nil {
log.Println(err)
}
}
func createDownloadsDir(destPath string) (*string, error) {
log.Printf("* Creating download/ in %s directory", destPath)
downloadPath := destPath + "download/"
err := createDir(downloadPath)
if err != nil {
return nil, err
}
return &downloadPath, nil
}
func createTargetDirs(destPath string) (error) {
log.Printf("* Creating target dirs in %s directory", destPath)
targets := [4]string{"darwin", "freebsd", "linux", "windows"}
for _, v := range targets {
err := createDir(destPath + v)
if err != nil {
return err
}
}
return nil
}
func copyBuildZips(destPath, distPath, repoName, version string) (error) {
log.Println("* Copying build zips")
shaSumContents, err := getShaSumContents(distPath, repoName, version)
if err != nil {
return err
}
// Loop through and copy each
for _, v := range shaSumContents {
zipName := v[1]
zipSrcPath := distPath + zipName
zipDestPath := destPath + zipName
log.Printf(" - Zip Source: %s", zipSrcPath)
log.Printf(" - Zip Dest: %s", zipDestPath)
// Copy the zip
_, err := copyFile(zipSrcPath, zipDestPath)
if err != nil {
return err
}
}
return nil
}
func getShaSumContents(distPath, repoName, version string) ([][]string, error) {
shaSumFileName := repoName + "_" + version + "_SHA256SUMS"
shaSumPath := distPath + "/" + shaSumFileName
shaSumLine, err := readFile(shaSumPath)
if err != nil {
return nil, err
}
buildsAndShaSums := [][]string{}
for _, line := range shaSumLine {
lineSplit := strings.Split(line, " ")
row := []string{lineSplit[0], lineSplit[1]}
buildsAndShaSums = append(buildsAndShaSums, row)
}
// log.Println(buildsAndShaSums)
return buildsAndShaSums, nil
}
// Create architecture files for each build target
func createArchitectureFiles(namespace, provider, distPath, repoName, version, gpgFingerprint, gpgPubKeyFile, domain string) (error) {
log.Println("* Creating architecure files in target directories")
// filename = terraform-provider-[provider]_0.0.1_darwin_amd64.zip - provider_name + version + target + architecture + .zip
prefix := fmt.Sprintf("v1/providers/%s/%s/%s/", namespace, provider, version)
pathPrefix := fmt.Sprintf("release/%s", prefix)
urlPrefix := fmt.Sprintf("https://%s/%s", domain, prefix)
// download url = https://example.com/v1/providers/namespace/provider/0.0.1/download/terraform-provider_0.0.1_darwin_amd64.zip
downloadUrlPrefix := urlPrefix + "download/"
downloadPathPrefix := pathPrefix + "download/"
// shasums url = https://example.com/v1/providers/namespace/provider/0.0.1/terraform-provider_0.0.1_SHA256SUMS
shasumsUrl := urlPrefix + fmt.Sprintf("%s_%s_SHA256SUMS", repoName, version)
// shasums_signature_url = https://example.com/v1/providers/namespace/provider/0.0.1/terraform-provider_0.0.1_SHA256SUMS.sig
shasumsSigUrl := shasumsUrl + ".sig"
shaSumContents, err := getShaSumContents(distPath, repoName, version)
if err != nil {
return err
}
// Get contents of GPG key
gpgFile, err := readFile(gpgPubKeyFile)
if err != nil {
log.Printf("Error reading '%s' file: %s", gpgPubKeyFile, err)
}
// loop through every line and stick with \\n
gpgAsciiPub := ""
for _, line := range gpgFile {
gpgAsciiPub = gpgAsciiPub + line + "\\n"
}
// log.Println(gpgAsciiPub)
for _, line := range shaSumContents {
shasum := line[0] // shasum of the zip
fileName := line[1] // zip file name
downloadUrl := downloadUrlPrefix + fileName
// get os and arch from filename
removeFileExtension := strings.Split(fileName, ".zip")
fileNameSplit := strings.Split(removeFileExtension[0], "_")
// Get build target and architecture from the zip file name
target := fileNameSplit[2]
arch := fileNameSplit[3]
// build filepath
archFileName := downloadPathPrefix + target + "/" + arch
var architectureTemplate = []byte(fmt.Sprintf(`
{
"protocols": [
"4.0",
"5.1"
],
"os": "%s",
"arch": "%s",
"filename": "%s",
"download_url": "%s",
"shasums_url": "%s",
"shasums_signature_url": "%s",
"shasum": "%s",
"signing_keys": {
"gpg_public_keys": [
{
"key_id": "%s",
"ascii_armor": "%s",
"trust_signature": "",
"source": "",
"source_url": ""
}
]
}
}
`, target, arch, fileName, downloadUrl, shasumsUrl, shasumsSigUrl, shasum, gpgFingerprint, gpgAsciiPub))
log.Printf(" - Arch file: %s", archFileName)
err := writeFile(archFileName, architectureTemplate)
if err != nil {
return err
}
}
return nil
}
func createDir(path string) (error) {
err := os.Mkdir(path, os.ModePerm)
return err
}
func copyFile(src, dst string) (int64, error) {
sourceFileStat, err := os.Stat(src)
if err != nil {
return 0, err
}
if !sourceFileStat.Mode().IsRegular() {
return 0, fmt.Errorf("%s is not a regular file", src)
}
source, err := os.Open(src)
if err != nil {
return 0, err
}
defer source.Close()
destination, err := os.Create(dst)
if err != nil {
return 0, err
}
defer destination.Close()
nBytes, err := io.Copy(destination, source)
return nBytes, err
}
func readFile(filePath string) ([]string, error) {
readFile, err := os.Open(filePath)
if err != nil {
return nil, err
}
fileScanner := bufio.NewScanner(readFile)
fileScanner.Split(bufio.ScanLines)
var fileLines []string
for fileScanner.Scan() {
fileLines = append(fileLines, fileScanner.Text())
}
readFile.Close()
return fileLines, nil
}
func writeFile(fileName string, fileContents []byte) (error) {
err := os.WriteFile(fileName, fileContents, 0644)
return err
}This will create a release/ directory, which can be pushed to the S3 bucket with something like aws s3 sync release s3://tfp.cjscloud.city. The directory structure of the release will look something like in the following image.
Using the provider
Define the provider and set the source and version under required_providers, as well as defining the provider object.
terraform {
required_providers {
mock = {
source = "tfp.cjscloud.city/cjh-cloud/mock"
version = "0.0.1"
}
}
}
provider "mock" {
foo = "test_foo_value"
}
resource "mock_example" "testing" {
not_computed_required = "some value"
dynamic "foo" {
for_each = [{ number = 1 }, { number = 2 }, { number = 3 }]
content {
bar {
number = foo.value.number
}
}
}
/*
* The above is equivalent to:
*
* foo {
* bar {
* number = 1
* }
* }
* foo {
* bar {
* number = 2
* }
* }
* foo {
* bar {
* number = 3
* }
* }
*/
dynamic "baz" {
// The variable inside the for_each block doesn't have to be the same as
// what you're assigning the value to.
for_each = [{ something = "x" }, { something = "y" }, { something = "z" }]
content {
qux = baz.value.something
}
}
/*
* The above is equivalent to:
*
* baz {
* qux = "x"
* }
* baz {
* qux = "y"
* }
* baz {
* qux = "z"
* }
*/
some_list = ["a", "b", "c"]
}
output "last_updated" {
value = mock_example.testing.last_updated
}Run terraform init to download the provider from the private registry, which should successfully initialise. Then run terraform apply to create a resource using the provider.
Cleanup
Empty the S3 bucket of files, then run terraform destroy to remove all the AWS resources when tearing down the private registry.
If you found this useful, please consider following!
Cheers,
