#terraform #coding #devOps

Recently I have been trying to setup a project that runs on GCP with resources managed via Terraform. For Reasons ($) I wanted to try doing the build of my docker image and Terraform plan/apply on the GitHub side of things. This causes a few chicken/egg problems. Those can be dealt with by layering up the resources when you start (there is probably a better way, but this worked for me) however there was one thing in the code that was really bugging me. I’m not a DRY above all else kind of programmer, but I was quickly running into an issue where each env var would need to be input by hand in the action workflow file, in addition to the local .tfvars files I was already using. I do not like making the same changes in more than one place, so I set out for a solution!

Wait, what?

each env var would need to be input by hand in the action workflow file

OK so this might not be true depending on what you’re doing, BUT consider this: I want to have at least two environments. To separate the GitHub Secrets, I prefixed them with the environment. Sounds simple! But you can NOT use the ${{ format() }} syntax in the env block of a job! What does that mean for us? Well, if you want to dynamically get DEV_VAR, you’ll need some eyesore like ${{ secrets[format('{0}_VAR', steps.env.outputs.env )] } , with a separate job for setting the env. That env output is set based on the branch name which is straightforward unlike the rest of this: echo "env=${{ github.ref_name == 'main' && 'prod' || 'dev' }}" >> "$GITHUB_OUTPUT"

Now we can get DEV_VAR or PROD_VAR based on the branch name. Before we get into the solution for passing these vars cleanly into terraform, let’s take a step back and see how they tell you to do it on The Internet™️.

The way everyone tells you to do it on the internet:

If we take a look at the Terraform docs on input variables, we can see there are a few ways to get our secrets:

1. Terraform cloud umm, no thanks!
2. the -var flag on the cli (this is what I was doing and it is not maintainable or scaleable without a script to do it for you but that involves fancy filtering or allowlists of secrets)
3. A file with the vars defined, like *.auto.tfvars or some json file or something either automatically found by terraform when running plan/apply, or passed in with -var-file
4. Environment variables that are your vars prefixed with TF_VAR

If you check StackOverflow, number 4 seems to be the recommended option. This makes a lot of sense because we can use the secrets context we get from the GitHub actions and use that as a value for the env block.

*record scratch*

But we can’t dynamically get our vars from the env block!

Another wrinkle: What if your vars are maps or objects? I like using objects because I can group variables by logical definitions like for instance my CMS vars can all live in one object called CMS_VARS.

(Maybe this isn’t the best way to do it, but I like this, because I can use the github_action_secret resource to set the secrets in GitHub based on my local (and gitignored!) .tfvars files!! Plus, I can set the env in those files so I can have dev and prod locally defined but separate. Again, I’m not sure if this is the best way to do things but it seems to be working for me so far. I am open to hearing alternatives, especially if there are security issues with this setup!)

So if you’re setting jsonencoded secrets in github, how do we get them back out for terraform to read them properly?

JSON, my only friend

Back to the list of accepted var inputs: 2 we tried and it’s kind of verbose. Not to mention when using an object, I could not figure out how to get it to parse correctly. Surely there is a way, but I was trying to move away from this due to the verbosity. 4 is also out since it’s about the same as 2 (I know because I tried!). That leaves us with 3 (Let’s be real, we were not ever considering 1 for this exercise). There could be negative security implications with this one, however I kind of go off the assumption that if somebody is reading this file I’m creating in a private repo CI runner, I have bigger problems!

So, in order to get our vars into terraform plan/apply, we can construct a json file with a simple heredoc. Here’s an example of the jobs for this:

      - name: get env
        id: env
        run: echo "env=${{ github.ref_name == 'main' && 'prod' || 'dev' }}" >> "$GITHUB_OUTPUT"

      - name: setup vars
        id: vars
        run: |
          cat << EOF >> ./tfvars.json
          {
            "my_vars_object": ${{ secrets[format('{0}_MY_VARS_OBJECT', steps.env.outputs.env )] }},
            "my_other_vars_object": ${{ secrets[format('{0}_MY_OTHER_VARS_OBJECT', steps.env.outputs.env )] }},
            "just_a_string": "${{ steps.env.outputs.env }}"
          }
          EOF

What’s going on here? We already sort of looked at get env. It simply echos “dev” or “prod” to the $GITHUB_OUTPUT so we can use it in the next step. (This could also be set as a env var with $GITHUB_ENV instead of $GITHUB_OUTPUT which may be a bit less verbose than steps.env.outputs.env. ) Next, we construct our json file with the heredoc syntax, piping the output to ./tfvars.json. The name of this file can be anything as long as you pass it into -var-file when you run plan/apply

Notice the double quotes around the just_a_string, where there are no quotes around the jsonencoded secrets. It is important that you jsonencode the secret in terraform or else it won’t work.

Wait, this isn’t any DRYer than the first solution

Well, not really. You will still have to update this file whenever a new object var is added. So there are still 4ish places to edit when adding a new variable: variables.tf, local tfvars, github.tf (my module for GitHub secrets etc), and the workflow yaml. There is probably a way to automate the secrets creation based on a for_each or something, but my terraform-fu is still quite green and I haven’t gotten to that optimization yet (maybe another day). This DOES allow me to add vars to the existing objects without touching the workflow files or any other terraform file outside of the tfvars which I think is a big win!

Closing thoughts

This took a lot more effort than I had hoped it would, which is why I’m sharing it to hopefully help out someone in the future. Please let me know if any of these methods are insecure and I will update the post, but I believe this is all Kosher.