Integrating GCP Secret Manager with NestJS Config
The NestJS Config module takes care of mapping your config.yaml to a neatly packed set of strings or objects that are ready for your app to consume so you don’t have to put environment-specific values directly in your code. However, it leaves you to fend for yourself when it comes to managing secrets. Let’s take a look at how to make the default NestJS Config module play nicely with GCP Secret Manager.
Storing and Recalling Config Values
If you, for example, have an external API dependency that is environment-specific (perhaps staging vs prod or even local dev), then it is entirely reasonable to have the following section in your `config.yaml` file:
# So far, so good
example_api: https://api.example.comSo now in your code, when you are ready to make that fetch call, you can grab the API url from your config. Your code does not need to know where it is running…so no more:
// This is bad
if (environment === 'prod') {
doThis();
} else {
doThat();
}Since your `config.yaml` represents the configuration values for the current environment, you can just take the value it gives you:
// This is good
const apiUrl = this.configService.get<string>('example_api');What About Secrets? (Passwords/Keys/Etc)
It is very likely that this (or any) API will also require some sort of credential for logging in. So just put the password/key in your config.yaml file?
# This is extremely bad
example_api: https://api.example.com
example_api_key: this_will_end_badlyThis is where Google Cloud Secret Manager saves the day. It lets you securely store your secret information and access it via their API using a GCP Service Account.
Note: The Service Account is typically a JSON file stored locally on the server and exposed to the runtime Node application with a GOOGLE_APPLICATION_CREDENTIALS environment var that points to the JSON file. Connecting your application using a GCP Service Account is not the point of this particular tutorial.
By using GCP Secret Manager, you can instead put the pointer to your current (or “latest”) version of the secret:
# This is better
example_api: https://api.example.com
# Just the latest version
example_api_key: projects/9999999/secrets/example-api-key/versions/latest
# Grab all active versions
example_api_keys: projects/9999999/secrets/example-api-keyNow, when you are once again ready to construct that fetch call, you can do the following:
// This is not bad
const apiUrl = this.configService.get<string>('example_api');
const apiKeyVersion = this.configService.get<string>('example_api_key');
const [accessResponse] =
await secretManagerServiceClient.accessSecretVersion({
name: apiKeyVersion,
});
const apiKeyValue = accessResponse.payload.data.toString();So…this gets us closer. We have managed to remove sensitive information from our config.yaml, however, there are still some gotchas.
— Doing this requires that you:
- access an API at request time (unless you implement caching elsewhere)
- know exactly which config keys are externally managed secrets
- implement special logic to circumvent the Secret Manager for local development where secrets can safely be stored locally (assuming they are never checked in)
We don’t want our app to have to know which config keys are backed by an externally managed secret and we want to request (and cache) the secrets from the Secret Manager API once and only once. To do that, we need to grab the secrets at startup while NestJS is loading the dependency chain.
Asynchronous Configuration
The first hurdle is the fact that the NestJS Config module is loaded synchronously by default. However, the call to Google Secret Manager is asynchronous (which requires an await — stop here and hit up StackOverflow if that causes you any anxiety).
That means we need to use a NestJS provider factory to load our config.
configuration.module.ts
import { Module } from '@nestjs/common';
import { ConfigService } from '@nestjs/config';
import configuration from './configuration';
@Module({
controllers: [],
providers: [
{
provide: ConfigService,
useFactory: configuration,
},
],
exports: [ConfigService],
})
export class ConfigurationModule {}By using the factory, NestJS knows to wait for the promise from the configuration function. It is in that function where we read from our config.yaml and also dynamically fetch our secrets.
Rewriting the Config Values On Load
The file below is where we load our config values. However, just before we send the Record object to the `ConfigService` constructor, we modify it by reference to swap out all Google Secret Manager versions with their corresponding secret contents.
import { SecretManagerServiceClient } from '@google-cloud/secret-manager';
import { Logger } from '@nestjs/common';
import { ConfigService } from '@nestjs/config';
import { readFileSync, existsSync } from 'fs';
import * as yaml from 'js-yaml';
import { isString, merge } from 'lodash';
import { join } from 'path';
import * as Bluebird from 'bluebird';
const logger = new Logger('XperienceConfigLoader');
const isSecretLocator = /^projects\/(?:\d{5,16}|.*)\/secrets\/.*$/;
const isSecretVersionLocator = /^projects\/(?:\d{5,16}|.*)\/secrets\/.*\/versions\/(?:\d{1,4}|latest)$/;
export interface XperienceConfigLoaderOptions {
baseDirectory: string;
configFilename: string;
localConfigFilename?: string;
secretManagerServiceClient?: SecretManagerServiceClient;
}
const rewriteRecordWithSecrets = async (records: any, level: string = '', secretManagerServiceClient?: SecretManagerServiceClient): Promise<any> => {
for (const key in records) {
if (isString(records[key])) {
if (secretManagerServiceClient && isSecretLocator.test(records[key])) {
try {
// Do we want a specific version or grab all active ones as an array?
if (isSecretVersionLocator.test(records[key])) {
const [accessResponse] = await secretManagerServiceClient.accessSecretVersion({
name: records[key],
});
records[key] = accessResponse.payload.data.toString();
} else {
const [versions] = await secretManagerServiceClient.listSecretVersions({ parent: records[key] });
const secrets: string[] = new Array();
await Bluebird.Promise.map(versions, async (version) => {
if (version.state === 'ENABLED' && isSecretVersionLocator.test(version.name)) {
const [accessResponse] = await secretManagerServiceClient.accessSecretVersion({
name: version.name,
});
secrets.push(accessResponse.payload.data.toString());
}
}, { concurrency: 1 });
records[key] = secrets;
}
logger.log(`Loaded secret from Google Secret Manager [${level}.${key}]`);
} catch (e) {
console.log(JSON.stringify(e));
logger.warn(`Failed to load secret from Google Secret Manager [${level}.${key}]`);
}
}
} else {
await rewriteRecordWithSecrets(records[key], `${level}.${key}`, secretManagerServiceClient);
}
}
};
export const XperienceConfigLoader = async (options: XperienceConfigLoaderOptions) => {
let cfg = yaml.load(
readFileSync(join(options.baseDirectory, options.configFilename), 'utf8'),
) as Record<string, any>;
if (!!options.localConfigFilename && existsSync(join(options.baseDirectory, options.localConfigFilename))) {
// Looks like we have a local settings file with which we
// want to override the prod values
const local = yaml.load(
readFileSync(join(options.baseDirectory, options.localConfigFilename), 'utf8'),
) as Record<string, any>;
cfg = merge(cfg, local);
}
await rewriteRecordWithSecrets(cfg, undefined, options?.secretManagerServiceClient);
return new ConfigService(cfg);
}First, lets take a look at the XperienceConfigLoader function at the bottom of the file. There are a few things happening here:
- Loads a base config.yaml file that it expects to always be there — if it is not there, bad things happen and the app will not run.
- Checks to see if there is another config called `config.local.yaml` that will override any matching settings from the base file (useful for local dev where you don’t always want to use “prod” values for stuff)
- Merges the two Record objects together, giving preference to the `local` values
- Rewrites the Record object with secrets
Since the configuration is in yaml, settings can be nested multiple levels deep. That means we are going to call our rewriteRecordWithSecrets function recursively as it iterates over each key and checks to see if it is the tip of a branch (in our case, a string) or if it is another Record that needs to be crawled further.
Note: This function is written somewhat verbosely. I’m sure there are quite a few clever devs who can reduce it to a one-liner (one thing I never liked about hard-core Ruby devs and lodash aficionados). However, as much as code should be tight and clean, it should also be clear and self-documenting. Humans maintain code (for now), so always aim to make your code readable with the intent as clear as possible.
So, when we finally do encounter the tip of a branch, we check to see if that string happens to be in the form of a Google Secret Manager version key. If it’s a match, we look up the secret and swap out the value. If not, we just move on leaving well enough alone. We also check to see if we need to grab the entire secret versions array or just a specific one. That will determine if your resulting config element is a string or an array.
This should be fine since you are stipulating in the config.yaml which you expect by virtue of how you format your locator string.
Recalling Secret Config Values
Now that all of the secrets have been fetched and swapped out in the config object, we can grab them from the config as if they had been there from the start.
// const apiKeyVersion = this.configService.get<string>('example_api_key');
// No longer need this bit
// const [accessResponse] =
// await secretManagerServiceClient.accessSecretVersion({
// name: apiKeyVersion,
// });
// const apiKeyValue = accessResponse.payload.data.toString();
// **This** is what we want
const apiKeyValue = this.configService.get<string>('example_api_key');The final value of the secret is available direct from the ConfigService. The config is cached and synchronously always available. You will need to be mindful of when you roll keys that you will need to reload the service, since fetching keys happens on startup.
