Hopefully this is the right place to ask this question. I’m upgrading our project’s developer experience and part of that is removing the need to manually create Doppler service tokens before being able to spin up a local environment. I got that successfully working by modifying our Makefile to generate ephemeral service tokens like this:
And make up will work as expected. The problem is if I am not connected to a network. In that scenario, I first need to fallback DOPPLER_TOKEN to the “use_fallback_file” variable as an empty string would throw an error. However, this will still produce the “Doppler Error: The fallback file does not exist” error.
My question is simple: How do I get this to work in offline mode? With static environment variable, it just worked. Thank you in advance!
This is pretty odd. The fallback file is encrypted and must be decrypted using the same access token it was created using unless you used the --passphrase flag to set a passphrase on the fallback file (which then allows you to decrypt it using another token so long as you pass in the correct passphrase). If you were running into that problem though you’d get something like this:
Unable to fetch secrets from the Doppler API
Doppler Error: Invalid Auth token
Reading secrets from fallback file
Unable to decrypt fallback file
Doppler Error: cipher: message authentication failed
Since you’re getting a “fallback file does not exist” error – I suspect the issue might be that it’s being stored to an ephemeral disk location. By default, all fallback files are stored in ~/.doppler/fallback. Could you check to make sure that’s a valid location on your image that’s persisted? In the latest CLI version you can specify --config-dir to change the ~/.doppler location to an arbitrary directory if that’s helpful. You can also generate a fallback file by hand using doppler secrets download (which also takes a --passphrase flag if you like) and can then specify the location it’s stored at using --fallback <file-path>.
Hope this helps! Let me know if you still need help after reviewing this!
I can confirm the fallback files are located in the correct folder. It makes sense my approach wouldn’t work since the tokens are always rotated, which leaves me with the passphrase option – I believe that’s what I’m looking for.
However, I haven’t been able to get this working just yet. On a positive note, I am now running into the decipher message you’ve shared, so I’m halfway there.
Here’s the command I am trying to run. It’s defined in my Makefile and executed as make up.
-include .env
DOPPLER_TOKEN ?= $(shell doppler configs tokens create dev --plain --max-age 1m)
.PHONY: up
up:
DOPPLER_TOKEN="$(DOPPLER_TOKEN)" \
doppler run --command="docker compose up --remove-orphans" --passphrase dev
Here’s what gets printed in the console when I am online.
(base) ➜ api ✗ make up
DOPPLER_TOKEN="<ephemeral_token>" \
doppler run --command="docker compose up --remove-orphans" --passphrase dev
[+] Running 2/2
⠿ Container api-db-1 Created 0.0s
⠿ Container api-app-1 Recreated 0.2s
Attaching to api-app-1, api-db-1
api-db-1 |
api-db-1 | PostgreSQL Database directory appears to contain a database; Skipping initialization
api-db-1 |
api-db-1 | 2022-11-16 01:08:35.818 UTC [1] LOG: starting PostgreSQL 14.6 (Debian 14.6-1.pgdg110+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit
api-db-1 | 2022-11-16 01:08:35.819 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
api-db-1 | 2022-11-16 01:08:35.819 UTC [1] LOG: listening on IPv6 address "::", port 5432
api-db-1 | 2022-11-16 01:08:35.822 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
api-db-1 | 2022-11-16 01:08:35.826 UTC [26] LOG: database system was shut down at 2022-11-16 01:08:06 UTC
api-db-1 | 2022-11-16 01:08:35.832 UTC [1] LOG: database system is ready to accept connections
All looks good. However, here’s the same thing when I am offline.
(base) ➜ api ✗ make up
Unable to create service token
Doppler Error: Post "https://api.doppler.com/v3/configs/config/tokens?config=<config>&project=<project>": dial tcp: lookup api.doppler.com on [::1]:53: read udp [::1]:63945->[::1]:53: read: connection refused
DOPPLER_TOKEN="" \
doppler run --command="docker compose up --remove-orphans" --passphrase dev
Unable to fetch secrets from the Doppler API
Doppler Error: Get "https://api.doppler.com/v3/configs/config/secrets/download?config=<config>&format=json&include_dynamic_secrets=true&project=<project>": dial tcp: lookup api.doppler.com on [::1]:53: read udp [::1]:55468->[::1]:53: read: connection refused
Reading secrets from fallback file
Unable to decrypt fallback file
Doppler Error: cipher: message authentication failed
=== More Info ===
Why did decryption fail?
The most common cause of decryption failure is using an incorrect passphrase.
The default passphrase is computed using your token, project, and config.
You must use the same token, project, and config that you used when saving the backup file.
What should I do now?
Ensure you are using the same scope that you used when creating the fallback file.
Alternatively, manually specify your configuration using the appropriate flags (e.g. --project).
Run 'doppler run --help' for more info.
make: *** [up] Error 1
You may notice above my token gets set to an empty string since the tokens create command fails. I’ve tried to remove that line completely, but the error didn’t change. In any case, I won’t have access to the original token, which seems logical to me. The question is, am I encrypting/decrypting the fallback file correctly? if I am, is Doppler trying to use the correct fallback file? (latest I assume)
Do you see what I am doing wrong here? Thank you in advance
EDIT: I suspect the issue is with the tokens create command as I do not define anywhere how to encrypt the file? But that command doesn’t accept either passphrase or fallback-passphrase flags, so I am not sure how to achieve that.
DOPPLER_TOKEN=$(doppler configs tokens create -p <project-name> -c <config-name> dev --plain --max-age 1m)
DOPPLER_TOKEN="${DOPPLER_TOKEN}" doppler secrets download -p <project-name> -c <config-name> fallback.json
DOPPLER_TOKEN="${DOPPLER_TOKEN}" \
doppler run --command="docker compose up --remove-orphans" --fallback= --passphrase dev
DOPPLER_TOKEN="" \
doppler run --command="docker compose up --remove-orphans" --fallback=/full/path/to/fallback.json --passphrase dev
Part of the issue you’re likely running into is that if you don’t manually create the fallback file, the one that’s automatically generated on a successful doppler run is created with a name based off of a combination of the token, project, and config (see here). So, when you use a different token, the filename being looked for changes unless you manually specify it.
Manually generating the file as described above also ensures the passphrase is applied as expected too. Simply running doppler run with --passphrase won’t cause the automatic fallback file to get encrypted using that passphrase (this is mostly because of the naming convention described earlier – the point of the passphrase is allow different access tokens to decrypt the file, but using a different token will result in the wrong automatic fallback file being used).
Thanks @watsonian! I think this brings me one step closer to the solution again. Excuse me if I ask silly questions, I had some more thoughts after trying the commands you shared above.
Does the ephemeral tokens approach work out of the box when offline? The mere fact we’re having this conversation would imply no, but maybe I am missing something. The original approach I used, to manually generate tokens with no expiry date in the Doppler UI, worked both online and offline. The reason why I am trying to adopt this ephemeral approach is for a better developer experience, i.e. we won’t have to create new tokens for every team member and remember to maintain those.
It seems like a lot of commands to compensate for the original feature. I have certain scripts that work with a different config (test environment), so you can imagine how much hairier these commands get with multiple configs. Had I used a monorepo, I imagine it would be even worse. I’d like to get to a state where I don’t have to make a tradeoff between the convenience and how hard it is to maintain it. For example, why can’t I simply add the fallback file path to the doppler configs tokens create command? It already accepts the same project and config flags I would’ve used in the doppler secrets download command anyway AND it downloads the secrets, too!
Tiny one, but the --fallback flag fails when supplied a path with directory that does not exist. If it’s a simple file, it will create it, but it won’t create directories. This one is easy to fix on my end, but again, should be handled by Doppler automatically.
On the topic of fallback files, I’d prefer if they were located in the project repository inside a .doppler directory by default. It’s easy to gitignore it and doesn’t hide the secrets inside my filesystem. Cache files work the same (e.g. mypy, pytest), node_modules, logs, too. Is there a reason this cannot be the default?
Similarly, this is my lack of knowledge, but why do the fallback file names have to be scrambled? Yes, you create them combining the token, project, and config, but the resulting name doesn’t make any of this obvious. Looking at the fallback files, I have no idea which config is inside .secrets-fd62d04d9d28be674b919d0ef26e653d47678b24107ae0b7de9a6ff9815206f3.json. Are there any security considerations? Wouldn’t it be simpler if these were located inside the current project and named after the settings I already may have in my doppler.yaml config file? I can guess which defaults you’re using based on that file, but the output name loses all that information.
This last one is totally on me, but I am still confused about how those --fallback and --passphrase flags work. Which command from those you shared above encrypts the config? Which one decrypts it? This is why I feel there are too many commands as I don’t see any passphrase being used in the tokens create nor secrets download step.
Overall, this feels like a rough feature, so I am curious about your thoughts! Am I approaching the implementation completely wrong?
Sorry for the long delay getting back to you. I’m afraid I lost track of this and am only just getting back to it. I’ll answer your points below in turn.
Ephemeral tokens will work offline for as long as they’re valid. However, if your scripts are relying on creating them on the fly like this:
then it won’t work when you’re offline because that command relies on being able to hit our API to generate a token.
Is there a specific reason you’re needing to generate these for each developer? Ordinarily, in a development environment users just use the CLI token that’s generated when you doppler login that has their accounts credentials. This will be used automatically when no other token is specified or can be accessed on-demand with doppler configure get token --plain. Assuming the user has signed in once, this will work fine with fallback files that have a passphrase attached if the need to work offline presents itself. Also, fallback files are automatically generated in ~/.doppler/fallback after successful doppler run invocations, so the CLI should fall back to using those in the event of a network outage.
That’s just not how it’s designed. The doppler configs tokens create command does exactly what it says – it generates an access token. This is completely separate from fetching secrets. If you want to reduce the number of commands being used, you can specify a location for the fallback file in doppler run using --fallback and if the file doesn’t exist it’ll be created there. You can also specify a passphrase as well. That said, in a deployment scenario (which is where this is typically geared for) you’ll likely want to explicitly generate the fallback file using doppler secrets download during the build phase.
If what you’re worried about is the ability for developers to access the secrets when offline, then just using their default CLI token and the automatic fallback files that are saved to ~/.doppler/fallback should work fine. You can see that in action by doing something like this:
doppler run --debug -- printenv | grep DEBUG
In the debug output, you’ll see lines like this:
Debug: Reading config file
Debug: Retrieving token from system keyring
Debug: Using metadata file /Users/$USER/.doppler/fallback/.metadata-d5d78a097706149a3d6d2c7b9690360bf489310eebbddb159acc028810d979f8.json
Debug: Reading metadata file /Users/$USER/.doppler/fallback/.metadata-d5d78a097706149a3d6d2c7b9690360bf489310eebbddb159acc028810d979f8.json
Debug: Performing HTTP GET to https://api.doppler.com/v3/configs/config/secrets/download?config=dev&format=json&include_dynamic_secrets=true&project=vercel-nextjs
Debug: Request ID 783a0322-bf43-4726-8116-293f94ee4ce1
Debug: Using cached secrets from fallback file
Debug: Using fallback file for cache /Users/$USER/.doppler/fallback/.secrets-d5d78a097706149a3d6d2c7b9690360bf489310eebbddb159acc028810d979f8.json
You can then run:
doppler run --fallback-only --debug -- printenv | grep DOPPLER
to see what happens when you tell it to ONLY use fallback files. It will still work and this simulates what would happen if the dev’s internet access wasn’t working (you can try turning off your wifi to see a real world fallback – the only difference is that there’s a timeout period first).
I can pass this feedback along. We do automatically create directories for some operations (e.g., --config-dir and the default ~/.doppler directory), so it’s not unprecedented here. I can’t say if/when that’ll get added though.
By default they’re created in ~/.doppler/fallback when you’re using automatic creation. When you’re specifying a location, the inference is that you plan to ship this with your project to use as a cached fallback file. When that’s not the intended purpose, then relying on the default, built-in fallback files should work fine.
They use a name like that for a couple reasons. First, if the files were ever leaked, it wouldn’t be obvious which projects they were for which would make attempting to brute force them harder. Second, it avoids potential name conflicts if you’re using multiple accounts. Finally, these files aren’t meant for human consumption and are meant to be consumed in an entirely automated fashion by the CLI behind-the-scenes. Referencing these files directly or modifying the ~/.doppler/.doppler.yaml file by hand outside of normal CLI usage is not recommended or supported really.
We’re basically discussing three commands in total here:
doppler secrets download – generates an encrypted secrets file (i.e., fallback file). by default, only the access token used to create it can decrypt it. if you use --passphrase then any access token can decrypt it if you use the correct --passphrase.
doppler run -- ... – fetches secrets from api.doppler.com and feeds them to your application as environment variables. this will automatically create a fallback file at ~/.doppler/fallback when successfully executed. if you want to control where the file is stored (or load an existing one) you can optionally pass in --fallback to designate that location. the fallback files created with this command have the same decryption requirements as doppler secrets download – meaning if you want a different access token to be able to access it (or you’re attempting read a pre-existing fallback file created with --passphrase) then you’ll need to use --passphrase. the encryption/decryption process only happens when fallback files are created/read.
doppler configs tokens create – creates an access token for the specified project and config.
In day-to-day usage for development purposes, you should only really need to use doppler run, which will automatically use the CLI token that was generated when you ran doppler login. That token will have access to all projects and configs your user has access to.
