Hi,
So I have this ML project in a Google Colab notebook that requires a couple of API keys and tokens to talk to third-party services. It’s on Google Colab for dev purposes, and I’ve been uploading the envs to the temporary environment every time I run the notebook.
Is there an easier way to import secrets from Doppler into my Google Colab notebook?
I was curious about this so I spent some time looking at it. With Colab being ephemeral in the free tier it’s a little tricky to automatically set things up.
The simplest thing I could find that works is to install the cli, login with it, then use it to get the secrets in dotenv format and import them, but this requires going through the login flow each time you start the Colab environment. I tried to find a way to use the Colab identity with GCP Secrets Management which would then be synced directly with Doppler but was unsuccessful. This is my second day using Doppler so take all this with a grain of salt 
.
Code Follows:
# Install doppler CLI
!(curl -Ls --tlsv1.2 --proto "=https" --retry 3 https://cli.doppler.com/install.sh || wget -t 3 -qO- https://cli.doppler.com/install.sh) | sudo sh
# Go through login flow via another tab
!doppler login -y
# Select the default project and environment
!doppler setup -p example-project -c dev
# This installs dotenv, captures the .env format file, turns it into a stream, and sends it to dotenv to load
# It then deletes the variables and shows the usual os.getenv() method to access
!pip install python-dotenv
from io import StringIO
from dotenv import load_dotenv
import os
env_cap = !doppler run --mount .env -- cat .env
env_s = StringIO(env_cap.n)
load_dotenv(stream=env_s)
del env_cap, env_s
os.getenv('DB_URL')
# Clear the login token
!doppler login revoke -y
Hi @mentat!
Welcome to the Doppler Community and thanks for posting your solution!
There are a couple additional things that might help you out here. First, if you can set environment variables on Google Collab, then you can set DOPPLER_TOKEN with a service token and the CLI will automatically use that as the token without the need to login or perform doppler setup (since the service token is tied to a specific project and config).
Also, you can simplify the CLI command you’re running a bit by changing:
doppler run --mount .env -- cat .env
to
doppler secrets download --no-file --format=env
As far as python goes, we do also have this package available:
It was meant more for local development though, so it may not work well for this specific use case, but I figured I’d mention it!
Regards,
-Joel
Edit: Updated format=env typo to --format=env
Hi @watsonian,
It doesn’t appear that there’s a way to set environment variables that persist across Colab startups. There is a project that mounts Google Drive and then sources an env file there (colab-env · PyPI).
Using a service token in vars.env in the Drive root, I was able to do the setup steps the same. It’s probably a slightly better experience in some ways but the Drive mount authorization is several clicks and sends a Security Alert to email every time it’s used. If one is already mounting Drive, it’s a good option, otherwise it seems like a less friendly flow to me.
I tried using the suggested secrets download but there appears to be a syntax issue which interestingly doesn’t cause the command to fail, instead it just outputs JSON. The correct command is
!doppler secrets download --no-file --format env
The situation with Drive mounting causing alerts and no other option for persistence really makes providing a secure and easy way to source environments more difficult than it should be. Perhaps the run_console() oauth flow to pull an env file from GCS would be a bit cleaner but doppler login seems like an ok experience considering the tradeoffs of the other options.
@mentat Thanks for catching that typo in the command! Unfortunately, the most ideal way for this to work would be for them to have a place where you can set env vars for the notebook that has an API. You could then either manually sync them yourself or we could potentially create an integration down the road. Hopefully, they’ll add something like that! In the mean time, it definitely looks like the experience won’t be 100% ideal. Thanks again for sharing your solution!
