# Doppler Secret Operator with GKE

**URL:** <https://community.doppler.com/t/doppler-secret-operator-with-gke/1238>\
**Category:** Need Help\
**Created:** [July 13, 2023, 6:46pm UTC](https://community.doppler.com/t/doppler-secret-operator-with-gke/1238 "2023-07-13T18:46:46Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![rrmangum](https://sea2.discourse-cdn.com/flex016/user_avatar/community.doppler.com/rrmangum/32/829_2.png) [@rrmangum](https://community.doppler.com/u/rrmangum)\
**Post date:** [July 13, 2023, 6:46pm UTC](https://community.doppler.com/t/doppler-secret-operator-with-gke/1238/1 "2023-07-13T18:46:46Z")

</div>

Previously my cluster was running the Doppler Secret Operator with no issue. My cluster settings were set to e2-medium machines with autoscaling, the cluster scaled to one e2-medium machine. Today I decided to switch my cluster from autoscaling to manual with 3 dedicated e2-micro machines. The rest of my cluster is functioning just fine, Only the doppler-operator-controller-manager is failing with the error: “Does not have minimum availability”. I do not believe it is an issue with resources as 3 e2 micro machines is pretty closely comparable to one e2-medium machine.

When I inspect the workload I have two errors: “Pod errors: Unschedulable” and “Does not have minimum availability”. The logs contain various messages some of which I’ll include here:

2023-07-13T18:00:45.964Z INFO controller-runtime.manager.controller.dopplersecret Stopping workers {“reconciler group”: “[secrets.doppler.com](http://secrets.doppler.com)”, “reconciler kind”: “DopplerSecret”}

“2023-07-13T18:00:41.972Z INFO controllers.DopplerSecret Reconciling dopplersecret {“dopplersecret”: “doppler-operator-system/doppler-token”}”

“2023-07-13T18:00:41.972Z INFO controllers.DopplerSecret Requeue duration set {“dopplersecret”: “doppler-operator-system/doppler-token”, “requeueAfter”: “1m0s”}”

“2023-07-13T18:00:41.978Z ERROR controllers.DopplerSecret Unable to update dopplersecret {“dopplersecret”: “doppler-operator-system/doppler-token”, “error”: “Failed to load Doppler Token: Failed to fetch token secret reference: Secret "doppler-token-secret" not found”}”

I did not make changes to the doppler-token or the Doppler secret operator configuration files. I believe this is an issue of the workload not being transferred to the new node pool since it lives in a different namespace.

---

<div class="post-metadata">

**Author:** ![watsonian](https://sea2.discourse-cdn.com/flex016/user_avatar/community.doppler.com/watsonian/32/267_2.png) [@watsonian](https://community.doppler.com/u/watsonian)\
**Post date:** [July 13, 2023, 7:43pm UTC](https://community.doppler.com/t/doppler-secret-operator-with-gke/1238/2 "2023-07-13T19:43:27Z")

</div>

Hi @rrmangum!

Our helm chart defines the following resource limits for the controller manager:

```auto
resources:
  limits:
    cpu: 100m
    memory: 256Mi
  requests:
    cpu: 100m
    memory: 256Mi

```

It’s also configured to create 1 replica by default. You should be able to adjust these in your `values.yaml` file if you’re using one. If you are, can you post what you’re using currently?

Regards,  
-Joel

---

<div class="post-metadata">

**Author:** ![rrmangum](https://sea2.discourse-cdn.com/flex016/user_avatar/community.doppler.com/rrmangum/32/829_2.png) [@rrmangum](https://community.doppler.com/u/rrmangum)\
**Post date:** [July 13, 2023, 7:47pm UTC](https://community.doppler.com/t/doppler-secret-operator-with-gke/1238/3 "2023-07-13T19:47:58Z")

</div>

I used kubectl to apply the secret operator. This is my .yaml for the secret operator. This same file also includes configurations for my application specific containers. I have them separated with the triple dash like this:

```auto
apiVersion: secrets.doppler.com/v1alpha1
kind: DopplerSecret
metadata:
  name: doppler-token
  namespace: doppler-operator-system
spec:
  tokenSecret: 
    name: doppler-token-secret
  managedSecret: 
    name: doppler-token
    namespace: default
---
# Application specific yaml configuration 

```

---

<div class="post-metadata">

**Author:** ![watsonian](https://sea2.discourse-cdn.com/flex016/user_avatar/community.doppler.com/watsonian/32/267_2.png) [@watsonian](https://community.doppler.com/u/watsonian)\
**Post date:** [July 13, 2023, 8:17pm UTC](https://community.doppler.com/t/doppler-secret-operator-with-gke/1238/4 "2023-07-13T20:17:43Z")

</div>

@rrmangum Could you try running this for me?

```shell
kubectl get secrets -n doppler-operator-system

```

Does it list the `doppler-token-secret` secret? The next thing to ensure is that the namespace under `metadata` is `doppler-operator-system` for all of your CRDs. All `DopplerSecret` CRDs must exist in the same namespace as the operator. The `managedSecret` can use other namespaces, but the `DopplerSecret` itself must be in the same namespace as the operator.

Could you also run the following commands and send me the output?

```shell
kubectl get pods -n doppler-operator-system
kubectl describe pod $NAME_OF_POD_FROM_PREVIOUS_COMMAND -n doppler-operator-system

```

Regards,  
-Joel

---

<div class="post-metadata">

**Author:** ![rrmangum](https://sea2.discourse-cdn.com/flex016/user_avatar/community.doppler.com/rrmangum/32/829_2.png) [@rrmangum](https://community.doppler.com/u/rrmangum)\
**Post date:** [July 13, 2023, 8:23pm UTC](https://community.doppler.com/t/doppler-secret-operator-with-gke/1238/5 "2023-07-13T20:23:07Z")

</div>

> [@watsonian](#):
>
> `kubectl get secrets -n doppler-operator-system`

```auto
kubectl get secrets -n doppler-operator-system

```

Returns “No resources found in doppler-operator-system namespace.” The doppler-token secret exists in the default namespace. Should it be in both namespaces? It was working before though and I didn’t change the location of the doppler-token secret.

I believe the only CRD I have is the one I sent you.

```auto
kubectl get pods -n doppler-operator-system

```

Returns:  
NAME READY STATUS RESTARTS AGE  
doppler-operator-controller-manager-cf7d9d8f-4zdz4 0/2 Pending 0 95m

```auto
kubectl describe pod $NAME_OF_POD_FROM_PREVIOUS_COMMAND -n doppler-operator-system

```

Returns:

```auto
Name: doppler-operator-controller-manager-cf7d9d8f-4zdz4
Namespace: doppler-operator-system
Priority: 0
Service Account: doppler-operator-controller-manager
Node: <none>
Labels: control-plane=controller-manager
                  pod-template-hash=cf7d9d8f
Annotations: cloud.google.com/cluster_autoscaler_unhelpable_since: 2023-07-13T18:45:22+0000
                  cloud.google.com/cluster_autoscaler_unhelpable_until: Inf
Status: Pending
IP:               
IPs: <none>
Controlled By: ReplicaSet/doppler-operator-controller-manager-cf7d9d8f
Containers:
  kube-rbac-proxy:
    Image: gcr.io/kubebuilder/kube-rbac-proxy:v0.14.1
    Port: 8443/TCP
    Host Port: 0/TCP
    Args:
      --secure-listen-address=0.0.0.0:8443
      --upstream=http://127.0.0.1:8080/
      --logtostderr=true
      --v=10
    Environment: <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-pglnv (ro)
  manager:
    Image: dopplerhq/kubernetes-operator:1.2.7
    Port: <none>
    Host Port: <none>
    Command:
      /manager
    Args:
      --health-probe-bind-address=:8081
      --metrics-bind-address=127.0.0.1:8080
      --leader-elect
    Limits:
      cpu: 100m
      memory: 256Mi
    Requests:
      cpu: 100m
      memory: 256Mi
    Liveness: http-get http://:8081/healthz delay=15s timeout=1s period=20s #success=1 #failure=3
    Readiness: http-get http://:8081/readyz delay=5s timeout=1s period=10s #success=1 #failure=3
    Environment: <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-pglnv (ro)
Conditions:
  Type Status
  PodScheduled False 
Volumes:
  kube-api-access-pglnv:
    Type: Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds: 3607
    ConfigMapName: kube-root-ca.crt
    ConfigMapOptional: <nil>
    DownwardAPI: true
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type Reason Age From Message
  ---- ------ ---- ---- -------
  Normal NotTriggerScaleUp 77s (x572 over 96m) cluster-autoscaler pod didn't trigger scale-up:
  Warning FailedScheduling 42s (x22 over 96m) default-scheduler 0/3 nodes are available: 3 Insufficient memory. preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod..

```

---

<div class="post-metadata">

**Author:** ![watsonian](https://sea2.discourse-cdn.com/flex016/user_avatar/community.doppler.com/watsonian/32/267_2.png) [@watsonian](https://community.doppler.com/u/watsonian)\
**Post date:** [July 13, 2023, 8:53pm UTC](https://community.doppler.com/t/doppler-secret-operator-with-gke/1238/6 "2023-07-13T20:53:39Z")

</div>

@rrmangum So, it looks like you have two potential problems. First, it looks like the Doppler operator pod isn’t running because there’s insufficient memory:

```auto
Warning FailedScheduling 42s (x22 over 96m) default-scheduler 0/3 nodes are available: 3 Insufficient memory. preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod..

```

Our controller has a 256Mi resource requirement, so it seems like your new cluster doesn’t have any nodes with sufficient free memory to schedule it on.

Second, it sounds like the `doppler-token-secret` secret is in the wrong namespace. It should be in the `doppler-operator-system` namespace. Note that this is distinct from the `doppler-token` DopplerSecret CRD you have described above. That CRD _references_ a k8s secret that contains the Doppler token used to fetch the secrets for the CRD entry. In our documentation, this is created with the name `doppler-token-secret` and it should be created in the `doppler-operator-system` namespace. It could be that it exists in the `default` namespace due to a historical issue depending upon when you installed originally. In previous versions of the operator, it could be in other namespaces, but this was changed as a security update and now it has to be in the same namespace as the operator (along with the CRDs). As mentioned, the `managedSecret` in the CRD can be in any namespace though.

Regards,  
-Joel

---

<div class="post-metadata">

**Author:** ![rrmangum](https://sea2.discourse-cdn.com/flex016/user_avatar/community.doppler.com/rrmangum/32/829_2.png) [@rrmangum](https://community.doppler.com/u/rrmangum)\
**Post date:** [July 13, 2023, 9:44pm UTC](https://community.doppler.com/t/doppler-secret-operator-with-gke/1238/7 "2023-07-13T21:44:21Z")

</div>

I scaled up the cluster to 4 nodes and that had no effect, each node is providing ~500mb of memory so adding one node should have been enough for the Doppler secret operator. I do not think it is resourcing issue?

I added the namespace tag for the tokenSecret in the configuration file, and applied it to my cluster trying to specify where the doppler-token-secret is located.

```auto
apiVersion: secrets.doppler.com/v1alpha1
kind: DopplerSecret
metadata:
  name: doppler-token
  namespace: doppler-operator-system
spec:
  tokenSecret: 
    name: doppler-token-secret
    namespace: doppler-operator-system
  managedSecret: 
    name: doppler-token
    namespace: default

```

That also had no effect as `kubectl get secrets -n doppler-operator-system` is still returning no resources found in doppler-operator-system namespace. When I run `kubectl get secrets -n default` I see the doppler-token kubernetes secret. How else can I move the doppler-token-secret?

---

<div class="post-metadata">

**Author:** ![watsonian](https://sea2.discourse-cdn.com/flex016/user_avatar/community.doppler.com/watsonian/32/267_2.png) [@watsonian](https://community.doppler.com/u/watsonian)\
**Post date:** [July 13, 2023, 10:06pm UTC](https://community.doppler.com/t/doppler-secret-operator-with-gke/1238/8 "2023-07-13T22:06:04Z")

</div>

@rrmangum You don’t need to add a `namespace` entry for the `tokenSecret`. That _must_ be in the operator namespace, so it’s inferred. You need to actually create the secret in that namespace:

```shell
kubectl create secret generic doppler-token-secret \
  --namespace doppler-operator-system \
  --from-literal=serviceToken=dp.st.dev.XXXX

```

With regard to adding another node – that may or may not help depending upon what gets deployed on each node. If anything else is being deployed, it’s very possible that the required space isn’t available. What’s needed is 256Mi of contiguous memory (i.e., if each node provides 1GB of memory and 900MB are used, having 5 nodes means you have 500MB of overall space free, but only 100MB of contiguous space on any given node). Try increasing the size of the nodes in your cluster rather than increasing the number of nodes and see if that helps.

---

<div class="post-metadata">

**Author:** ![rrmangum](https://sea2.discourse-cdn.com/flex016/user_avatar/community.doppler.com/rrmangum/32/829_2.png) [@rrmangum](https://community.doppler.com/u/rrmangum)\
**Post date:** [July 13, 2023, 10:59pm UTC](https://community.doppler.com/t/doppler-secret-operator-with-gke/1238/9 "2023-07-13T22:59:09Z")

</div>

So I created the secret in the doppler-operator-system and I migrated workloads to a pool with e2-small machines. The secret operator started functioning correctly.

As a test I deleted the doppler-token-secret from the the doppler-operator-system namespace and the secret operator is still functioning correctly.

The cluster is working as intended now, thanks for the help!

---

<div class="post-metadata">

**Author:** ![rrmangum](https://sea2.discourse-cdn.com/flex016/user_avatar/community.doppler.com/rrmangum/32/829_2.png) [@rrmangum](https://community.doppler.com/u/rrmangum)\
**Post date:** [July 21, 2023, 12:02am UTC](https://community.doppler.com/t/doppler-secret-operator-with-gke/1238/10 "2023-07-21T00:02:39Z")

</div>

@watsonian

Bringing this back because I clearly didn’t understand what I was doing! LOL

At this point I have created all the secrets necessary and applied them with my CRDs. I also added the annotation tag to my deployment CRD.

When I run `kubectl describe dopplersecrets -n doppler-operator-system` I can see the controller is running with messages like “Controller is continuously syncing secrets” and “Controller is ready to reload deployments. 2 found.”

However my application is failing to get my secrets. Any ideas?

---

<div class="post-metadata">

**Author:** ![rrmangum](https://sea2.discourse-cdn.com/flex016/user_avatar/community.doppler.com/rrmangum/32/829_2.png) [@rrmangum](https://community.doppler.com/u/rrmangum)\
**Post date:** [July 21, 2023, 12:10am UTC](https://community.doppler.com/t/doppler-secret-operator-with-gke/1238/11 "2023-07-21T00:10:24Z")

</div>

I can also see my secret names using `kubectl describe secret doppler-token`. It just shows the size of each secret instead of its actual value.

---

<div class="post-metadata">

**Author:** ![rrmangum](https://sea2.discourse-cdn.com/flex016/user_avatar/community.doppler.com/rrmangum/32/829_2.png) [@rrmangum](https://community.doppler.com/u/rrmangum)\
**Post date:** [July 21, 2023, 10:33pm UTC](https://community.doppler.com/t/doppler-secret-operator-with-gke/1238/12 "2023-07-21T22:33:16Z")

</div>

Found out the backend secret retrieval is working. I have two secrets on the frontend that are both undefined. Did I miss a configuration step?

---

<div class="post-metadata">

**Author:** ![watsonian](https://sea2.discourse-cdn.com/flex016/user_avatar/community.doppler.com/watsonian/32/267_2.png) [@watsonian](https://community.doppler.com/u/watsonian)\
**Post date:** [July 26, 2023, 6:57pm UTC](https://community.doppler.com/t/doppler-secret-operator-with-gke/1238/13 "2023-07-26T18:57:10Z")

</div>

@rrmangum Sorry for the delay getting back to you! Lost track of this thread.

It’s probably a misconfiguration of some kind. Could you post both the working and broken `DopplerSecret` CRD?