Of course I have a backup!

Random blobs of wisdom about software development

Kubernetes on GKE with cloudsql proxy, and cronjobs

Thursday, February 22, 2018

If you are running kubernetes on google cloud, you probably run into the issue that it's not so straightforward to connect to a cloudsql instance from within the cluster. The official best practice is to run cloudsql proxy as a sidecar container in your pods, and connect through that. There are two problems with this, and I will address both.

So the two problems are:

  1. This will create a sidecar for everything that needs a database connection, CI, one off jobs, the actual application, they all take up resources, which will be a problem for small clusters
  2. You will not be able to run cronjobs

Cronjobs have landed in k8s in 1.8, a much needed feature, however, one big problem is that cronjobs only consider themselves done, when all the containers in a pod finish, either by exiting succesfully, or failing. This poses a problem with the cloudsql-proxy, because it will persistently stay open, and even though your actual cronjob finishes, the pod will never be unscheduled, and it will very quickly clog up your nodes, since every invocation of the cronjob will create a new pod, until you run out of resources.

There is already a ticket on cloudsql, and another, more general one here, about better support for sidecar containers. Until then, one manageable workaround is the following:

Instead of using the the advised method of deploying cloudsql as a sidecar container, you deploy it as a standalone deployment, and a service:

apiVersion: extensions/v1beta1
kind: Deployment
    name: cloudsql
    replicas: 1
                app: cloudsql
                - name: cloudsql
                  image: gcr.io/cloudsql-docker/gce-proxy:1.09
                      - "/cloud_sql_proxy"
                      - "-dir=/cloudsql"
                      - "-credential_file=/secret/gcp-key/keyfile.json"
                      - "-instances=sql-instance-here=tcp:" # accrording to Jose M. you need the last "=tcp" bit for it to listen on all interfaces
                    - name: gcp-key
                      mountPath: /secret/gcp-key
                    - containerPort: 5432
                      name: sql
              - name: gcp-key
                    secretName: gcp-keyfile
apiVersion: v1
kind: Service
  name: cloudsql
  type: ClusterIP
    app: cloudsql
    - port: 5432
      name: sql
      targetPort: sql

And then use the service, cloudsql, as the hostname for connecting your clients. As long as your clients don't cache dns requests, this shouldn't be a problem, even if the cloudsql pod cycles, since the dns name is going to automatically point to the new IP.

This was written by Norbert Kéri, posted on Thursday, February 22, 2018, at 23:12

Tagged as:
manu wrote
Are you sure this work ? The proxy accepts connection only on localhost.

2018-08-28 19:01:50

Norbert Kéri wrote
Yeah, it worked fine ~1 year ago when I was setting it up.

2018-09-08 20:11:57

Mats wrote
Hey, This works for me, but how do I scale this though? It seems like there's a limit to the number of writes it can pull of and it seemingly doesn't help increasing the replicas. Any ideas?

2018-09-09 18:08:45

Jose M. wrote
Hey guys, just a heads up; instances should be updated as follows: -instances=<INSTANCE_NAME>=tcp:, so it listens for external connections.

2018-09-13 23:28:30

Norbert Kéri wrote
@Jose M.
I added a note about the tcp bit, thanks.

I have no idea sorry, the site we used haven't run into any limits, and I haven't been using cloudsql for a while now.

2018-09-19 09:31:53

Post a comment

Providing your email is optional, it is never published or shared, it is only used for auto approval purposes. If you already have at least 1 approved comment(s) tied to your email, you don't have to wait for moderation, otherwise the author must approve your comment.

Please solve this totally random captcha