Updating SSL certificates in Kubernetes on Oracle Cloud

When you secure your servers and APIs, you need SSL certificates. Since they don't last for ever, you will have to update them every year, or every other year.

In OKE (Kubernetes on Oracle Cloud), there are two slightly different ways of doing this, depending on the kind of component you are using. Unfortunately, the Oracle documentation does not describe how to update SSL certificates in Kubernetes. 

Update an Ingress controller 

An ingress that is used for SSL termination looks something like this: 

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: my-ing
  annotations:
    kubernetes.io/ingress.class: "nginx"
spec:
  tls:
  - secretName: my-ssl-secret-name
  rules:
  - http:
      paths:
      - backend:
          serviceName: my-backend-svc
          servicePort: 8088

The Oracle documentation has a clear description of how to setup an ingress controller with TLS. Basically you create an ingress controller and a TLS secret and apply an ingress.yaml.  Note that when you check the load balancer in the OCI console that is associated with the ingress, there is no SSL certificate listed there, this is completely handled from within the Kubernetes cluster.

There is no mention of how to update the certificate but that is actually very simple: you can update the values in the existing Kubernetes secret (for example using the dashboard or a script) and the ingress controller will pick up the new certificate automatically. 

Update a service of type Load Balancer

A service of type load balancer yaml looks something like this: 

kind: Service
apiVersion: v1
metadata:
  name: my-frontend-service
  annotations:
    service.beta.kubernetes.io/oci-load-balancer-ssl-ports: "443"
    service.beta.kubernetes.io/oci-load-balancer-tls-secret: my-ssl-secret-name

spec:
  selector:

Again, the Oracle documentation has a clear description of how to setup a service with a load balancer.  It describes a similar procedure: you create a service and secret in Kubernetes and apply this. However, because the kind of deployment is a service which specifies a load balancer, the certificate is picked up by the OCI Load Balancer and copied there.

Again there is no mention of how to update the certificate. When you change the value of the certificate in the secret in Kubernetes, the change is not picked up by the OCI Load Balancer. There are a lot of warnings in the documentation not to change it in OCI directly, so that is not an option.  The procedure to update the certificate is:

1. Create a new secret with a new certificate and private key
2. Update the service yaml to point to the new secret
3. Apply the updated service yaml (don't create the service, just apply the updated yaml on the existing service) using kubectl apply -f [name of your yaml]
4. Delete the previous secret.

To streamline the procedure, we have decided to create a new secret for both the ingress controller and the service so we don't end up with multiple secrets containing the same wildcard certificate and to make sure we can use one procedure for both types of components.

I hope the documentation will be updated by Oracle to include an instruction for updating SSL certificates. In the mean time I hope this post helps you when you need to update your SSL certificates. 

Happy coding! 😀

Migrating from Developer Cloud Service classic to Developer Cloud in OCI

One of our customers was still running a Developer Cloud Classic instance. We decided to migrate to Developer Cloud. Since the inception of Oracle Cloud and Oracle Platform as a service, services have moved from "Classic" to "Autonomous" to "Native". What does this mean and why does Oracle have all these different types of services in their cloud? And why would you care? Let's go back in time a little bit.

A history lesson

Oracle started with a Cloud Infrastructure and Platform  (PaaS) and Software services (SaaS) to catch up with competitors like Amazon (AWS) and Microsoft (Azure). Soon after realizing the first generation infrastructure an improvement project was started. This resulted in the current Oracle Cloud Infrastructure (OCI) with compartments, services that can be used by different PaaS services, improved networking etc. At the same time, the first generation of PaaS services was improved, to become "Autonomous": the customer does not have to worry about maintaining the platform, this is done by Oracle. This resulted in products like Oracle Autonomous Data Warehouse  and Autonomous Oracle Integration Cloud. The name "Autonomous" was quickly dropped from most services except Database related products, but the advantage was clear: Oracle manages the platform instances, so customers don't have to worry or worry less about availability, upgrades and patches. A number of these services were not using the new features of the Oracle Cloud Infrastructure.  Oracle released new platform services: so called "native" services that use the networking, compartment, notification and other features of OCI. Note that Autonomous Database products and services like Kubernetes Engine and Function were native from the start.

Still confused? Let's look at an example: Oracle Integration Cloud.

Example: OIC

Integration (also known as Oracle Integration Cloud or OIC) started with Integration Classic. When you provisioned OIC Classic it was provisioned in the Oracle's first generation, or classic, infrastructure. The next generation, which was called Autonomous OIC for a while, is/was running in Oracle Cloud Infrastructure. However, it is not making use of all the native services or functionality that OCI offers: you can't define the compartment it is provisioned in, it does not use the notification services, it does not have Terraform support etc.  It is running in the specifically dedicated ManagedCompartmentForPaaS compartment, that gets automatically created when you provision (non-native) Oracle Integration The latest (and hopefully last) installment is Oracle Integration, which is native. When you provision it, you must create a compartment for it first and it uses all the available services (networking, notification etc.) of Oracle Cloud Infrastructure.

It has several advantages to move to Oracle Cloud Infrastructure native services:

• Organize cloud resources into a hierarchy of logical compartments.
• Create fine-grained access policies for each compartment.
• Support for Terraform for provisioning
• Usage of Security and network features in Oracle Cloud Infrastructure
• Last but not least, new features are being added to the native service, not to the non-native services.

Migration of Oracle Developer Cloud Classic

Now that we know how Oracle moves her services, what is the situation for Oracle Developer? Oracle Developer Cloud Classic is provisioned in the classic infrastructure and as a result, uses Oracle Classic infrastructure components for load balancing, networking, storage, etc. There is no native Developer Cloud service (yet?). However, you can move your Developer Cloud  Service Classic to OCI (in the dedicated ManagedCompartmentForPaaS compartment). Developer Cloud service offers different features: a GIT repository, build jobs, deployment jobs, wiki, issue system, to name a few. For a number of these features you need resources. From Developer Cloud Classic you can connect to OCI to use the resources. For example: a build job needs a build VM and storage to store the artifacts. So even though the Developer Cloud Service itself is not running in a compartment of your choice, the jobs you are running and the code you are storing is making use of these features. But Developer Cloud Service itself, also uses networking to give you as a developer access to the console. Think about the load balancer, an IP address etc. For this customer, we wanted to remove all resources from the Classic Infrastructure, we already migrated Mobile Cloud Service to Mobile Hub and API Platform Classic to API Platform. Developer Cloud Classic was the last man standing...

If you are not interested in my experience of the migration, but just want to go ahead and do it, you can find the Oracle documentation here.

We executed the following steps:

• Create the Developer Cloud instance in OCI
• Create the storage resource for migration of the project
• Migrate the project
• Add the users
• Remove the Developer Cloud  Classic instance 

Create the Developer Cloud instance in OCI

Because it is not native yet, you can't create the instance using Terraform. The easiest way is to use the user interface:

• Login to OCI
• Select "Developer Cloud" from Platform Services
• Enter the name you want to use 
• Add a description and some tags (optional)
• Confirm 

You can now create the organization properties, by copying them from the Developer Cloud Classic instance or create your project using new properties (which what we did).

If you are planning to create build jobs in the future (which we are planning for Kubernetes), you need to make a connection to OCI from Developer CS.

Now we have our Developer Cloud setup, it is time to migrate the stuff from the old instance.

Create storage resource

The first thing to do when you want to export and import the git repository, is to setup an OCI Object Storage bucket to host the data from the exported project. You can use a common container for all projects, but we recommend keeping separate storage buckets per project.

To avoid mistakes and make sure we have consistent naming conventions, we use Terraform scripts to create resources. You can find the documentation here.

Note: you can skip all the optional fields mentioned, all you need is the compartment_id, the bucket_name and the bucket_namespace.

We will use a compartment we created for this purpose: DEVCS compartment, and the user, devcs.user with a public and private key, the group and the policies. For more information, see the documentation. Note that in the documentation, separate users and groups are setup for both managing the resources from Developer Cloud Service and reading and writing the data from the storage resource to manage the import and the export. In our case, it did not add much value to have separate users and policies, so we reused the devcs.user for this purpose. This means we din't add the policies because devcs.user can already manage all resources in the compartment.

Move the git repository

We are now ready to move our git repository.  We will migrate the entire project that the git repository is part of.

• Export your project. You find extensive documentation here.
• Import the data into a new project. You find extensive documentation here.
• Create project
• Import project
• Add users (those are not migrated, unfortunately)

Please make sure you add users to the application role DEVELOPER_USER of the new Developer Cloud Instance in IDCS first. This is a bit hidden in the documentation. Please note that users who have accounts that are locked can't be added to the project. You have to unlock them first. This often occurs because some tool like SourceTree poll the repository. If the password has expired, this leads to a locked account.

Alternative

Because in our case, we only had a git repository we wanted to migrate, the alternative would have been to skip the export/import step and simply create a new remote git repository in the new project of the new instance and then push the code to that remote.

How to do that is described here

So, what is the preferred method? After executing the steps above my recommendation is:

• If you only have a git repository and no Wiki, Jira or build jobs you need to import just create a new project and a new remote git repository
• If you have build jobs, Wiki pages, Jira issues that you want to keep, use the import/export methodology described in this blog post.

Remove the Developer Cloud Classic instance 

Now that we are done and have the new project up and running, we need to clean up the old instance. The documentation describes how to do that here.

Happy coding 😀