Certificates

BOSH Cert Rotation

The best way to replace the certs now is to do the following: https://bosh.io/docs/nats-ca-rotation/#expired

You’ll need to remove the existing creds, recreate the director (no downtime for deployed apps), and then recreate –fix all your deployments. You may experience downtime here, depending on how your deployed jobs are configured.

Here’s what I did to update bosh:

cd bosh
source vsphere-vars.sh
# delete all the certs in the director-vars-store.yml that have expired
# bosh int ./vars/director-vars-store.yml --path /default_ca/ca | openssl x509 -noout -dates
# bosh int ./vars/director-vars-store.yml --path /director_ssl/ca | openssl x509 -noout -dates
# bosh int ./vars/director-vars-store.yml --path /nats_server_tls/ca | openssl x509 -noout -dates
# bosh int ./vars/director-vars-store.yml --path /nats_ca/ca | openssl x509 -noout -dates
# bosh int ./vars/director-vars-store.yml --path /nats_clients_director_tls/ca | openssl x509 -noout -dates
# bosh int ./vars/director-vars-store.yml --path /nats_clients_health_monitor_tls/ca | openssl x509 -noout -dates
# bosh int ./vars/director-vars-store.yml --path /credhub_ca/ca | openssl x509 -noout -dates
# bosh int ./vars/director-vars-store.yml --path /credhub_tls/ca | openssl x509 -noout -dates
# bosh int ./vars/director-vars-store.yml --path /blobstore_server_tls/ca | openssl x509 -noout -dates
# bosh int ./vars/director-vars-store.yml --path /blobstore/ca | openssl x509 -noout -dates
# bosh int ./vars/director-vars-store.yml --path /mbus_bootstrap_ssl/ca | openssl x509 -noout -dates
# bosh int ./vars/director-vars-store.yml --path /uaa_service_provider_ssl/ca | openssl x509 -noout -dates
# bosh int ./vars/director-vars-store.yml --path /uaa_ssl/ca | openssl x509 -noout -dates
cp ./vars/director-vars-store.yml ./vars/director-vars-store-backup.yml
rm ./vars/director-vars-store.yml
bbl up --iaas vsphere --debug
eval "$(bbl print-env)"
bosh ds
bosh -d concourse vms
bosh -d concourse recreate --fix -n
bosh -d concourse vms
bosh -d minio recreate --fix -n
bosh -d minio vms
source target-concourse-credhub.sh

Here’s what I did to update jumpbox:

cd bosh
./delete-jumpbox.sh

Credhub CA Rotation

https://github.com/pivotal/credhub-release/blob/main/docs/ca-rotation.md

CredHub offers support for zero-downtime rotation of certificate credentials by allowing certificates to have multiple “active” versions at the same time.

The workflow at a high-level for transitioning to a new certificate is:

  1. If not already configured, redeploy your CredHub server with the credhub.certificates.concatenate_cas: true option. This will combine all active versions of a CA in a certificate’s .ca field.
  2. Regenerate your CA certificate with the transitional flag. This creates a new version that will not be used for signing yet, but can be added to your servers trusted certificate lists. Then, propagate the concatenated CAs to your software (e.g. BOSH redeploy).
  3. Remove the transitional flag from the new CA certificate, and add it to the old CA certificate. This means that the new certificate will start to be used for signing, but the old one will remain as trusted.
  4. Regenerate certificates that are signed by the CA, and then propagate new certificates to your software (e.g. BOSH redeploy).
  5. Remove the transitional flag from the old CA certificate. Optionally, again propagate changes to your software to remove old CA (e.g. BOSH redeploy).

Step 1: Regenerate

For the purpose of this example, we’ll be using the credential path /example-ca to refer to a CA stored on the CredHub server and /example-cert to refer to a certificate signed by that CA. Replace this value with the path of the CA you wish to rotate.

First we’ll need to get the ID of the CA certificate. Ensure you’re targeting and logged-in to the proper CredHub server.

$ credhub curl -p "/api/v1/certificates?name=/example-ca"

{
  "certificates": [
    {
      "id": "00000000-0000-0000-0000-000000000000",
      "name": "/example-ca",
      ...
    }
  ]
}

Next, we use that ID to generate the new, transitional version:

$ credhub curl -p "/api/v1/certificates/00000000-0000-0000-0000-000000000000/regenerate" -d '{"set_as_transitional": true}' -X POST

You should now see that when you request the “current” versions of that CA credential, that both versions are returned:

$ credhub curl -p "/api/v1/data?name=/example-ca&current=true"

{
  "data": [
    {
      "id": "11111111-1111-1111-1111-111111111111",
      "transitional": false,
      ...
    },
    {
      "id": "22222222-2222-2222-2222-222222222222",
      "transitional": true,
      ...
    }
  ]
}

Also, if you request /example-cert, you should see both versions of the CA certificate in the ca field:

$ credhub get -n /example-cert

name: /example-cert
value:
  ca: |
    -----BEGIN CERTIFICATE-----
    <OLD-VERSION>
    -----END CERTIFICATE-----
    -----BEGIN CERTIFICATE-----
    <NEW-VERSION (TRANSITIONAL)>
    -----END CERTIFICATE-----
  ...

Step 2: Moving the transitional flag

To move the transitional flag off of the new CA certificate and onto the older version, we’ll need to grab the older version’s ID:

$ credhub curl -p "/api/v1/certificates?name=/example-ca"

{
  "certificates": [
    {
      "id": "00000000-0000-0000-0000-000000000000",
      "versions": [
        {
          "id": "11111111-1111-1111-1111-111111111111",
          "transitional": false,
          ...
        },
        {
          "id": "22222222-2222-2222-2222-222222222222",
          "transitional": true,
          ...
        }
      ]
    }
  ]
}

Find the ID of the CA certificate version that currently has transitional: false, and then pass it to the next command:

$ credhub curl -p /api/v1/certificates/00000000-0000-0000-0000-000000000000/update_transitional_version -d '{"version": "11111111-1111-1111-1111-111111111111"}' -X PUT

You can confirm that now the two CA certificate versions have swapped:

$ credhub curl -p "/api/v1/data?name=/example-ca&current=true"

{
  "data": [
    {
      "id": "11111111-1111-1111-1111-111111111111",
      "transitional": true,
      ...
    },
    {
      "id": "22222222-2222-2222-2222-222222222222",
      "transitional": false,
      ...
    }
  ]
}

The order has also been swapped in the ca field of /example-cert:

$ credhub get -n /example-cert

name: /example-cert
value:
  ca: |
    -----BEGIN CERTIFICATE-----
    <NEW-VERSION>
    -----END CERTIFICATE-----
    -----BEGIN CERTIFICATE-----
    <OLD-VERSION (TRANSITIONAL)>
    -----END CERTIFICATE-----
  ...

Step 3: Regenerate certificates

Regenerate all certificates that are signed by the new CA certificate.

Step 4: Removing the transitional flag

After you have regenerated all your certificates, you can safely remove the transitional flag from the old one:

$ credhub curl -p /api/v1/certificates/00000000-0000-0000-0000-000000000000/update_transitional_version -d '{"version": null}' -X PUT

You can now confirm that only one CA certificate version is active:

$ credhub curl -p "/api/v1/data?name=/example-ca&current=true"

{
  "data": [
    {
      "id": "22222222-2222-2222-2222-222222222222",
      "transitional": false,
      ...
    }
  ]
}

And only the new CA certificate version is returned in the ca field of /example-cert:

$ credhub get -n /example-cert

name: /example-cert
value:
  ca: |
    -----BEGIN CERTIFICATE-----
    <NEW-VERSION>
    -----END CERTIFICATE-----
  ...

Here’s what I did to rotate concourse creds in credhub

credhub curl -p "/api/v1/certificates"

credhub get -n /bosh-bbl-env-malawi-2019-08-12t00-42z/concourse/atc_tls -k ca | openssl x509 -noout -dates
certificate_id=768b0dd0-2898-4b61-af1f-913771599892
credhub curl -p "/api/v1/certificates/$certificate_id/regenerate" -d '{"set_as_transitional": true}' -X POST

credhub get -n /bosh-bbl-env-malawi-2019-08-12t00-42z/concourse/atc_ca -k ca | openssl x509 -noout -dates
certificate_id=1c29bdb2-2822-46f2-8d15-9a1c5eb028a1
credhub curl -p "/api/v1/certificates/$certificate_id/regenerate" -d '{"set_as_transitional": true}' -X POST

credhub curl -p "/api/v1/certificates?name=/bosh-bbl-env-malawi-2019-08-12t00-42z/concourse/atc_ca&current=true"
credhub curl -p "/api/v1/certificates?name=/bosh-bbl-env-malawi-2019-08-12t00-42z/concourse/atc_tls&current=true"

certificate_id=768b0dd0-2898-4b61-af1f-913771599892
credhub curl -p /api/v1/certificates/$certificate_id/update_transitional_version -d '{"version": "3dfbd41d-012d-4856-98e9-e99a91ff43bc"}' -X PUT
certificate_id=1c29bdb2-2822-46f2-8d15-9a1c5eb028a1
credhub curl -p /api/v1/certificates/$certificate_id/update_transitional_version -d '{"version": "8c7eda25-2090-4866-85e8-7cf72708e9de"}' -X PUT

credhub curl -p "/api/v1/certificates?name=/bosh-bbl-env-malawi-2019-08-12t00-42z/concourse/atc_ca&current=true"
credhub curl -p "/api/v1/certificates?name=/bosh-bbl-env-malawi-2019-08-12t00-42z/concourse/atc_tls&current=true"

bosh -d concourse man > concourse.yml
bosh -d concourse deploy concourse.yml --no-redact
rm concourse.yml

/dns_api_client_tls
certificate_id=5943a70e-d2a8-40e0-acfb-97efbf9789c8
credhub curl -p "/api/v1/certificates/$certificate_id/regenerate" -d '{"set_as_transitional": true}' -X POST

/dns_api_server_tls
certificate_id=fb151107-70f3-4c6f-a2f1-a8dab0910669
credhub curl -p "/api/v1/certificates/$certificate_id/regenerate" -d '{"set_as_transitional": true}' -X POST

/bosh_dns_health_client_tls
certificate_id=74813987-ee24-4bc3-ae8d-ee0d8d9816c0
credhub curl -p "/api/v1/certificates/$certificate_id/regenerate" -d '{"set_as_transitional": true}' -X POST

/bosh_dns_health_server_tls
certificate_id=d3b43bf7-49cb-457e-b3d7-e1c779ec3338
credhub curl -p "/api/v1/certificates/$certificate_id/regenerate" -d '{"set_as_transitional": true}' -X POST

credhub curl -p "/api/v1/certificates?name=/dns_api_client_tls&current=true"
certificate_id=5943a70e-d2a8-40e0-acfb-97efbf9789c8
credhub curl -p /api/v1/certificates/$certificate_id/update_transitional_version -d '{"version": "bef22604-df47-41a2-8f71-5586dceb2f80"}' -X PUT
credhub curl -p "/api/v1/certificates?name=/dns_api_client_tls&current=true"

credhub curl -p "/api/v1/certificates?name=/dns_api_server_tls&current=true"
certificate_id=fb151107-70f3-4c6f-a2f1-a8dab0910669
credhub curl -p /api/v1/certificates/$certificate_id/update_transitional_version -d '{"version": "7bf678ea-55f3-44fc-aab7-95d41ccc7744"}' -X PUT
credhub curl -p "/api/v1/certificates?name=/dns_api_server_tls&current=true"

credhub curl -p "/api/v1/certificates?name=/bosh_dns_health_client_tls&current=true"
certificate_id=74813987-ee24-4bc3-ae8d-ee0d8d9816c0
credhub curl -p /api/v1/certificates/$certificate_id/update_transitional_version -d '{"version": "316f785d-5a83-485a-b0e2-ed06c9769f64"}' -X PUT
credhub curl -p "/api/v1/certificates?name=/bosh_dns_health_client_tls&current=true"

credhub curl -p "/api/v1/certificates?name=/bosh_dns_health_server_tls&current=true"
certificate_id=d3b43bf7-49cb-457e-b3d7-e1c779ec3338
credhub curl -p /api/v1/certificates/$certificate_id/update_transitional_version -d '{"version": "b5dcdeff-85fa-47f9-b748-4b51e8fec26b"}' -X PUT
credhub curl -p "/api/v1/certificates?name=/bosh_dns_health_server_tls&current=true"

Check that your pipelines are updated as well

source target-concourse-credhub.sh
credhub get -n /concourse/main/install-harbor-homelab-pipeline/credhub_ca -k certificate | openssl x509 -noout -dates
cd ..
./scripts/credhub-generate-pipeline-secrets.sh true

source "./scripts/set-om-creds.sh"
om expiring-certificates
credhub get -n /platform-automation/certificate_authority -k ca | openssl x509 -noout -dates
om credentials -p pivotal-container-service --credential-reference .pivotal-container-service.pks_tls --credential-field cert_pem | openssl x509 -noout -dates
om credentials -p harbor-container-registry --credential-reference .properties.server_cert_ca --credential-field value | openssl x509 -noout -dates
./scripts/credhub-generate-platform-certificates.sh true

Update homelab harbor cert 

kubectl get secrets prod-wildcard-certs -n cert-manager -ojsonpath={.data.'tls\.key'} | base64 -d > prod.key
kubectl get secrets prod-wildcard-certs -n cert-manager -ojsonpath={.data.'tls\.crt'} | base64 -d > prod.crt

Grab the first cert in the prod.crt and use that to update the server_cert_key cert_pem and grab prod.key and use that to update the server_cert_key private_key_pem

To rotate, all you should really need to do is update the cert_pem because the private key will be the same unless you updated cert manager with a new private key.

This project is for educational and home lab purposes.