This is a surprisingly common issue. I’ve had it happen at least once in every job I’ve worked. This is usually the responsibility of the devops or devsec teams, and they are usually heavily underfunded since they are cost centers that do not bring in profit.
I work in DevOps, this is one of the easier things to automate. It’s common for certs to be issued on a 90 day basis these days, no way that would be maintainable without automating.
Yeah I’ve had certbot mess up a few times, though more often it was the scripts that actually shuttle the updated certs to their proper locations and restart services after updating
Certbot / LE has to be running on some machine and that machine can be accidentally turned off, payments not fulfilled, was supposed to be moved but the new instance doesn’t work, gateway configuration changed, etc.
Automation requires maintenance and that introduces human error
Like dgdft said, if you’re using certbot, it should typically be running on the machine that your endpoints are hosted on. Enterprise solutions don’t require this, but they have other means of deploying certificates automatically and alarming if they are unable to, before they expire. My organization has dashboards showing which certs expire and when, and it triggers alarms at least a month before anything goes wrong.
High stakes automation should always have alarms on error, and since certs have set expiration dates baked into them, you can alarm far before anything goes wrong. Apparently, Riot didn’t have that.
Also, more frequent renewals make it so that people are less likely to forget it exists. Because of that, along with the possible security ramifications, 2 to 10 year certs should never be used, in my opinion. A 10 year cert will always get kicked on to the next team and it’s very possible for things to fall through the cracks.
Certbot/LE should typically be running on the box that’s terminating TLS for you, right? If the box handling your traffic is down, shouldn’t that be a self-evident problem?
I’ve been running Caddy and certbot for nearly a decade and never found a way for them to break without it being 100% my fault. They’re more or less self-healing too. I’m with AmbiguousProps; cert renewals have been pretty damn reliable to automate compared to any other piece of tech, IME.
Cool story if everything you have has an API or code based. Try doing it on hundreds of switches and other embedded devices. The whole 42 day thing they’re floating is gonna be a massive nightmare because they don’t realize all the other things out there that use certificates.
What makes you think I don’t do this on embedded devices? I’m not about to dox my self with specifics, but I do this exclusively for embedded hardware as my job. We even do it for devices not directly attached to our network. It’s really not difficult so long as you have control of your enterprise hardware (which, you should, unless your management is terrible at their jobs). Hell, even the routers we use have this functionality built in, failure alarms and all.
If this is a problem for you, it’s probably at an organizational level, and not a technical issue.
I’m young enough that I never had to experience anything but let’s encrypt/ACME. Manually renewing certs sounds like such a major PITA that I’d switch to it as soon as I could…
I had to do it a couple of times, major pain, mostly as you do it that infrequently that you forget everything that you have to do to enable and change them
You’d be surprised. DevOps are, at least by my experience, SEVERELY overworked and understaffed. Even such things as writing a script is not always so easy, especially when security credentials are involved. Depending on the company there may be many layers of red tape for using security credentials stored in a secrets vault, so sometimes such things aren’t even possible since there is no official work request for that automation.
A lot what you mention is for sure true. My problem usually is deferring infra work to people that should not do it. More often than not I see so called “devops teams”, and once I see the elements on those, is very clear people that are actually for infra are either insufficient, missing, or straight up not enough permissions to do stuff. Yo would be shocked how many times I hear managers say " ah well, I bet one ofb our developers knows how to do infra work"
until one of your employer’s multi-million dollar customers insists on a commercial certificate so it’s a yearly effort to buy and distribute the damn thing
You are not logged in. However you can subscribe from another Fediverse account, for example Lemmy or Mastodon. To do this, paste the following into the search field of your instance: [email protected]
No game suggestions, friend requests, surveys, or begging.
No Let’s Plays, streams, highlight reels/montages, random videos or shorts.
No off-topic posts/comments, within reason.
Use the original source, no clickbait titles, no duplicates.
(Submissions should be from the original source if possible, unless from paywalled or non-english sources.
If the title is clickbait or lacks context you may lightly edit the title.)
This is a surprisingly common issue. I’ve had it happen at least once in every job I’ve worked. This is usually the responsibility of the devops or devsec teams, and they are usually heavily underfunded since they are cost centers that do not bring in profit.
I work in DevOps, this is one of the easier things to automate. It’s common for certs to be issued on a 90 day basis these days, no way that would be maintainable without automating.
The problem sometimes is the automation failing for some reason.
Have you had Certbot or LE fail on prod for you before?
I’m sure stuff happens, but I usually view them as one of the most robust moving parts on a server.
E: I don’t mean to express disbelief at all; just curious to learn about possible footguns.
Yeah I’ve had certbot mess up a few times, though more often it was the scripts that actually shuttle the updated certs to their proper locations and restart services after updating
Certbot / LE has to be running on some machine and that machine can be accidentally turned off, payments not fulfilled, was supposed to be moved but the new instance doesn’t work, gateway configuration changed, etc.
Automation requires maintenance and that introduces human error
Like dgdft said, if you’re using certbot, it should typically be running on the machine that your endpoints are hosted on. Enterprise solutions don’t require this, but they have other means of deploying certificates automatically and alarming if they are unable to, before they expire. My organization has dashboards showing which certs expire and when, and it triggers alarms at least a month before anything goes wrong.
High stakes automation should always have alarms on error, and since certs have set expiration dates baked into them, you can alarm far before anything goes wrong. Apparently, Riot didn’t have that.
Also, more frequent renewals make it so that people are less likely to forget it exists. Because of that, along with the possible security ramifications, 2 to 10 year certs should never be used, in my opinion. A 10 year cert will always get kicked on to the next team and it’s very possible for things to fall through the cracks.
Certbot/LE should typically be running on the box that’s terminating TLS for you, right? If the box handling your traffic is down, shouldn’t that be a self-evident problem?
I’ve been running Caddy and certbot for nearly a decade and never found a way for them to break without it being 100% my fault. They’re more or less self-healing too. I’m with AmbiguousProps; cert renewals have been pretty damn reliable to automate compared to any other piece of tech, IME.
Future generations using Ai to automate this kind of thing will make it even worse probably.
Cool story if everything you have has an API or code based. Try doing it on hundreds of switches and other embedded devices. The whole 42 day thing they’re floating is gonna be a massive nightmare because they don’t realize all the other things out there that use certificates.
What makes you think I don’t do this on embedded devices? I’m not about to dox my self with specifics, but I do this exclusively for embedded hardware as my job. We even do it for devices not directly attached to our network. It’s really not difficult so long as you have control of your enterprise hardware (which, you should, unless your management is terrible at their jobs). Hell, even the routers we use have this functionality built in, failure alarms and all.
If this is a problem for you, it’s probably at an organizational level, and not a technical issue.
I’m young enough that I never had to experience anything but let’s encrypt/ACME. Manually renewing certs sounds like such a major PITA that I’d switch to it as soon as I could…
I had to do it a couple of times, major pain, mostly as you do it that infrequently that you forget everything that you have to do to enable and change them
Yeah. Oh it’s 10 years until we gotta replace it. That’s someone else’s job
There are tools to actually remind you to do this on a timely fashion… Also, some of them go as far as doing auto renewal. Is this such a hard thing?
You’d be surprised. DevOps are, at least by my experience, SEVERELY overworked and understaffed. Even such things as writing a script is not always so easy, especially when security credentials are involved. Depending on the company there may be many layers of red tape for using security credentials stored in a secrets vault, so sometimes such things aren’t even possible since there is no official work request for that automation.
A lot what you mention is for sure true. My problem usually is deferring infra work to people that should not do it. More often than not I see so called “devops teams”, and once I see the elements on those, is very clear people that are actually for infra are either insufficient, missing, or straight up not enough permissions to do stuff. Yo would be shocked how many times I hear managers say " ah well, I bet one ofb our developers knows how to do infra work"
If the reminder goes to someone, who was fire two months ago or someone on holiday, this can easily fail.
And if the reminders are not set on a mailing list and you have no one looking at that, someone failed hard
Or it goes to a mailing list, but none of the readers thinks it’s their responsibility.
until one of your employer’s multi-million dollar customers insists on a commercial certificate so it’s a yearly effort to buy and distribute the damn thing