A Caddy Cert Expired Because Systemd-Resolved Was Selectively Broken

5/12/2026 at 5:29:39 AM

What's the server's hostname? If it's the same as the domain that's not working, perhaps you're running into this bug [0]? If that's the case, then adding

  # /etc/systemd/system/systemd-resolved.service.d/override.conf
  [Service]
  Environment=SYSTEMD_RESOLVED_SYNTHESIZE_HOSTNAME=0

might fix it.

[0]: https://github.com/systemd/systemd/issues/34897#issuecomment...

by gucci-on-fleek

5/12/2026 at 5:03:13 AM

There's this adage about consuming whatever newspaper reporting, everything seems fine and dandy, until one day they report on something in your field. Say, chemistry, and you're a chemist, and... it's all wrong. Even the basics.

My field is networking, roughly from Ethernet physicals to TCP/UDP.

systemd has no f*cking clue what they're doing on networking. You need to not use systemd-resolved, and not use systemd-networkd or systemd-timesyncd either.

I really hope they are service manager people and know how to write a service manager, and I have no counterindications on that. But don't let them touch your networking, aside from service-managing on that.

(My recommendation for DNS - on servers - would be to install unbound locally and use that. It's not great for clients since it doesn't deal well with frequently changing network connectivity, or rather, it does, but only if you restart it on network attachment change, which implies flushing all cached data.)

by eqvinox

5/12/2026 at 5:24:42 AM

> systemd has no f*cking clue what they're doing on networking. You need to not use systemd-resolved, and not use systemd-networkd or systemd-timesyncd either.

Can you give some more details, or do you have some examples of where systemd is bad here? Because I'm quite happy with systemd-resolved and systemd-networkd, but I admittedly know hardly anything about networking.

> [...] But don't let them touch your networking, aside from service-managing on that.

What do you recommend as an alternative then? systemd-resolved and systemd-timesyncd are easy to replace, since there is lots of good DNS/NTP software available, but systemd-networkd seems trickier: I don't like NetworkManager, I'd rather not write a bunch of ad-hoc shell scripts, and I'm not aware of any other alternatives.

by gucci-on-fleek

5/12/2026 at 5:41:54 AM

> Can you give some more details, or do you have some examples of where systemd is bad here?

I can't, because it's death by a thousand cuts. I remember systemd-timesyncd having security issues, but not what exactly. systemd-resolved was initially missing basic security best practices (source port randomization, if I remember correctly), despite their being well established and well known in the DNS community - AFAIK the systemd people just didn't know of them.

> systemd-networkd seems trickier

Indeed; I was primarily speaking about servers, and there my answer is: either ifupdown-ng or ifupdown2. (The latter if your networking needs are more complex.)

On client devices the choice is between bad and worse. Personally speaking, I consider NetworkManager bad and systemd-networkd worse. So I'm using NetworkManager on my laptop... it still occasionally breaks IPv6 for me.

I will admit: this is burned-in experience for me, learned over several years, and the original reasons have expired from my brain's cache. And I probably didn't give systemd-networkd much of a chance (I don't remember anything on that, for resolved and timesyncd I have at least vague bits), with that being more recent (at least for me) than systemd-resolved and systemd-timesyncd.

Is it possible it has gotten better? Definitely. But TFA indicates at least systemd-resolved hasn't.

by eqvinox

5/12/2026 at 8:40:50 AM

> systemd-resolved was initially missing basic security best practices (source port randomization, if I remember correctly), despite their being well established and well known in the DNS community

https://lists.dns-oarc.net/pipermail/dns-operations/2016-Jun...

It was fixed in 2016. RFC5452 is 2009.

As the first paragraph states it's not a big problem for a local forwarder but all other bullet points are on the case.

by justsomehnguy

5/12/2026 at 8:09:51 AM

A lot of knowledgeable people hate systemd, and I start suspecting that they have good reasons for that. However, it's extremely hard to get a good list of related substantiated facts about systemd in order to convince a larger community. If you encounter something specific, please write some kind of article about it.

by fsflover

5/12/2026 at 11:20:08 AM

You're right in that I should really be able to back my position up better. I don't know if/when a situation where I have to deal with the systemd network components will come up though :/

by eqvinox

5/12/2026 at 8:25:49 AM

> in order to convince a larger community

Even if you would get such a list it would be dismissed anyway, because 'works on my machine'. Eg: see a sibling response to the OP.

by justsomehnguy

5/12/2026 at 2:26:13 PM

If I ever have a networking problem in a personal Linux system, it gets solved by removing the weird networking thing that the distro decided to use and move to full systemd-resolved / networkd. In my experience, that works every time.

I'm not doubting commenter's expertise. It's entirely possible that resolved and networkd are utter horror inside but still work from at least this user's POV.

by vga1

5/12/2026 at 8:56:40 AM

hot take "predictable network names" should have been a kernel flag - give us all our eth0 in peace. i shouldn't need to set a flag to get back a default feature.

just what i needed to do: configure ens328452356aflhjdslhfda to get my network going.

by kotaKat

5/12/2026 at 10:40:13 AM

ln -s /dev/null /etc/systemd/network/99-default.link Or net.ifnames=0 Take your pick...

by nubinetwork

5/13/2026 at 10:37:53 AM

net.ifnames=0 should be the default: if someone needs it to be the stupid ENS nonsense, they should be the ones forced to set it to 1 at the command line, not me.

by kotaKat

5/12/2026 at 11:40:09 AM

> hot take "predictable network names" should have been a kernel flag - give us all our eth0 in peace. i shouldn't need to set a flag to get back a default feature.

No, because the feature being on by default does nothing but make your interface names longer, but being off can be the difference between a restarted server having or not having network access because of undocumented behaviour with it's firmware causing the network interfaces to come up in different orders.

by ChocolateGod

5/12/2026 at 1:06:39 PM

Personally, I find the predictable network names feature infuriating 100% of the time. Changing the name of network interfaces made nothing better, and actively made things worse for systems with a single NIC (virtually none of the systems I use with a single nic use the same name). Network interface configurations could already be bound to MAC addresses rather than names, and that had been implemented even before the enpXXX style names came into being.

Persistent ethX names were far better to me, and we had those before enpXXX via udev. At least then when I logged into a random system with a single NIC it was called eth0. With 2 NICs they were eth0 and eth1. Simple to predict, no thinking required (unless the system already had bound those names to other MAC addresses).

Usability dies by a thousand cuts. Forcing needlessly complex behaviour onto people with simple use-cases is not an improvement.

by bcrl

5/12/2026 at 5:42:22 PM

< Persistent ethX names were far better to me

They weren't persistent, that was the problem.

by ChocolateGod

5/13/2026 at 10:15:48 AM

eth0 is predictable: a given system will always have the same interface for the same physical hardware

enp0s31f6 is unpredictable

The whole operation is a lie!

by JackSlateur

5/12/2026 at 11:17:56 AM

> My recommendation for DNS - on servers - would be to install unbound locally and use that.

At least on Ubuntu 26.04, you can't easily bind Unbound on a WireGuard interface because of services dependency order, so the Unbound service errors during system boot because the WireGuard interface doesn't exist yet. And IIRC neither `ip-transparent` nor `interface-automatic` fixed it.

On Alpine Linux all this just works.

by quectophoton

5/12/2026 at 6:45:39 AM

> There's this adage about consuming whatever newspaper reporting, everything seems fine and dandy, until one day they report on something in your field. Say, chemistry, and you're a chemist, and... it's all wrong. Even the basics.

Gell-Mann amnesia effect: https://en.wiktionary.org/wiki/Gell-Mann_Amnesia_effect

> systemd has no fcking clue what they're doing on networking. You need to not use systemd-resolved, and not use systemd-networkd or systemd-timesyncd either.

I'm not sure how to reconcile that with enjoying each of those:

systemd-resolved is the default on many distributions, and does DNS-over-TLS stub resolution for me. What do you suggest as an alternative that handles changing network connectivity, and DoT?

* systemd-networkd is very well documented/organised. I find it pretty useful on servers.

* systemd-timesyncd doesn't seem to cause issues for me. It being an SNTP client (rather than NTP) seems appropriate for distributions. I would like to move to NTS or Roughtime at some point.

by farfatched

5/12/2026 at 8:15:51 AM

> My recommendation for DNS - on servers - would be to install unbound locally and use that

And now your developers are running around and cursing you because nothing works anymore.

Because Docker silently retargets the interna Docker resolver to 8.8.8.8 if it sees 127.0.0.1 as a resolver address on the host.

Because people who wrote Docker have no fucking clue how the system works.

NB: see https://news.ycombinator.com/item?id=47441785 to solve the Docker issue with a local resolver.

> It's not great for clients since it doesn't deal well with frequently changing network connectivity

This is is something Linux specific I guess, I run Unbound locally on my Windows laptops for years and never had a problem which would require the Unbound restart.

> which implies flushing all cached data

It doesn't really matters in 2026. Just look in your cache and note the default TTLs for like 90% of records.

by justsomehnguy

5/12/2026 at 11:18:51 AM

Running on 127.0.0.53 isn't a systemd-resolved exclusive magic trick, you can just configure unbound to do the same thing.

by eqvinox

5/12/2026 at 9:00:36 PM

Not 127/8

by justsomehnguy

5/12/2026 at 11:00:10 PM

by eqvinox

5/13/2026 at 8:32:19 AM

You need to configure your resolver to anything but 127.0.0.0/8 to not to trigger Docker automatic "help".

Most of the time local resolver is bound to 127.0.0.1 just because it shouldn't be available from the network and because this is how it's always done.

The problem is what Docker actually silently break things because it's devs are too good to bother, the same thing you note about systemd devs.

Hell, it was easier to just provide internal Docker resolver on a some network Docker uses internally by default for the NAT shenanigans - but Docker devs 'knew better' and instead made it to point to a 3rd-party absolutely not related service which brokes local resolution on the containers.

by justsomehnguy

5/12/2026 at 9:32:06 AM

Some docker issues archaeology says that there was tension between docker team and redhat / systemd

by zaphirplane

5/12/2026 at 11:33:58 AM

That's probably why podman exists.

by burnt-resistor

5/12/2026 at 11:41:57 AM

It is why Podman exists.

by ChocolateGod

5/12/2026 at 9:05:30 PM

Yes?

RH wanted to control the stack and Docker was the thing what sidelined RH to a mere distro provider. And don't forget about RH dips into OpenStack at that time.

by justsomehnguy

5/12/2026 at 4:19:37 AM

There was interesting commentary on https://lobste.rs/s/z0ozbb/caddy_cert_expired_because_system...

by farfatched

5/12/2026 at 11:50:25 AM

Was there? I see 14 comments, of which only 3 have any value at all which are: fanf finding an irrelevant bug, oliverpool explaining the use of LetsEncrypt staging, and hexadecimal suggesting setting up expiration alerts.

The rest is:

  - People bitching about systemd and replies to their bitching totaling 7 posts (literally half the comments) 
  - conversational detritus like "well written blog post" and the whiteboard comment

by tom_alexander

5/12/2026 at 2:01:59 PM

[flagged]

by ddactic