Unicode is good. If you’re designing a data structure or protocol that has text fields, they should contain Unicode characters encoded in UTF-8. There’s another question, though: “Which Unicode characters?” The answer is “Not all of them, please exclude some.”

This issue keeps coming up, so Paul Hoffman and I put together an individual-submission draft to the IETF and now (where by “now” I mean “two years later”) it’s been published as RFC 9839. It explains which characters are bad, and why, then offers three plausible less-bad subsets that you might want to use. Herewith a bit of background, but…

Please · If you’re actually working on something new that will have text fields, please read the RFC. It’s only ten pages long, and that’s with all the IETF boilerplate. It’s written specifically for software and networking people.

Source code · I’ve written a little Go-language library to validate incoming text fields against each of the three subsets that 9839 specifies, here. I don’t claim it’s optimal, but it is well-tested.

Details · Here’s a compact summary of the world of problematic Unicode code points and data formats and standards.

Notes:
[1] XML allows C1 controls.
[2] XML and YAML don’t exclude the noncharacters outside the Basic Multilingual Pane.
[3] YAML excludes all the legacy controls except for the mostly-harmless U+0085, another version of \n used in IBM mainframe documents.