It depends on how you use your phone and what the physical attacker aims

• if you use a custom ROM with decrypted /data partition by default and no way to encrypt it, the attacker can get access to all of your data from recovery even if you’ve set a lock (like password/PIN/pattern) in the ROM, but if your custom ROM is encrypted and protected with a lock, the attacker must know your password to decrypt /data partition in recovery
• if the attacker aims to replace a part of your phone with a sus one (like a boot partition for example), he must be a developer who knows how to build things designed for your exact phone model, otherwise your phone will get bricked
• if your phone is rooted and you give root permission to sus modules and apps, it’s possible to install malware and do shady things in it without physical access

My recommendations:

• only use trusted ROMs
• only use an encrypted ROM ( official LineageOS is encrypted if I’m not wrong) , encrypted ROMs are slightly slower than unencrypted ones, but safer
• set a lock to the ROM
• avoid giving ROOT access to untrusted modules and apps
• (if you’re paranoid) clean flash every time you update or switch ROMs, as this will replace any sus partition flashed by an attacker
• (if you’re using decrypted ROM and custom recovery) set a password to the recovery, BUT if it’s orangefox make sure to remove the password before updating the recovery, otherwise you’ll get troubles