Conditional button card in a horizontal stack messes up its widht

I have a horizontal stack with 4 custom:button-card in there.
However I want to make the visibility of each button card conditional so I wrap them in a default conditional card but that sets the width to an undesired value.

You can see here when there is only 1 button card in a conditional card. The most right one.
image

And if I push it forward it messes up everything because of the 154px width which is not there when there is no conditional card wrapping.

image

While the normal button card is just the correct width:
image

Why does the conditional card decide to be 154px wide?