Registering late has two consequences and the first is much larger than the second. If you register late you must pay VAT on any sales you have made since the date you should have registered, whether or not you charged it to anybody. That is the real cost, and on a consumer-facing business it comes straight out of the margin because those customers are not going to be re-invoiced.
The VAT comes first
The obligation runs from the date registration should have taken effect, not from the date you noticed. Every taxable sale since then is treated as VAT-inclusive if you did not charge VAT on it, so at the standard rate a sixth of everything you invoiced in that window is HMRC's. You can reclaim input tax for the same period against it, which softens the total, and getting those purchase records together is the first useful thing to do once you realise.
Then a penalty, depending on how much and how late
A penalty may be charged for failing to notify, and it depends on how much you owe and how late the registration is. HMRC's guidance on the late registration penalty sets out how it is worked out. The practical point is that the penalty scales with both the tax at stake and the delay, which is why registering the moment you notice is materially cheaper than waiting until the next quarter or until HMRC raises it with you.
The exception and the exemption, which are different things
If turnover went over the threshold only temporarily you can apply for a registration exception, and HMRC will consider it and write to confirm; if they do not grant it, they register you. Separately, if most of your taxable supplies are zero-rated you may be able to apply for an exemption from registration, which needs HMRC's permission. Both are applications made deliberately, neither happens by default, and neither is a reason to delay a registration you already owe.
Questions people ask about late vat registration
What happens if I register for VAT late?
You must pay VAT on any sales made since the date you should have registered, whether or not you charged it, and you may have to pay a penalty depending on how much you owe and how late you are.
Can I recover the VAT from my customers afterwards?
Business customers can often be re-invoiced and will reclaim it. Consumers generally cannot be, which is why late registration usually lands on the seller's margin.
What if I only went over the threshold for one month?
You can apply for a registration exception. HMRC considers it and writes to confirm; if it is refused, they register you.