This is dedicated to @blueunicorns2002 who came up with this reason.

Voldemort was a mouth breather so his nose got sad and fell off/ran away.

~District_Gryffindor