I think that if Voldemort really wanted to kill Harry Potter the night the spell didn't work on him, he could've just picked him up and thrown him out a window given the fact that he was a one year old infant.
This is why wizards are fudging useless.