Fuck it. People in the harem don't have to be cared for by Y/N. They can have a rough past but they could be also living a happy life at the moment.