I had an experience similar. The way I handle it is: I try to get consent of all involved characters' owners for the drawing before starting the commission. After that if later anyone wants the image removed from my gallery it's my decision based on their case, but they can't force me to do anything.

If a person's character was drawn by you without their consent, then they have a case and maybe you should straight out accept the request for taking it down.

Basically my advice is to get consent from everybody involved before making the art, after that they can ask but it's only your choice to keep or remove from your gallery.