Fear of pain
I think our fear of pain is so great that we end up being cruel because of it.
What I mean by this is that rather than dealing with it and fighting it, we choose to avoid it, unless it is our own pain and one that cannot be avoided.
Let me pick some examples. Think of most villains in tales and movies. They are often victims of some sort of violent crime in the past. Examples include the woman/man who has acid thrown on his face and his face is damaged because of that. The person whose face has been burned, one who has a huge gash on his/her face because of a knife/axe wound. Shouldn't we be sympathizing with these people and loathing the people who caused these horrible wounds to these people. But no, instead, we make them the villains and loathe them. To justify our hatred we make up stories of how these people were evil in the first place and they deserved this disfigurement. Some other times, we make up stories of how these people were fine at first, but then, now they are frustrated because of their misfortune and take out their frustration by hating and attacking the whole people. Whatever be the case, the subliminal message is "disfigured=bad". Perhaps the real reason is not all this, but that their disfigured face reminds us of pain, a thought that we don't like to have and the real reason we are angry with them is for inconveniently (for us), just being alive and making us face these unpleasant thoughts. Perhaps this kind of sentiment also explains the old tendency in movies for disfigured victims to be killed (often by "accident") in the story so that we didn't have to fell uncomfortable about how miserable society was going to make their lives. Another example here is the frequent ending in movies of an earlier era where rape victims took their own lives so that we could continue ours with a "too bad..that really sucks".
There are more examples, in the form of villains suffering from various diseases or having mechanical limbs in them. In real life, of course, I see little connection between people with various physical impairments and their being "bad people" in any way. If anything, they are kinder for they understand the value of kindness better.
Then there is the regular stigmatization and labeling of the victim of violent secual crime - a female rape victim is "not chaste" or even "a whore" based on which society we are thinking about. Even in very liberal societies she may be viewed as not worthy of our company because of "having issues". A male victim of rape might be labelled a "homo" in homophobic circles (which itself is viewed a bad thing). Sometimes an air of dirtiness is attached to the victim and it is as if the person needs to be ashamed of the themselves for what happened with them, almost as if they chose it. Remarkably commonly, the perpetrator is not ostracized nearly as much.
I have no expertise whatsoever in psychology and none of what I have said is informed by anything but personal observation and listening to others' experiences and thoughts. I don't have any statistics to back any of this up. But I do not wish to make a case here (maybe another day). So that's alright. I am content with describing some thoughts. It does appear to me that our basic tendency to not want to even look at pain leads us to avoid providing support to victims and in some cases, even to turn against them. Of course, strong individuals fight these tendencies within themselves and societies as a whole have made significant improvement in dealing with these issues. For instance, many organizations have made serious strides, either voluntarily, or forced by laws (awakening societies) in not hushing up matters and in providing protection and support to victims of crimes and abuses. Nevertheless, the desire to project "that kind of thing doesn't happen here" is in constant struggle with "there is a problem and we need to fix it".