Im sorry for your bad experience, as I'm also the victim too.
To be truth from what i learned in gta online, reporting player hostile is not what we expected. The reason for reporting is to list down which player is hostile.
As in gta, rockstar will pick anyone from this list to spawn in our solo session, exactly when we start selling cargo. hardly to say that is coincidence, because happen all the time. I assume same system is apply in RDO. Those player wont be punished, as the game itself encourages them to do so.
Most of time i just set myself to defensive mode. Not because I'm scare with them, but i just want my gaming time is worth to spend after working. Rather than having bad headaches with them.