Your experience is, in my opinion, the principal reason why I dislike performing shops where you must take photos surreptitiously or secretly. There always seems to be someone who sees you, and, in rare instances like yours, becomes angry.
For some of the shops, that are secret and then revealed, like the "Green" and "Yellow" gas stations, we, as shoppers, are provided "official" nametags that, for the most part, seem to more pacify these "concerned" photo sensitive folks.
I wish the "Blue" gas stations offered them.
Although, in your described experience, I doubt anything would have appeased the man's concern.
Sometimes I wonder if some of these angry people are upset about other things and act out their rage at seemingly explainable incidents.
I'm sorry it happened to you.