Former prime minister Dr Mahathir Mohamad has lamented that those who had complained to him on various matters seemed to have "crossed over" to the other side.