Only the rakyat can decide to change the prime minister of Malaysia, says Dr Mahathir Mohamad.

In an apparent reminder to Johor Crown Prince Tunku Ismail Sultan Ibrahim, the prime minister said today that the case (his post) was different from that of the prince.

"If we can remember, once upon a time, there was a TMJ (Tengku Mahkota Johor) who was replaced.

"This happened a long time ago, when (the post was instead) given to his brother to become Johor crown prince.

"The prime minister, meanwhile, can only be changed on the rakyat's will," Mahathir told a press conference during his official visit to Beijing.

Read more: Join politics if you want to talk politics, Rais tells TMJ

The premier's comment was in response to a reporter's question over a video that went viral on social media recently, which showed Tunku Ismail making a remark about changing the country's prime minister.

Mahathir was referring to the case of late Sultan Iskandar Sultan Ismail of Johor, then the crown prince, when he lost the status in 1961 over alleged misbehaviour, and was replaced by his younger brother.

His father, Sultan Ismail Sultan Ibrahim, reinstated him as the crown prince in 1981. He was subsequently installed as the Johor ruler.

During the press conference today, Mahathir also said Tunku Ismail can vote him out if he can.

"He is not a big shot (orang besar), he is just a small fry (orang kecil).

"If he can and want to, he can vote to oust me," he added.