A deal for the United Nations to start work on a remote Bangladeshi island where the government has sent thousands of Rohingya refugees offers no guarantee they will be allowed to move freely to the mainland, according to a copy of the agreement.

The Bangladesh government has moved nearly 19,000 Rohingya refugees, members of a persecuted mostly Muslim minority from Myanmar, to Bhasan Char island from border camps, despite protests by refugees and opposition from rights groups...