Malaysian Prime Minister Mahathir Mohamad has submitted a letter of resignation to Malaysia's king, amid talk of forming a new governing coalition.