Boris Johnson refuses to address row with girlfriend