There's a bit of hesitation. Not because of the concept of Phasma residing with him, because he actually prefers that. There's no point in having one of his few allies here living somewhere else. They're stronger together than apart.

At least, Hux feels they are, for obvious reasons.

"Not exactly, at the moment. We found ourselves in a predicament not too long ago..."

See: they done fucked up, Phas.