Genesis 40:4

40:4 The captain of the guard appointed Joseph to be their attendant, and he served them.

They spent some time in custody.

Genesis 40:7

40:7 So he asked Pharaoh’s officials, who were with him in custody in his master’s house, “Why do you look so sad today?”