The servant was looking for fun with his owner when her husband was out of the house