Is it unrealistic to think that we would not end up having an affair?

I think his marriage is a little rocky at the moment and he's talking to you because he needs someone. He doesn't care for you like you do him. He has a wife. There is a reason why YOUR not his wife.