You are perplexed because, as far as you were concerned, all the signs were there. Unfortunately, you are using signs applicable to Western women. Although it is changing, direct eye to eye contact is, to the contrary, considered quite rude in Japan.
