I have a question for you. Would it be easy or difficult for you to speak to another person in your target language if that person has the same native language as you? As for me, I find it impossible to speak to someone in Japanese if they can speak English well. They don't even have to be a native English speaker.
I think what that means is, that I'm not at the level where I can comfortably speak Japanese. Why would I want to struggle here and stumble there when I could just say it in English? Speaking my target language, when I don't need to be, feels very artificial. It's very limiting.
I'll only be able to speak Japanese to another English speaker when I can do it without thinking and when the words just come to mind and come out of my mouth without me really realizing that I could be speaking English. Unless of course we have some kind of agreement to speak Japanese. That would make it a bit easier.