Roku TV Headpone Mode will only play on my phone's speaker.

I have my earbuds connected to my phone. If I play a video on my phone the audio comes through my earbuds, but if I play it on my Roku with Headphone Mode the audio comes through my phone's built in speaker instead. Can anybody please help?