I agree with your first statement. It seems that the controller and the phone are fighting for power. I have taken off the guards to lighten the drone and it is a lot better.
I am using the Co Drone Pro controller (build your controller with exposed wiring).
It leaves the ground but seems hit and miss when I try and control it via controller.
The drone needs to be < 2 metres away or else the drone isn't responsive.
I'm thinking maybe I need a stronger battery?