I agree, remove the excluder, for at least a week. After that, make sure you check the queen is downstairs before putting the QE back.
Also, if brood in the super bothers you, have a quick look in the super frames for eggs/larvae which might have been laid in the last week. If you find them...