I don't understand why the power button would have an effect. If it was just that the computer didn't fully boot and the icons never came up, I'd suspect a bad driver or a hard drive problem. But you say if you hold the button down a little longer, the computer boots normally.
I've had computers not fully boot when their power supplies were starting to fail, but they never got as far as the desktop. In those cases, letting the computer fail the first attempt, sit there stuck but still running for 10 or 15 minutes , then rebooting would often result in a successful boot. Of course the real solution was to replace the power supply.
Do you have any peripherals connected by USB that might be trying to power up after you get to the desktop?