"The only downfall of this method is that once you’ve set the password, you’ll never need to enter it again, so long as you’re logged in as you."
There is no need to save the password to your Keychain, even if it is the default state of the checkbox. Just uncheck that! :)
~eric
AAM: Creating a Password Protected Folder