I think Apple stopped loading the passphrases into the agent automatically. I think they're strictly only loaded from the keychain now, per connection, unless you explicitly use -A to add them to the agent.

Edit: Ah, and another comment here provided the answer: https://news.ycombinator.com/item?id=12654917