The Doctor explicitly tells River and Rory to keep an eye on at least one Silent when they arrive in the fake tardis. As long as they can see one, the effects of the memory loss do not take effect.

Don't ask me about how River shot the Silent behind her though.