During the course of the narrative, facts are often implicitly (or explicitly!) established about the world. These facts do not always map cleanly to either characters or locations, and characters and locations are not always visible to the players at all times.

I would suggest allowing the Narrator (and possibly the players) to create Fact cards- cards which do not map directly to challenges, but capture established facts about either the world or relationships between characters.