I wouldn't really be concerned about the performance differences between the two, for what you are probably doing you won't notice the difference.
What I'd be concerned about is support. XNA has been pretty much all but completely abandoned by Microsoft, it won't be getting any updates, it's not supported by the newest version of visual studio, won't be supported at all on Windows 8, etc.. Any bugs in the current implementation of XNA won't ever get fixed. Your software will also run only on Windows.
SFML on the other hand, is open source, and is actively developped, really popular and really well supported by the community. SFML-2.0 is just around the corner, and is shaping up to be an awesome API. SFML also runs on Windows, Linux and Mac so you can reach a broader audience. Also, if there are any outstanding bugs you can file a report and it will probably get fixed quickly or you can fix it yourself because it's open source.
Another option that is quite good is libgdx for Java, and it runs on Windows, Mac, Linux and Android. It's similar to SFML in many ways, but you can avoid the pitfalls of native programming.