When I watched the episode again, I realized that the men he killed were the masked zealots who helped Agnes kidnap Sophie. So they were just as culpable as she was. But yeah, it wasn't really clear, since they were all wearing MASKS in the kidnapping scene.