The Shadow of Unfairness