Using Swift on server side is an attractive idea, especially its memory management scheme, ARC, makes GC based languages including Java, Ruby, Go simply too bloat.
However, server side apps and client side apps have very different priority set. For most businesses, spending extra thousands of $$ on RAM is not a big deal. Reliability, constant performance, rich 3rd party library supports, cross language inter-operativiblity, and backward compatibility are far more important factors than cool language features. Just look at the Go VS. Java drama, it's clearly Go is a better tool for server side programming, the adoption rate is still far below where it should be.
Another factor is related to the vendor. My theory is an open sourced, backed by dedicated reputable vendor would greatly increase the market share. For server side business users, it's a great insurance there is a reputable company charges for services. In this regard, Google has its fair share of blame not to support Golang as aggressively as it could. Google is reputable, its technology rocks, but not charging customers also means no commitment. That's scary. In this regard, users would be very sceptical to adopt Swift on the server side: Apple doesn't show any interests in a server side app business (although historically, Apple had a server side product WebObjects), Apple is famous for frequent technology deprecation/updating, and Apple even made no promise to maintain a stable API.
Having said all above, I'm still interested in adding Swift into my server side toolkit. Maybe, the rising of Micro-Service provides such opportunity for Swift to the game. Welcome, Swift!