I think it has something do with the realistic element. Sci-fi is within the realms of possibility, if not probability. Fantasy hasn't enough rules for my liking.
I recently came across the distinction between sci-fi and fantasy worded perfectly by my hero and sci-fi legend Philip K. Dick. His definition comes from a letter he wrote in 1981, now serving as the preface to Beyond Lies The Wub:
Fantasy involves that which general opinion regards as impossible; science fiction involves that which general opinion regards as possible under the right circumstances. This is in essence a judgment-call, since what is possible and what is not possible is not objectively known but is, rather, a subjective belief on the part of the author and of the reader.
Philip K. Dick (May 14, 1981)
Reading this made me reevaluate what it is I'm writing and I guess I'm writing part-sci-fi part-fantasy. There are elements I regard as possible and those I regard as impossible in my story. Arse.