I don't have the book handy, but the prophecy is quoted, and says: Born to those who have thriced defied him; (Gotten away from Voldemort three times) Born as the seventh month dies; (At the end of July) The Dark Lord will mark him as his equal; (He attacked Harry giving him the scar)
Up until the last part, the prophecy could have referred to Harry or Neville Longbottom, but Voldemort "marked" Harry, not Neville.
The prophesy could have referred to either Harry or Neville. But Voldemort didn't hear that last bit, about marking the child as his equal. He didn't 'know' it referred to Harry, and in fact, it really didn't. He just guessed that it did, and in effect, made it true. It only refers to Harry because he made it refer to harry.
He went after Harry because he was a half-blood. He feared another half-blood prodigy, like himself or Albus.