"....and generations of inbreeding made them violent and unstable"  It is likely they probably just didn't care for all the 'impurity' they would encounter at. Hogwarts; even in the Slytherin house there there might be half bloods or muggle borns. Since Hogwarts has a fund for students without funds (such as Tom Riddle Jr.) and most first year students don't seem to be able to do anything at first when they arrive at school, it is not likely that they would pass up the chance if it was because of poverty or lack of magical skill.
Education at Hogwarts was never mandatory (until Voldemort captured the ministry of magic, but that was quickly repealed when he was killed). Parents always had the right to educate their children at home. Presumably, that was what the Gaunts did. Though their insanity, mania, and instability made for poor instructors, and combined with Marvolo's dislike of his second born daughter, she could only learn through imitation, and her abuse made her a poor witch.
Ironically, Merope's lack of skill made the manic Marvolo believe she was a squib, and his constant abuse of her for this made her even worse.