The Ancients' cutscene says "We encountered other advanced life, and we subdued it or we crushed it." Perhaps the Ancients visited Vasuda Prime when the Vasudans were still very primitive, and enslaved them instead of eliminating them. Later, when the Ancients withdrew to their home system, it's possible that the Vasudans learned a little bit of what was going on. Afterwards, the incomplete facts became myth and legend.
Consider, from the Vasudans' point of view:
1) Real powerful aliens arrive, take over the planet, and enslave us.
2) The aliens tell us glorious stories of their empire and their people (not outside the realm of possibility).
3) The aliens wage war with the destroyers.
4) The destroyers start winning.
5) The aliens evacuate our planet because the destroyers are coming.
6) Nothing is heard from the aliens or the destroyers again.
(We can speculate that the Shivans ignored Vasuda Prime because nobody was travelling through subspace after the Ancients left.)
So the last thing the Vasudans hear is that "the destroyers are coming". When nothing happens after a number of years, that possibly transforms into "the destroyers came after the aliens, and they will come after us when we become like the aliens".