So no, not ruddy-colored -- rough. Also, not rude. The rough guideline, so to speak, is "larger than grains of sand." This is a modern coinage (by geologist A. W. Grabau, from Latin root rudis coarse, rough) that replaced a much older synonym, arenaceous, which had the general meaning of sandy (borrowed in the 1640s from Latin harēnaceus, from harēna, sand) and was also used botanically, as in describing things that grow in sandy soil. Contrast with argillaceous, fine-grained (also from Latin, meaning clayey, borrowed around 1730).
When I described the ancient fort as "rudaceous," I did not mean that the stones were roughly cut, for indeed they stones were well-dressed and tightly fitted, but rather that the stone was rough-grained.