So the bearings were clearly too tight before and that was just not right. So today I pulled the entire machine apart and made sure to shim everything properly with aluminum foil to get those last 1/100s.
I also realized that my cheating with the bearing fasteners for the Y axis had come back to haunt me as the Y axis was both too tightly shimmed and still wiggled at the same time. not sure you can see it in the image below, but the pin does not go all the way through meaning the axel is not fastened symmetrically in the bearing holder.
So nothing else to do but pull these pieces apart and refasten everything correctly.
Now this is how a bearing should behave. 😀
Finally all pieces are back together and the wiggle is now down to a consistent 5/100. Given that many of the solid parts have a flex of about 1/100, I think this is as good as it gets.
