Constraining an entire surface makes it stiffer than it would be otherwise. And, as was mentioned above, there's no way to model the interaction with the surface other than creating that interaction in an assembly. If you are getting a singularity, you determine the solution is not converging (p-loop pass, max element order, etc.), and you don't care about the stresses in those areas, then you can can tell autogem to ignore the stresses in those areas for the purpose of convergence...try some mesh refinement by hand in the area first though.
A contact analysis will be more accurate than just constraining surfaces to represent interaction with other components. Making the bolts infinitely stiff is a slight deviation from accuracy to the real world. You can improve on that if you can change the bracket material from linear elastic to at least elasto-plastic. Furthermore, creating a radius on the bolt where it interfaces with the part may help a little...don't reduce the contact diameter though. You could also look at constraining the part in a different way such as fastener features or weighted links, etc. I'd like to hear if anyone has had success with another method.
Ultimately it comes down to engineering judgement and what you deem is important. Your definition of a failing part may be different than others. Bearing failure is common and a reason for spontaneous bolt losening, corrosion, etc. But that may not necessarily deem it a failure in your application. On the analysis side, it helps to get a second opinion either with a hand calculations, another anaylsis from a different approach, or validation through actual testing.