Huge number of users/groups without inheritance can even cause acl trackers to fail as well due to heavy payload being sent to solr nodes. " --> When you a lot of folders/files where you have disabled the permission inheritance, then it would degrade the performance of queries
It is recommended to make more and more use of groups in this situation and add users requiring appropriate permissions to the groups.
Write a script/webscript to find out nodes where you have disabled the inheritance, analyse and tune it accordingly.
Admin bypasses the permissions that's why you get proper results.
Enable following logs and see if you can find something:
Also have a look at this tech talk on permissions:
(ACSCE, AWS SAA, Azure Admin)