Sharing rules are a Salesforce platform mechanism that determines whether a user can see a row in a table (SObject) or not. An example use case is that a manager should see all the rows (records) of the sales people who report to them but each sales person can only see their own rows (records).
Queries done from Apex code that has with sharing will automatically filter out the rows (records) that the User making the request should not see. Very occasionally you may have a special reason to turn this mechanism off by having without sharing; if in doubt leave sharing on.
Sharing is a separate concern from whether the user can see a table (SObject) at all or a column (field) at all; those access rights are controlled by profiles and permission sets. By default, Apex code operates in "system mode" where the profiles and permission sets are ignored. You have to add your own Apex code that makes describe calls to correctly respect the profiles and permission sets. (Note that Visualforce's apex:inputField and apex:outputField do respect the field level permissions automatically.)
Do read the links in the other answers and comments for more detail.