The Sharpshooter feat works with Improvised Weapons just fine.
You cannot take -5 to hit to gain +10 to damage unless you are proficient with the improvised weapon, however. There are two general ways this can occur.
a) The GM rules the improvised weapon is similar to a listed weapon that you are proficient with, such as a thrown rock being similar to a sling's damage, but with shorter range. They are encouraged to do this.
Often, an Improvised Weapon is similar to an actual weapon and can be treated as such. For example, a table leg is akin to a club. At the GM’s option, a character proficient with a weapon can use a similar object as if it were that weapon and use his or her Proficiency Bonus.
An object that bears no resemblance to a weapon deals 1d4 damage (the GM assigns a damage type appropriate to the object).
'Often' being the usual case, and 'no resemblance to a weapon' being required for it to be treated as a generic improvised weapon. So generally speaking, when a character snatches up a fireplace poker and tries to stab someone with it, that's a Club, and when the barbarian picks up the fancy glass sculpture and belabours his foes with it, that's an axe.
b) The Tavern Brawler feat. It grants proficiency with improvised weapons. That refers to the kind that 'bear no resemblance to a weapon' and thus do 1d4. This would cover situations such as a barbarian beating his foes with a duck, or a rogue using a playing card to slice someone's throat.
As a Side Note:
5th Edition of DnD has a common situation where players of the game will invent houserules and justify that as 'interpreting' the existing rules. This has existed in all editions of DnD and many other games but is very prevalent in 5e as it is the gateway to the hobby for many newer players (and has a fairly simple set of rules that stresses randomness and is open to houseruling (much like ADND or DND2e)). This is often used to curb any use of the rules that is deemed 'abuse', which covers anything exotic or strange (such as beating someone over the head with a duck or hurling coins into their eyes). It might seem odd in a fantasy game that people wish to remove the fantastical, but that is often the case.
In situations such as that - where people are 'interpreting' the rules in fairly crazy ways - just be aware that this is their workaround for not wanting to introduce houserules into the game yet wanting to ban or remove something they dislike. Arguing with people in this scenario over the exact wording of the rules or the like will often lead to bad feelings - to resolve the situation, I suggest instead simply accepting their 'view' of the rules and ask for an exception for your character. 'Can we houserule it so I can hurl improvised weapons at people, it only does a d4 so it's not very strong I just like the fluff' will work better than 'well that's not what the rules say so why are you saying they do?'.
People are often more willing to allow something they'd previously thought was 'bullshit' after looking at it again and thinking about the actual math, but are much less willing to admit the rules say something different than they think or decided, as that has additional reputational costs.