Do not take out, without a substitute, the initial proficiency bonus.
The reason to have proficiency bonus at all is because it is a measure of experience and expertise in some areas. What do I mean by that?
In the current system, a wizard and a fighter can both use swords with no need for training at all. The difference is that the fighter is trained with the use of those martial weapons, therefore, the fighter can add the bonus that represent his knowledge on the use of the weapon to the attack roll. Without the proficiency bonus there is no difference between a wizard with a sword and a fighter with a sword (aside from class bonuses).
But proficiency bonus do more than add things to the attack roll. They help with saves and skills rolls. For some DM they can be used to allow or block players to try an action, for example, trying to lock-picking a seemingly complex lock, something that one without the "training" would not be able to do no matter how good the roll is. The logic behind that would be, is it a fairly easy task? everyone can try it, do you need an expert? Sorry, you are not. But the validity of that is up to the DM and outside of the scope of this answer.
The game is balance around the proficiency bonus
The game takes into consideration the proficiency bonus and put the challenge accordingly. This is more evident at higher levels. If we take the example of the wizard and the fighter and pit them with melee weapons against a monster with a moderately high AC (20), the difference is obvious.
With proficiency bonus of 6 at 20
- Fighter at 20 (Strength 20 [+5]) vs AC 20 would need to roll 9 to
bypass the AC.
- Wizard at 20 (Strength 10 [+0]) vs AC 20 would need to roll a natural
20 to bypass the AC.
- Wizard at 20 (Strength 20 [+5]) vs AC 20 would need to roll 15 to
bypass the AC (That is a beefy wizard)
Without proficiency bonus
- Fighter at 20 (Strength 20 [+5]) vs AC 20 would need to roll 15 to
bypass the AC.
- Wizard at 20 (Strength 10 [+0]) vs AC 20 would need to roll a natural
20 to bypass the AC.
- Wizard at 20 (Strength 20 [+5]) vs AC 20 would need to roll 15 to
bypass the AC (our beefy wizard again)
As you can see, the difference between the wizard and fighter is clear with the proficiency bonus while without it is blurry at best.
+1 is a big deal, therefore, taking out the initial bonus is even "worse"
Taking out the initial bonus will only increase the difficulty of the encounters and task that require rolls. This is because all creatures has this bonus in consideration and has its own bonuses (Player Basic Rules 57). There are calculation out there for what is called "bounded accuracy" that is very relevant to this topic, a link here.
Monsters also have this bonus, which is incorporated in their stat
blocks.
For the weapon part, having a +1 is an advantage versus any creature and much more of a big deal than in previous editions. Also, D&D 5e has the philosophy of low tier magical items in its design, making them much more valuable than in precious editions.
Everything can be adjusted if it feels too easy or too hard
If you find that the encounters are too easy or hard, adjust them. That is the best you can do to provide a challenge to you and your players. A little Homebrew comes with 5e.