Gloom Stalker is better
I'll get through all the gory details below, but in summary, the extra attacks, Archery and better intiative that Gloom Stalker provides beat the Assassin's extra damage from Sneak Attack and Assassinate on all levels.
I assume the ASI on level 4 was used to boost Dex to 18 from 16. When you play all the way to level 8, you might be able to boost Dex to 20 again for more damage, or you might opt to take some other feat. I made the assumption you max Dex, although it does not change the overall outcome.
Initiative
We are assuming you surprise your opponent, as clarified in comments. However if you can take advantage of that will depend on if you win intiative. If you cannot consistently surprise them, Assassin loses much more (because you cannot Assassinate then, so only single attack simple damage), while Gloom Stalker retains most of its damage, except for the bugbear extra damage. I think in practice, you often cannot surprise opponents, which skews this even more in favor of Gloom Stalker.
Monsters of challenge ratings 0-10 that you mostly will encounter when playing on levels 4-8 have an average Dex bonus of +1 to their initiative roll.
Assassin: With the modified Alert feat, and Dex 18, you have a bonus of +3 from the feat to the roll (+2 on level 4) and +4 from Dex, total of +7. As you win on ties due to your higher Dex, you'll win initiative 77.25% of the time. (73.75% on level 4).
Gloom Stalker. Dread Ambusher allows you to also add your Wisdom Modifier. You have not provided that, but since it is an important Stat also for your spellcasting, assuming standard array and your best stat on Dex, you can get +2 bonus on Wisdom. That means you will have +9 (+8 on level 4) to your Inititive roll, or win 83.5% (80.5%) of the time.
You only can make attacks while the opponent is surprised if you win intiative, otherwise, on the first round, they go first and (doing nothing) end their turn before yours, and with that lose the status of being surprised when you are up, and the extra 2d6 to damage from Bugbear, and the damage from Assassinate's sneak attack.
Attacks
As a Assassin rogue, you get one attack on the surprise round. As a Gloom Stalker ranger, you get an additional attack, with an extra d8 to damage. Also as a ranger on level 5 you will get an Extra Attack each of your rounds.
Raw Damage per Attack
Assassin: You deal d8 + 4 damage on your attack. If the opponent is surprised, you add +2d6 (bugbear), and due to Assassinate, you get to Attack with Advantage which unlocks Sneak Attack to add 2 to 4 d6. All of these dice will get doubled if you hit, again due to Assasinate. If the opponent is not surprised, none of that extra damage happens.
Gloom Stalker: As a ranger, you have access to hunter's mark, which you can cast on your first turn as a bonus action to add 1d6 to your damage. You otherwise also deal d8 + 4 damage per attack, plus 2d6 if the opponent is surprised, plus 1d8 on the Dread ambusher attack.
Chance to hit
We'll assume the standard 65% chance to hit that holds up well across all levels against normal ACs. If you had a magic weapon this would go to 70%. With Advantage, that turns to 87.75% (91%). Gloom staker adds +2 from archery, so will enjoy a 75% to hit chance (93.75% with Advantage).
Assassin, you get Advantage when the opponent is surprised.
Gloom Stalker, you are invisible in darkness and can see in Darkness (due to Umbral Sight), which grants you Advantage to your attacks in dark environments. So your to hit chance will depend on wether it is dark or not. This will of course depend on the type of adventure you are playing, and you have not provided guidance on that.
Example calculation
Here is the expamle calculation for expected damage for level 4:
Assassin: 26.25% [lost ini] * 65% * (4.5 [d8] + 4 [dex]) + 73.75% [won ini] * 87.75% [advantage] * (4.5 [d8] + 4 [dex] + 7 [2d6 bugbear] + 7 [2d6 sneak attack] + [from critical] 4.5 [d8] + 7 [2d6 bugbear] + 7 [2d6 sneak attack]) = 28 damage
Gloom Stalker, light environment: 19.5% [lost ini] * 75% * (4.5 [d8] + 3.5 ([hunter's mark d6] + 4 [dex] [first attack] + 4.5 [d8] + 3.5 [hunter's mark d6] + 4 [dex] +4.5 [d8] [dread ambusher attack]) + 80.5% [won ini] * 75% * (4.5 [d8] + 3.5 ([hunter's mark d6] + 4 [dex] + 7 [2d6 bugbear][first attack] + 4.5 [d8] + 3.5 ([hunter's mark d6] + 4 [dex] + 7 [2d6 bugbear] + 4.5 [d8] [dread ambusher attack]) = 30 damage
Gloom Stalker, dark environment: 19.5% [lost ini] * 93.75% [advantage] * (4.5 [d8] + 3.5 ([hunter's mark d6] + 4 [dex] [first attack] + 4.5 [d8] + 3.5 [hunter's mark d6] + 4 [dex] +4.5 [d8] [dread ambusher attack]) + 80.5% [won ini] * 93.75% [advantage]* (4.5 [d8] + 3.5 ([hunter's mark d6] + 4 [dex] + 7 [2d6 bugbear][first attack] + 4.5 [d8] + 3.5 ([hunter's mark d6] + 4 [dex] + 7 [2d6 bugbear] + 4.5 [d8] [dread ambusher attack]) = 37 damage
So, in light environments on level 4, the gloom stalker beats the Assassin by 2 points, and in dark environments, it is ahead by 9 points.
Overview over all levels
The extra attack on level 5 skews this further in the gloom stalker's favor. Here is the overall outcome by level:
| Level |
Damage Assassin |
Damage Gloom Light |
Damage Gloom Dark |
| 4 |
28 |
30 |
37 |
| 5 |
34 |
44 |
54 |
| 6 |
34 |
44 |
54 |
| 7 |
39 |
44 |
54 |
| 8 |
39 |
46 |
57 |
Note that this is not even making use of Sharpshooter on level 8. With Sharpshooter, Gloom Stalker becomes even stronger.