silly me is an informal way to say that your inattention caused something unintended (usually minor, the "silly" part) to happen. It comments on the person making the statement.
So if you heard your friend say:
Silly me, I shouldn't be so absent minded.
any response depends on your relationship with that person, from a statement that is rude to one that is supporting.
But a response is definitely not required. If you are not on good terms with that person, it might be understood as hostile..
Regarding your boss, you might be respectful and say:
Don't worry about it, we all can be forgetful at times.
though it might be better not to say anything if it might be used against you later.