Can I get sued for slander or defamation of character?

I dont think so, If you would have said something like " you didnt pay me and you were late paying me and you are cheeting on your wife and beat kittens in your back yard" then you could get sued. I believe for it to hold up in court what you said has to be a lie, so if he really mistreated you as an employee then you have every right to say what you said. Check out that link, its a better definition.