First of all, some more informations would be nice:
what type of server do you use? (bukkit/spigot, forge...)
if you get a crash log, please post that too.
and what types of mods do you want to install?
i would guess that you try to install a mod that would not run with the installed version of the server or maybe it interruppts with an already installed mod.
sometimes you have to use an older version of the mod, to make it run on the current server version.