So, today, I was doodling in my notebook a few character concepts for fun. Recently, I've been wanting to do lots of DnD style monsters, but as likable characters, you know? Sort of a Phil Foglio type thing. So first I did a cute little goblin girl,a sorceress I guess. Then I started to experiment and do weirder stuff. The crowning glory was an Ilithid done in a Jimmy Buffet style flower shirt, but with that huge v neck collar all Ilithid clothes have. I got a good laugh at it, then thought to myself, "Gee, it would be great to play this cat in a game."
I've always enjoyed playing DnD monsters in DnD games, you see, and though I understand why the world of dungeons and dragons is filled with hundreds of nasty intelligent races and only a few with the potential for good, it doesn't mean I have to like it. I mean, all peoples have demons they have to confront, it doesn't make them particularly evil. It just makes them, if not humanoid, real. Just because trolls are brutish and ilithids are megalomaniacs and goblins are weakwilled bullies, that doesn't mean they have to be. What if there was say, a nature loving troll, eh? Maybe a laid back ilithid who just wants to research his work and surf a bit, right? Or heck, maybe a young rebel goblin who wants to go against everything her race is made out to be. These people, of course, are considered by both the race that spawned them and the lucky "good" races as not fitting into the natural order of things. In other words, freaks.
This is sort of the basis behind the game I've decided to go forward with, entitled "FREAKS!". The idea is that it's about the monsters of a standard fantasy world, but on an individual level. Each of them is blessed with special abilities that makes them unique and powerful in their own right, but also a terrible vice that makes them "monsters". The characters are adventurers of course, since thats where the people who don't function within the system of Medevial fantasy end up in. Heres my first ideas on how the system works.
-Four generic traits- Mind, Body, Coordination, Soul. These are pools of points. You get to choose one that's your star trait, which you get extra skills in.
-Next, you choose your skills. Each skill is built using trees of abilities, which are bought with points in that skill's trait.
Finally, you choose your Vice. The vice is the attitude or manner or whatever that makes your race a "monster". For goblins, it's cowardly opprotunism. For ilithids, it's insatiable power hunger. See how it works? Anytime you do something that's connected to your vice, you gain points in your vice. There are two things connected to this. First, your vice works in the same way as any of the other traits. You're able to put skills in it, and you can spend vice points to use skills specific to your race, like an orc going into a savage rage or a goblin torturing a prisoner with great precision. The flipside to this is the more you use your vice, the more the vice controls you. Each time you tap into your vice's powers, the DM can roll to see if the vice takes over, and the GM gains control over you. Should this happen, you can roll to see if you can control yourself again, but this costs a lot, and the price gets steeper the more you use your vice. There comes a point where you've allowed your base urges to consume you and you become what everyone thinks you to be, a monster.
There is another side to it. All beings have virtue, and by focusing and distilling that which is virtuous within you, you can turn away from the path of your vice. Virtue is a trait that is used to both buy off ranks in your vice, but to also gain new levels. By practicing virtue, you grow as a person. By practicing vice, you grow into a monster. I'm going to post more systemy stuff later on, but this should do for now. Please comment, tell me what you like so far.

This is sort of what was
This is sort of what was pointed out in another post, about resource-management games, as a balancing mechanism; akin to that in cyberpunk games where characters who load up on cyberware enhancements (to improve dice throws in combat resolution) may "lose their humanity." I've always been a fan of the "monsters" in rpgs having as involved a personal life as the "good guys: both abstracts such as goals and adherence to the physical and social structures of the game's background world, and game mechanics such as statistics and skills and such. Ken St. Andre was the first published game designer I know to do this, in his "Monsters! Monsters!" spin-off of his "Tunnels & Trolls" rpg (one of the first, if not the first, commercially published - and certainly the first successful - rpg inspired by "D&D"). Your idea, of forcing the characters to adhere to a moral code - similar to the "cyberpunk loss of humanity" - is excellent. The one quibble I'd have though is you're putting a humanocentric viewpoint on "what makes a monster?" For almost any human society "cowardly opportunism" or "insatiable power hunger" are practiced to fine arts but not officially virtues (well, unless you're running for president). But in a properly constituted goblin society, might not what we consider vices be considered excellent virtues? I gather though that the point of your proposed game is that the player-character monster specifically wants to get along in the human society, "stop to smell the flowers & can't we all just get along," and so is striving to practice human rather than goblin virtues, and the game is about having him avoid the latter when he will get better dice rolls to accomplish other goals (E.g. you get experience points for acquiring treasure and the best way to acquire treasure is some sort of cowardly opportunism). Problem I see with that is the player knows what he wants the character to do and will deliberately avoid fulfilling short-term goals for the character in favor of not practicing monster-virtues/human-vices.