I agree let the g/f and brother handle a lot of it, let them tell him its cool when rolls get botched, its part of the fun. 
I think that part should be empathized is that when things go south the experience is enhanced for everyone around the table and that he is cheating them of the true experience.

As far as what to do in game to bring balance?  That is a hard call my friend. 
You could just let it go, everyone is their to have fun first and foremost.  This would be my choice.
That being said a little karma being handed out via story isn't to bad as long as it doesn't take away from everyone having fun.

Here is how I would handle what your looking for.
Have your characters roll there attack
if they get a number in within his threat range and activates it with his action dice
then the character hits the creature with the giant eye in the eye.

Thanks, that would have been my interpretation too (sadly not that of my GM )

Well this is when you show him the first page with the writing credits than show him Alex Flagg's post right above yours.

That is not to say he isn't welcome to run his game his way.  However it is how it was intended for the game to run.

Oh man that just brought back Memories of a 2nd edition AD&D game where a random encountered Hobgoblin came to be a well respected and revered member of a royal court that the PCs where heavily involved in. 

Dwarf and the Saurian become obsessed with the idea that a random (literally random) lady of the night is a member of the king's spydom.  Gets so hammered trying to 'unlock' her secrets, both are rendered unable to work.

So my question to you is this than, do you go ahead and make her part of the spydom to keep things rolling along or just let them act the fool, or find some other clever way of using her as in maybe a spy that isn't with the king but someone else.

This brings back some strange memories of the Rifts Vampires book

Pommel Bash is way too sterile a name for that .

So Dirty Boxing then. 

I think the best solution is to simply let him pick another species feat.  Maybe let it be a 1st level only one. 

If you have enough coin, people will work for you, decent reputation or not.

I would say this is true only with a reasonable expectation of returning alive or as you said "enough coin".   

Going off to Mt. Doom that is across the desert of death, throw the swamp of sorrow to find the lich kings tomb, with a group of unknown adventures is crazy. 

The required amount to be "enough coin"  that should be paid up front most likely would exceed what an adventuring parties capability to pay if they have yet earned enough rep & renown. 

The "monster" system is by far one of the best features for a GM to use.  Its easy, making adjustments on the fly is possible, it really allows you to build an NPC the way you want them to be.

so at this stage I am more figuring out what I want to represent first, and then working on the how.

One of the things that is a real pain with this, is it is a fantasy game, a human grappling a human easy to know what it looks like and how it plays out more or less. 
You start talking about grappling Unborn, Dragons, Giant Purple Worms, Rootwalkers then it gets a bit more complicated...

As it is, at best, you force your GM into some stupid skill check pissing contest because he doesn't want his setpiece villain to be down for the count in the first round. 

See here is the problem this shouldn't be happening this is what Mooks are for, to start stabbing you when you have the big bad held down.  This becomes very dangerous real quick for a grappling PC as they are subject to sneak attack damage and other issues with losing ones dex bonus.

"I am a shark, the ground is my ocean and most people can't even swim."
-Rickson Gracie

I like this quote and I think its very apt for any sort of "mini game" inside of a game.  A charter should feel that their is a good reason to pull someone else into their element outside of their comfort zone.  This is the way it  currently works.

Maybe the problem is their is next to no way to prevent being pulled into the minigame.  As opposed to the minigame itself.

I think one of the problems is 
codified grappling such as
Judo, Brazilian Jujitsu, Catch as you Catch can, Glima, ect...
gives you situations such as
the little 130lb Purple belt I roll with that is not particularly strong or large but he knows his grappling and I have seen him tie up skillful very Athletic 250lb men.
Is having ranks in athletics the same as developing highly technical grappling maneuvers that allow someone who is significantly smaller dominate over someone much larger and stronger?
I will say I really and mean really don't like it being BAB based at all because one of the best things about the current system is it is divorced hitting people is so different then using leverage.
I have seen world class Kick Boxers against relatively new grappler just be destroyed because it is so far outside of their element.
Now I know this isn't  simulation and its cinematic but to walk so far away from reality would be a disservice.
So is grappling going to be
http://www.youtube.com/watch?v=FCU503Locjs
or
http://www.youtube.com/watch?v=9erKI3pxUWQ&feature=related
 

Without interfering too much in the discussions of people with far more play experience than I have, since I read the grapple rules for the first time I have wondered why it's based on Athletics rather than unarmed attack bonus?

To me this would make more sense? I understand that opposed attack rolls are not really used whereas opposed skill rolls are, but surely somebody trained to fight and with an unarmed forte should be better at grappling than an experienced swimmer/climber?

Well I will tell you I personally love the fact that its a separate skill and not based on the unarmed attack bonus (or any attack bonus).  It means that the courtier can be an excellent wrestler. 
Grappling and Striking are 2 very different skills. 
Take a look at your typical High School Wrestler they can grapple the heck out of a person but it doesn't mean they can box you.
Also I think Athletics is a stable of Soldiers.
Interesting factoid for you the regions around the global that developed and made heavy use of metal armor have well codified grappling systems.  It is a battlefield art.