It can be painful and confusing to watch the guy you like become transformed when he gets in front of his friends. There are several possible explanations for his behavior.

One is that this guy has a hard time showing his friends this side of himself. Some people have trouble mixing friendships with other relationships, whether they are with a girlfriend, parents, a sibling or someone else. This is especially common for people who are younger or are forming new friendships with a group of people.

On the other hand, maybe he's just trying to get comfortable with the idea of dating in general. It may be that he is nervous and totally unsure of how to deal with his feelings and with having a girlfriend. He might be afraid that his friends will make fun of him.

I think you have some choices. One option is to wait it out and give him time to get more comfortable with himself and with the situation. Another is to try things that might put him at ease when you are with his friends...such as acting as casual as possible about the relationship without being distant. If you can show that you are comfortable and it is no big deal, it might help him calm down a bit.

Or you could tackle the issue directly and ask him what is going on. It may be awkward but it could give him a chance to explain his behavior and to understand how it makes you feel. If you are feeling bad all the time, then I think you should definitely let him know about it and give him a chance to change before you call it quits.