That's because, even though you guys were "casual," you still had some sort of emotional thing happening.
And if they are someone you consider to be a friend, just be aware that this can be a bit harder to navigate. Make sure you both understand that this is only meant to be physical and casual to avoid hurt feelings later on.
It's hard not to get excited when you start developing feelings for a new flame.
However, as you start talking about him to others, you find yourself stopping in mid-sentence to wonder what you should call him.
Should you be referring to him as the boyfriend or is he nothing more than the occasional evening romp? However, there may be a few things he's already doing that will tell you loud and clear if he's serious or just a temporary situation. He doesn't rush off after sticking around for an hour nor does he relegate you to the 2 a.m. Instead, a boyfriend not only makes the time, but makes you a highlight of his plans so that you spend meaningful time together.
A guy that's simply looking to hook up isn't usually interested in getting involved with other parts of your life, which includes meeting other people you know.