I don’t like random combat encounters. I prefer fights to have some context in the story. That context may be as little as, “the party’s in a bad part of town, so some street thugs are going to try to shake them down to illustrate this.”
But sometimes those street thugs are working for the big bad guy.
If I feel I need a combat encounter, to bring tension or excitement to a session, I look for more story-specific reason for the combat, rather than a wandering monster.
Let some skeletons out of the PCs’ closets.
If you’re players behave like mine, they’re borderline criminals. They’ve probably run roughshod over more than one NPC. Maybe they killed someone, maybe they beat someone up, maybe they stole from someone.
In the context of a living, breathing game world, these actions have consequences, and the people who have been wronged by the party will seek justice. They may hire assassins. They may send minions. They may hunt them down themselves. Perhaps a killed NPC will have a relative seeking vengeance.
When there’s a lull, let one of these folks catch up to (or stumble upon) the party. Think about the scene in pulp fiction when Butch is driving away from his apartment and he sees Marcellus Wallace walking across the street with a box of donuts and a cup of coffee. It’s a great “Oh Shit!” moment.
Make the big bad guy a little more proactive
You’ve got a crime lord or archmage or whatever with his grand, nefarious plan. The party is slowly uncovering the plan, figuring out who’s behind all the mysterious disappearances, the strange murder scenes, etc.
If your big bad guys is the bad ass you claim he is, perhaps he will figure out that someone is on to him and may interfere with his plans for world domination.
Maybe he’ll send a group of thugs to scare the party off his trail. Maybe he’ll send a band of undead creatures to end their meddling.
The police as “random” monsters
Don’t forget the structure of law and order in your world. If you have a group of people running around, breaking and entering, beating up people in alleys, setting fires to orphanages, etc., the law is eventually going to take note and try to do something about it.
If all these crimes are happening in one area, the city is likely to step up patrols to catch the perpetrators.
Having the second story man suddenly bathed in a spotlight and hearing the words, “sir, please step away from that window and drop the crowbar,” is another opportunity for a great “Oh Shit!” moment.
It doesn’t have to end in combat, but it certainly will complicate the story.
A case of mistaken identiy
Perhaps the party is breaking into the wrong shop, and the shop owner thinks he’s being robbed (maybe he is!). Maybe the son of one of the big bad guy’s victims thinks the party is behind the murder and is hunting them down.
In many games the party acts very suspiciously — and often on the fringes of the law. If there has been a series of break-ins in the neighborhood, and the party is skulking around in the alleys, perhaps the local citizenry will take matters into their own hands and go after them — I mean, really, how many innocent people go skulking in alleys, anyway?
Combat, in my opinion, should be a source of tension and drama, not the result of a random table. If you feel your players are itching for a fight, there’s no reason to deny them, but if you can give the fight some context in the story, it will be more meaningful and will likely raise the stakes of the game.