The sort of enemies who can casually toss off a plane shift can probably come up with ways to come back meaner than ever when given a minute of timeout, without making the players feel so much like they got cheated out of their cool move as if they just come back the next turn. They should be buffing themself up or rounding-up up to eight willing minions.
Others have, of course, noted that Plane Shift doesn't work as you used it. You would generally need to use Gate for a creature to come straight back to the fight, though it is not a stretch to have a teleportation circle near a fight you are anticipating will likely involve an extraplanar being getting banished.
ncG1vNJzZmivp6x7prrWqKmlnF6kv6h706GpnpmUqHyjrc2iqqGllaPBbq3NnWSippOWvaKvyK2YrZ2UY4N6hJZybWg%3D