Yes. In Big Walk’s in-game menu, select “Session Details”. Then, select “Moderation Settings”. From this handy screen, you can manage all of the players currently in your game.
(At this point, consider changing the Password for your session.)
Also, please note: the “Players” setting for a game only adjusts how the challenges are configured. It does not limit the number of players that may join.