You did right by leaving them alone. They look for building materials and then move on unless they’re swarming (thousands of bees).