There are growing signs of workers walking off the job