A great swap I have found is switching out plastic bottles for glass.
I have found glass spray bottles and also glass pump bottles that I use now instead of the plastic versions. I keep a glass soap pump at each sink in the house. I just refill them when they get low, so there is not plastic to throw away.
The pumps I use I found on Grove. You can get a silicone sleeve to go around the base of the glass to help with sliding or breakage if you need. They also have the glass water spray bottles I use. They also come with silicone sleeves for the base.
Unfortunately, there is still plastic in the spray nozzle and pump, but at least it is a better option than all plastic.
Try them out and see what you think.