I guess it is exactly about these two scenarios in toxicity, production and final product.

Production:

A major part of the process adaption cost for POET in a foundry will deal with the waste of the production. There is waste from eaching, doping, cutting and polishing of wafers/chips. With silicon the waste is - after neutralisation, quite harmless: Neutral sats and silicates. With GaAs these waste streams are highly toxic - they will contain Gallates and Arsenates.
But all these processes are running in closed, very well controlled production processes: I am sure this valuable waste will be collected and recycled - and will end up in production of further GaAs. Ecology meets economy.

Final product:

In deed users and consumers will hardly get in contact with the GaAs inside the capsuled/packed chips. So the concern will deal with the end of the life cycle of a GaAs-chip. After the final use the hardware has to be collected and treated under a certain control, so it will not end up in mother nature. Already today this is common practice, so let us hope that this will not change with low cost IoT products.

So it is clear,like in many industries (especially the chemical inductry) there are concerns and problems, but they will be solved.