Passing doppler token as build arg or env safety concern

I built high availability integration path with docker(on digital ocean droplets). I followed the doc and everything seems to be working fine its just I have some security concerns. How safe is it to use doppler token as build arg, and then send it to container as an env? I have read few posts around this being unsafe. Was wondering if those concerns are valid or if I am missing something