Pass environment variables to the python interpreter

[lud-wj]

I would like to know how I can pass environment variables dynamically. It seems that the interpreter inherits environment variables from where the application is booted. But is there a way to add some (from loaded .env file in runtime.exs) or remove/overwrite some, to avoid leaking sensitive data or swap an API key for another ?

Thank you :)

(oh and not worth its own issue: how do you pass a pid to send_tagged_objects ? The encoder does not support passing pids in globals)

[jonatanklosko]

Yes, when the BEAM starts Erlang uses the process env vars to initialise it's internal env vars store, but further modifications such as System.put_env only affect the Erlang data structure. I think the same applies to Python when modifying sys.environ dict. So System.put_env will not be visible to Python, even if it happens before initialization.

I think it would be reasonable for us to pass System.get_env() when initializing Python and set sys.environ to match it exactly. I will have a look at that.

(oh and not worth its own issue: how do you pass a pid to send_tagged_objects ? The encoder does not support passing pids in globals)

You should be able to pass pid via globals, the encoder was added together with send_tagged_objects, however note that those are only available if you install Pythonx from main, not on latest. We have been waiting on feedback regarding send_tagged_objects before making a release.

[jonatanklosko]

We decided to not mirror the env vars. Setting os.environ in Python automatically sets the OS process environment variables, so the side-effects can go beyond Python. See #43 (comment). On a sidenote, there is a number of gotchas to make it work on Windows.

Instead we documented the current behaviour (#44).

If you want to mirror specific env vars, you can do Pythonx.eval and set os.environ keys accordingly. A good place to do that may be you def start in your Application, so it happens after Python is initialized, and before you start any processes relying on it.

[lud-wj]

So if I understand correctly, the python environment is set only once when the OTP app boots? There is no way to set env like in System.cmd("foo", [], env: %{"FOO" => "123"}) or something like that for a single interpreter invocation?

[jonatanklosko]

So if I understand correctly, the python environment is set only once when the OTP app boots?

Pretty much. Technically it is set when you initialize Python interpreter, which is either an explicit call to Pythonx.uv_init, or on Pythonx application boot if configured via config :pythonx, :uv_init, .... Regardless, the initial Python env vars reflect the env vars given to the Elixir OS process, any further System.put_env calls are not accounted for.

There is no way to set env like in System.cmd("foo", [], env: %{"FOO" => "123"}) or something like that for a single interpreter invocation?

The interpreter is initialized once and from that point forward the env is global across all Pythonx.eval calls, since it's the same OS process. The only way to modify those envs is to directly update Python's os.environ within Pythonx.eval (and it is persisted into the future evals).

[lud-wj]

Ok thank you, I understand.

Is it a goal for you to support multiple instances of the interpreter in the future? Like defining our own instances with different pyprojects in the app supervision tree and call them with something like Pythonx.eval(:some_name, "x = 1") ?

For now I should be able to make an initial call to set the environment variables from a Task in the app boot sequence, or something like that.

[jonatanklosko]

Is it a goal for you to support multiple instances of the interpreter in the future?

Not really, see #16 (comment).

For now I should be able to make an initial call to set the environment variables from a Task in the app boot sequence, or something like that.

You can do Pythonx.eval synchronously in your application def start that sets os.environ.

[lud-wj]

Yeah ok that was my idea.

Thank you :)