Allow relative paths for mounting volumes

[scomtott]

It would be useful to be able to mount a volume from the current directory rather than specifying an absolute path with:

`docker_switches:

• -v
• /absolute/path/to/kube/config:/root/.kube`

Floki is intended to create reproducible containers with the minimum amount of pre-setup. By only allowing absolute paths in volume mounting, a difference in behaviour is created if another user has a different file structure.

[rlupton20]

Hi scomtott@, thanks for raising this. This kind of use case has come up before, and it's a bit tricky, so it would be good to think it through. I don't think relative paths solve the issue either (where are these files relative to the floki.yaml in general?).

One thing to say about docker_switches is it's really intended as a break-glass mechanism for when you just need to get something done. It makes no guarantees about reproducibility, so ideally we wouldn't use it too heavily (it has its reproducible uses though, e.g. adding debug capabilities to containers for e.g. gdb).

The essential issue here is we want to inject some external configuration which we don't want to master with the source. That means we can't make any guarantees about its reproducibility. I'm not sure I have a solution for this yet which is satisfactorily ergonomic.

One way you might solve this now is using the volumes feature to add a volume which persists between runs. You could even add a step in init which pokes the volume to see if the required files are there, and helps inject them if needed.

Typically these volumes are used as build caches, but there's no reason they need to be. You could even link the backing directory to /kube/config if you wanted. I imagine there is room to improve the ergonomics of this process if it turns out to be a reasonable way forward - we can also discuss ideas here.

I'm no longer a switchy, I moved on about a year ago, so I can't see inside your slack archives I'm afraid. @maxdymond can, but just be wary of cross posting/linking internal things.

[maxdymond]

The way I envisioned this working was having first-class citizenship for some kinds of files - kubeconfig would be one of these (lives in a well-known location). Having said that, it moves the line between reproducible environment and "ease of use".

Maybe my pitch is "You can enable some files to be mapped, but a friendly warning is printed out that your environment might not be reproducible because of it".

[scomtott]

@rlupton20 Thank you for your in depth response and apologies about the internal links - I (wrongly) assumed that the only people working on this would be Metaswitch people. (For reference the Slack link was somebody essentially asking for the same thing that I am asking for).

What I'm trying to achieve is to have a specific kubeconfig file injected in to the right place in the container - this may also not be the same as having the person's well-known-locaiton kubeconfig injected in as per @maxdymond's suggestion. Having relative paths should solve this because I can make sure that the file exists at the right place relative to where the floki.yaml file is in the git repo.

An init step mounting a volume and making sure that the right file is in that volume would work fine so long as the same volume can still be mounted in different use cases (virtual machine or not for example). I'm not quite sure about how to do this though.

[rlupton20]

@scomtott no worries about the links :-).

Is the kubeconfig file part of the git repo? If so, could you symlink it into place in init? Otherwise I'm not sure how you make such guarantees.

@maxdymond it's a thought. I'm allergic to special features for special situations (there are similar problems with AWS, Azure, OpenStack etc) but your pitch is open enough that there may be some room to play with here.

I think it's ok to sacrifice some reproducibility (I guess we don't want auth to be reproducible...), provided the ergonomics stay reasonable.

To draw some distinctions we really want

• reproducibility of builds (but auxillary stuff we can flex a bit)
• not-too-much-pain to get going (else there becomes no point in using floki, really)

A first, although I'm not sold on this, way to do this might be to have

capture:
  name_of_capture:
    description: "Special file for special things"
    path: /some/path/in/container

and have floki keep a folder of symlinks/directories with symlinks which it generates on first run

This environment expects a file at PATH (description: special file for special things)
Enter a host path to capture: /home/all_my_credentials

I don't like it, but its a start.

Another possibility I thought of was to create a floki-auth service which provides a unix socket to query such things (in much the same way as we do SSH_AUTH_SOCK), but that's much more work.

The cheap and cheerful approach might be just to .gitignore a file in your repo, and check in something blank there, then have init reject it if its empty (and link it to the right place if its not).

A few thoughts. I'm not sold on any in particular atm.

[scomtott]

Yes the kubeconfig will be part of the git repo - I can try that.

[bossmc]

An alternative approach (that would work with .kubeconfig but maybe not other, similar, files) is to set KUBECONFIG=/src/.kubeconfig in the env section of the floki.yaml file. All the k8s tools should look at that environment variable in preference to checking the default locations for the file.

[scomtott]

Hopefully this is helpful feedback (and it is certainly intended as such) but I gave up and wrote my own Dockerfile.

[rlupton20]

That's another way to do it - I'd rather decouple the hacks to get something working v.s. figuring out if there are enhancements we should make here, so I'm glad you have a solution.

I think @bossmc has the best workaround for now.

@scomtott are you happy for me to repurpose the issue to track thoughts and ideas to make this not terrible more ergonomic?

[scomtott]

@rlupton20 No problem.

[podusowski]

Hi, my team has very similar case, although we're not storing kube config in git but want to use the one user has in his home. I was thinking that if you're OK to sacrifice reproducibility a bit, then maybe Floki could parse and expand environment variables from the yaml file? Both docker compose and VSCode's dev containers does that. This could even be limited to the docker_switches or some new unsafe section of the config.

[rlupton20]

There does seem to be a general problem around this that needs a solution, so yeah, perhaps it's worth thinking it through a little more deeply.

If we want to capture environment variables, we can do that, and even have floki enforce that they are set. I'm using reproducibility a little loosely, what I really mean is that people ought to be able to use a build environment with minimal faff. Maybe if floki errors and says it needs a KUBECONFIG set in the environment that's still a relatively reasonable workflow. Was this the kind of thing you had in mind?

Another possibility with the above is to have a global volume and link the kubeconfig into it. You can script the init section to test for the existence of the file.

I've also been toying with the idea of running a floki auth container that you can link as a sidecar (ala dind), and then some kind of wizard style interface to link credentials/config into the container, but this seems overly complex.

[podusowski]

I was rather thinking that Floki could just expand the variables in the yaml. I could then do:

docker_switches:
- "-v${HOME}/.kube:/root/.kube"
- "-v${PWD}/.kube:/root/.kube"

(sorry if I got the syntax wrong)

This could solve both use cases and possibly more.