Question: How to manage merge conflicts with MAC signature

[chroto]

Hi. Great stuff here. Putting a PoC together for the team using git as a storage and distribution mechanism. We are currently using https://www.passwordstore.org/ and I'm attracted to sops for its support for AWS KMS.

What are your recommendations for settling merges since they (almost?) always result in a conflict on the mac signature with neither signature going to be the correct one.

I'm comfortable with sops --ignore-mac and :wq to re-generate a mac, but this could be a roadblock for adoption since it's additional training to micro-manage the MAC every time there is a merge.

Any recommendations for reducing the scope of this problem? How do you folks handle this issue?

[jvehent]

That's a very valid point, thanks for bringing it up.
Conflict resolution has not been an issue yet, because we manage secrets in small files and it's very unlikely two persons will be modifying the same file at the same time. Generally speaking, I think a lot of small files is better than a single big one.

I have not looked into what git provides, but maybe we could invoke a sops command to merge two versions of a file when a conflict occur? That way, sops would verify both MACs, then merge the files and generate a new MAC. It might end up tricky to implement, but worth investigating.

[twolfson]

I took a shot at this tonight. I couldn't find anything straightforward with .gitattributes but following @jvehent's suggestion, I started on a custom merge configuration for SOPS. Here's a proof of concept demonstrating it handling decryption (which includes MAC validation) of source files and opening the default mergetool:

Code available here:

https://gist.github.com/twolfson/962b1eb776ce9947a09d4924d91fd8b2/b18db5fbcea4f3dd4632c9df8d94f14b644c1e52

There is a snag, we need a SOPS command to replace an entire files contents while maintaining the keys. After that is built, the script should be more/less good to go (maybe port to Python for portability).

Additionally, there is nothing to handle a key rotation. Maybe we should have other commands which dump metadata so we can diff/mergetool those as well?

I should also mention that another short term alternative would be to leverage well tested git hooks for merge/rebase like those in victorious-git:

https://github.com/twolfson/victorious-git

Edit: Locked down revision for gist

[jvehent]

Very nice analysis @twolfson ! 👍
I only skimmed through your scripts, but my first impression is that we need a bunch of local git configuration, which hurts user experience a bit. Do you think these could be wrapped under a sops --merge <file> command which handles all the magical git stuff?

[twolfson]

Upon reviewing my code, the .git/config + .gitattributes are more designed for automatic dependency resolution when running git merge (there shouldn't be a mergetool window open, it's all supposed to be automated).

Instead there's git mergetool which is used by users when a conflict is encountered. We should be able to use that without any .git/config or .gitattributes:

git mergetool --tool=./sops-mergetool.sh example.yaml

Unfortunately, I'm not in a good spot to try that out at the moment but will later.

[twolfson]

Outside of that I have a couple more notes:

• For replacing an entire file's contents, we should be able to accept input from STDIN but I don't think it works (tried last night)
  • cat foo.yaml | sops example.yaml
  • sops example.yaml < foo.yaml
• For handling changing meta data like keys, maybe we should pass through meta data to the mergetool as well (maybe via a command like sops --decrypt-with-meta). For example:

# Inside the 1 column of the mergetool
foo: bar
sops:
    mac: ABCDEF
    version: 1.7
    pgp:
    -   fp: 1022470DE3F0BC54BC6AB62DE05550BC07FB1A0A
# ...

Edit: Removed data: key from YAML example

[jvehent]

For replacing an entire file's contents, we should be able to accept input from STDIN but I don't think it works (tried last night)

Two solutions:

• sops -e foo.yaml > example.yaml
• cp foo.yaml example.yaml && sops -e -i example.yaml

That omits --pgp and --kms, but those can be driven through a .sops.yaml file.

For handling changing meta data like keys, maybe we should pass through meta data to the mergetool as well (maybe via a command like sops --decrypt-with-meta).

I think you're looking for sops --show_master_keys example.yaml which is already implemented.

[twolfson]

With respect to STDIN, is foo.yaml raw data (e.g. foo: bar)? If so, how does sops -e know which keys to use when outputting content?

With respect to sops --show_master_keys, is there a way to re-encrypt from that output format? That is convert

foo: bar
sops:
    mac: ABCDEF
    version: 1.7
    pgp:
    -   fp: 1022470DE3F0BC54BC6AB62DE05550BC07FB1A0A

to

foo: ENC[AES256_GCM,data:9l9h,iv:c/QuxSHYuqV4DWLbMMg1MKEKVbsP14sKFYChHSjxJeU=,tag:EDyqP6ckK0aRhE/XcNt6WQ==,type:str]
sops:
    mac: ENC[AES256_GCM,data:eiKgBSVnGKMIMvLKYees7Q74MXCw+XYSjjOXIWuyJfRuHTNU8dFsKjBhgZBL6RHqu4M5LkzS0MAfg0l9GMV1obnnzD0noht8N/Ge3PQ8QXf2aYYsciDsmBkGksQHC2XWHLF7eheLuicHlyZ2S73i+VAs/gOBywwXR+59x58Vntc=,iv:UlCCFAdOs1HuGrya8gCeoWnW8xj4Q2IarmRxbkwPV5I=,tag:z7l7aYnRqWdfL9wnPXDr8w==,type:str]
    version: 1.7
    pgp:
    -   fp: 1022470DE3F0BC54BC6AB62DE05550BC07FB1A0A
# ...

[twolfson]

Ah, I have answered my own question in both respects. -e accepts data that already has a sops key and will use that as its keys

https://github.com/mozilla/sops/blob/f63597f901f50f07ff72452b4bdb485518b85de7/sops/__init__.py#L511-L516

So the answers are:

• No, foo.yaml isn't raw data. It is data + a SOPS branch
• Yes, we can re-encrypt sops --show_master_keys via sops -e

[twolfson]

Alright, I explored git-mergetool. Unfortunately, we would need to have users set up .git/config with a [mergetool "sops-mergetool"] to get it working. That being said, the gist is now focused on git-mergetool and feels like a good experience (after the .git/config):

https://gist.github.com/twolfson/962b1eb776ce9947a09d4924d91fd8b2/1978689a51b7b2f388121fe71a530eae145863fa

[twolfson]

After thinking for a bit, there's no reason why that script has to be used via git mergetool (and consequently configured via .git/config). As I learned during iterations, we can generate all the necessary files from the conflicted state:

https://git-scm.com/book/en/v2/Git-Tools-Advanced-Merging#_manual_remerge

git show :1:example.yaml # Contents of `$BASE`
git show :2:example.yaml # Contents of `$LOCAL`
git show :3:example.yaml # Contents of `$REMOTE`

With respect to keeping the tool agnostic (e.g. agnostic to git/hg/svg), I don't think it's plausible for git as we need to find its mergetool configuration.

[cjbottaro]

I'm using SOPS via Helm Secrets. Is there a way to turn on --ignore-mac via a sops.yaml file or some global config?

Also, I can't believe this isn't a bigger issue. We ran into this almost immediately, and we're not even that big of an engineering team.

[autrilla]

I'm using SOPS via Helm Secrets. Is there a way to turn on --ignore-mac via a sops.yaml file or some global config?

I think this is a bad idea. You should use --ignore-mac reaaaaaally sparingly. It's the only thing that ensures your file hasn't been tampered with by e.g. reordering entries.

Also, I can't believe this isn't a bigger issue. We ran into this almost immediately, and we're not even that big of an engineering team.

This is probably because of your workflow. I've never had a single merge conflict with our SOPS-encrypted files at Mozilla in over two years, and we're a 10-engineer team. We try to isolate secrets into one file per 'application' and environment.

[reidab]

This is probably because of your workflow. I've never had a single merge conflict with our SOPS-encrypted files at Mozilla in over two years, and we're a 10-engineer team. We try to isolate secrets into one file per 'application' and environment.

I agree that global disabling MAC checking is a bad idea, but I'd be curious to know more about your workflow. Like @cjbottaro, I'm using sops + helm-secrets.

We currently store secrets files in the same repos as our application code, one file per environment. The place where we most often hit MAC conflicts is when two feature branches each require adding an additional key to the secrets file, and are both merged down to master. Git cleanly merges the parallel additions, so the only conflict left to resolve at merge time is the MAC.

My current workaround, like the OP mentions, is re-saving each conflicted file sops --ignore-mac, but this doesn't seem ideal.