[BUG] Salt-minion pillars don't always refresh when they should #57180
Description
Description
Salt-minion pillars don't always refresh when they should.
This is trickery to reproduce and seems to involve a race condition. The steps to reproduce have to be executed very fast.
I am pretty sure from testing that this is caused by changes in this PR, #54942 . I tried reverted the patches and the issue went away.
Steps to Reproduce the behavior
Locally:
cd salt-pillars
mkdir test
echo "base:
'*':
- test" > top.sls
echo 'test' > test/init.sls ; git add test/init.sls ; git commit -m 'Testing' ; git push
Master:
/usr/bin/salt-run git_pillar.update ; /usr/bin/salt-run fileserver.update ; salt `hostname -f` state.apply base
test [email protected]:org/salt-grains:
None
test [email protected]:org/salt-pillars:
True
True
fqdn:
Data failed to compile:
----------
Pillar failed to render with the following messages:
----------
SLS 'test' does not render to a dictionary
Locally:
echo 'test: a' > test/init.sls ; git add test/init.sls ; git commit -m 'Testing' ; git push
Master:
/usr/bin/salt-run git_pillar.update ; /usr/bin/salt-run fileserver.update ; salt `hostname -f` state.apply base
test [email protected]:org/salt-grains:
None
test [email protected]:org/salt-pillars:
True
True
fqdn:
Data failed to compile:
----------
Pillar failed to render with the following messages:
----------
SLS 'test' does not render to a dictionary
salt `hostname -f` saltutil.refresh_pillar
fqdn:
True
salt]$ salt `hostname -f` state.apply base
fqdn:
----------
ID: hosts_file
Function: file.managed
Name: /etc/hosts
Result: True
Comment: File /etc/hosts is in the correct state
Started: 17:16:12.353157
Duration: 52.312 ms
Changes:
Summary for fqdn
-------------
Succeeded: 1
Failed: 0
-------------
Total states run: 1
Total run time: 1.230 s
Expected behavior
The error on the first state.apply is expected. There is a pillar syntax error. The error on the second state.apply is not expected, because the syntax has been fixed. The git_pillar.update has pulled in new pillars as we can see by the True for salt-pillars. When we then refresh_pillar for that minion we get the correct result we should have gotten on the second state.apply.
Versions Report
Salt Version:
Salt: 3000.2
Dependency Versions:
cffi: Not Installed
cherrypy: unknown
dateutil: 2.6.1
docker-py: 2.5.1
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 2.10
libgit2: 0.26.0
M2Crypto: Not Installed
Mako: Not Installed
msgpack-pure: Not Installed
msgpack-python: 0.5.6
mysql-python: 1.4.4
pycparser: Not Installed
pycrypto: 2.6.1
pycryptodome: 3.4.7
pygit2: 0.26.2
Python: 3.6.9 (default, Apr 18 2020, 01:56:04)
python-gnupg: 0.4.1
PyYAML: 3.12
PyZMQ: 16.0.2
smmap: Not Installed
timelib: Not Installed
Tornado: 4.5.3
ZMQ: 4.2.5
System Versions:
dist: Ubuntu 18.04 bionic
locale: UTF-8
machine: x86_64
release: 5.3.0-1017-aws
system: Linux
version: Ubuntu 18.04 bionic