[BUG] state.apply fails if pillar uses custom grain from _grains

[bdrx312]

Description

state.apply fails and does not sync custom grains and modules scripts from /srv/salt/_grains and and /srv/salt/_modules directories if a pillar uses the custom grain or custom execution module.

Setup

Please be as specific as possible and give set-up details.

• VM (KVM) running on AWS ec2 instance.
• onedir packaging
• masterless

Steps to Reproduce the behavior

• Setup directories and files; set minion to masterless; create a custom grain script and a pillar that uses custom grain returned from the script:

  cat > /etc/salt/minion <<'EOF'
  file_client: local
  master_type: disable
  EOF
  
  mkdir /srv/salt /srv/pillar /srv/salt/_grains
  cat > /srv/salt/top.sls <<'EOF'
  base:
    '*':
      - test
  EOF
  
  cat > /srv/salt/test.sls <<'EOF'
  "do nothing":
    test.nop: []
  EOF
  
  cat > /srv/salt/_grains/custom_grain.py <<'EOF'
  def main():
      return {'custom_grain': 'test_value'}
  EOF
  
  cat > /srv/pillar/top.sls <<'EOF'
  base:
    '*':
      - defaults
  EOF
  
  cat > /srv/pillar/defaults.sls <<'EOF'
  mypillar: "{{ grains['custom_grain'] }}"
  EOF
  

• Run state.apply which results in an error

salt-call --local state.apply

salt.exceptions.SaltRenderError: Jinja variable 'dict object' has no attribute 'custom_grain'; line 1

---
mypillar: "{{ grains['custom_grain'] }}"    <======================

---
[CRITICAL] Pillar render error: Rendering SLS 'defaults' failed. Please see master log for details.
local:
    Data failed to compile:
--------
    Pillar failed to render with the following messages:
--------
    Rendering SLS 'defaults' failed. Please see master log for details.

Expected behavior
The custom _grains and _modules should be synced before rendering the pillars and state.apply should run successfully to completion.

Screenshots
If applicable, add screenshots to help explain your problem.

Versions Report

salt --versions-report

Salt Version:
    Salt: 3006.2

Python Version:
    Python: 3.10.12 (main, Aug 3 2023, 21:47:10) [GCC 11.2.0]

Dependency Versions:
    cffi: 1.14.6
    cherrypy: 18.6.1
    dateutil: 2.8.2
    Jinja2: 3.1.2
    msgpack: 1.0.2
    packaging: 22.0
    pycparser: 2.21
    pycryptodome: 3.9.8
    python-gnupg: 0.4.8
    PyYAML: 6.0.1
    PyZMQ: 23.2.0
    relenv: 0.13.3
    timelib: 0.2.4
    Tornado: 4.5.3
    ZMQ: 4.3.4

System Versions:
    dist: rhel 8.8 Ootpa
    locale: utf-8
    machine: x86_64
    release: 4.18.0-477.13.1.el8_8.x86_64
    system: Linux
    version: Red Hat Enterprise Linux 8.8 Ootpa

Additional context
Running salt-call --local saltutil.sync_all --pillar-root=/dev/null before running state.apply syncs the _grains and _modules correctly and allows the state.apply to run correctly.

[dmurphy18]

@bdrx312 Please retry the issue with the latest 3006.2, a number of fixes have been made since 3006.0 was released.

[bdrx312]

@bdrx312 Please retry the issue with the latest 3006.2, a number of fixes have been made since 3006.0 was released.

I did not realize that 3006.2 was out already. I updated and tried again but got the same error. I will edit the post to put the update version info.

[dmurphy18]

@bdrx312 Tried this with Salt 3005.1 classic packaging and it failed, wondering if there is something missing in the instructions

local:
    Data failed to compile:
----------
    Pillar failed to render with the following messages:
----------
    Specified SLS 'defaults' in environment 'base' is not available on the salt master

Trying to ensure it used to work and now broken

[dmurphy18]

Also from a salt-master

root@Unknown:/srv/pillar# salt td11 pillar.items
td11:
    ----------
    _errors:
        - Specified SLS 'defaults' in environment 'base' is not available on the salt master
root@Unknown:/srv/pillar#```

The salt-master

root@Unknown:/srv/pillar# l
total 8.0K
drwxr-xr-x. 4 root root 30 Aug 3 2020 ..
-rw-r--r--. 1 root root 28 Aug 25 16:23 top.sls
-rw-r--r--. 1 root root 41 Aug 25 16:23 default.sls
drwxr-xr-x. 2 root root 88 Aug 25 16:25 arch
drwxr-xr-x. 3 root root 49 Aug 25 16:25 .
root@Unknown:/srv/pillar#
root@Unknown:/srv/pillar# cat top.sls
base:
'*':
- defaults
root@Unknown:/srv/pillar# cat default.sls
mypillar: "{{ grains['custom_grain'] }}"
root@Unknown:/srv/pillar#

[bdrx312]

@bdrx312 Tried this with Salt 3005.1 classic packaging and it failed, wondering if there is something missing in the instructions

I added the salt minion configuration /etc/salt/minion to set it to masterless mode. I think all that is needed is the master_type: disable, and I went ahead and added file_client: local. I will have to check at work on Monday to see if any other settings are needed, but I believe that should be all to get it working.

[dmurphy18]

@bdrx312 Even in masterless with file_client local, I am still unale to reproduce this with Salt 3006.1, and do not see a SaltRenderError in the logs:

[DEBUG   ] Gathering pillar data for state run
[DEBUG   ] Finished gathering pillar data for state run
[INFO    ] Loading fresh modules for state activity
[DEBUG   ] The functions from module 'jinja' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded jinja.render
[DEBUG   ] The functions from module 'yaml' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded yaml.render
[DEBUG   ] The functions from module 'highstate' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded highstate.output
local:
    Data failed to compile:
----------
    Pillar failed to render with the following messages:
----------
    Specified SLS 'defaults' in environment 'base' is not available on the salt master
root@tdeb11:/srv/pillar# 

Can you reexamine the configurations on the minion in order to reproduce the issue you are experiencing ?, but so far I have been unable to reproduce the rendering error.

[bdrx312]

@bdrx312 Even in masterless with file_client local, I am still unale to reproduce this with Salt 3006.1, and do not see a SaltRenderError in the logs:

[DEBUG   ] Gathering pillar data for state run
[DEBUG   ] Finished gathering pillar data for state run
[INFO    ] Loading fresh modules for state activity
[DEBUG   ] The functions from module 'jinja' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded jinja.render
[DEBUG   ] The functions from module 'yaml' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded yaml.render
[DEBUG   ] The functions from module 'highstate' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded highstate.output
local:
    Data failed to compile:
----------
    Pillar failed to render with the following messages:
----------
    Specified SLS 'defaults' in environment 'base' is not available on the salt master
root@tdeb11:/srv/pillar# 

Can you reexamine the configurations on the minion in order to reproduce the issue you are experiencing ?, but so far I have been unable to reproduce the rendering error.

I had a typo/mismatch in the file name of the pillar defaults file. In the pillar top file I specified defaults, but in the file creation it was just /srv/pillar/default.yml (no s). I have corrected the original post to add the s making it /srv/pillar/defaults.yml.

[dmurphy18]

So with the default.sls -> defaults.sls, and re-run salt-call --local state.apply
It appears to work for me with Salt 3006.1

[DEBUG   ] File /var/cache/salt/minion/accumulator/139644020863808 does not exist, no need to cleanup
[DEBUG   ] The functions from module 'state' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded state.check_result
[DEBUG   ] The functions from module 'highstate' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded highstate.output
local:
----------
          ID: do nothing
    Function: test.nop
      Result: True
     Comment: Success!
     Started: 12:52:45.047935
    Duration: 1.222 ms
     Changes:   

Summary for local
------------
Succeeded: 1
Failed:    0
------------
Total states run:     1
Total run time:   1.222 ms
root@tdeb11:/srv/pillar# l
total 16K
drwxr-xr-x 2 root root 4.0K Sep  5 12:52 .
drwxr-xr-x 4 root root 4.0K Aug 25 16:14 ..
-rw-r--r-- 1 root root   41 Aug 25 16:14 defaults.sls
-rw-r--r-- 1 root root   28 Aug 25 16:14 top.sls
root@tdeb11:/srv/pillar#

Update to Salt 3006.2 and same result, it works for me with Salt 3006.2 too
From salt-call --local grains.items

    cpuarch:
        x86_64
    custom_grain:
        test_value