Monitor keyring not being populated

[ghost]

Both /etc/ceph/ceph.client.admin.keyring and /var/lib/ceph/mon/ceph-ip-172-16-13-217/keyring are empty after running state.apply.

Ceph Version 12.2.1
Salt Version 2017.7.2
Ubuntu Version 16.04.3

From looking at the code it seems that I should be populating salt mine data somehow but there's no reference to how it should be collected in the documentation.

/var/log/ceph/ceph-mon.ip-172-16-13-217.log:

2017-11-03 02:10:19.683021 7fdc04ca2700  0 -- 172.16.13.217:6789/0 >> 172.16.12.99:6789/0 conn(0x55d10f90a000 :-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0 l=0).handle_connect_reply connect got BADAUTHORIZER
2017-11-03 02:10:19.971350 7fdc044a1700  0 mon.ip-172-16-13-217@1(probing) e0 ms_verify_authorizer cephx enabled, but no authorizer (required for mon)
2017-11-03 02:10:19.971357 7fdc044a1700  0 -- 172.16.13.217:6789/0 >> 172.16.14.152:6789/0 conn(0x55d10f9cb000 :6789 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg: got bad authorizer
2017-11-03 02:10:19.972182 7fdc044a1700  0 mon.ip-172-16-13-217@1(probing) e0 ms_verify_authorizer cephx enabled, but no authorizer (required for mon)
2017-11-03 02:10:19.972187 7fdc044a1700  0 -- 172.16.13.217:6789/0 >> 172.16.14.152:6789/0 conn(0x55d10f9cb000 :6789 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg: got bad authorizer
2017-11-03 02:10:20.522536 7fdc044a1700  0 mon.ip-172-16-13-217@1(probing) e0 ms_verify_authorizer cephx enabled, but no authorizer (required for mon)
2017-11-03 02:10:20.522544 7fdc044a1700  0 -- 172.16.13.217:6789/0 >> 172.16.12.99:6789/0 conn(0x55d10f91d800 :6789 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg: got bad authorizer
2017-11-03 02:10:20.523929 7fdc044a1700  0 mon.ip-172-16-13-217@1(probing) e0 ms_verify_authorizer cephx enabled, but no authorizer (required for mon)
2017-11-03 02:10:20.523936 7fdc044a1700  0 -- 172.16.13.217:6789/0 >> 172.16.12.99:6789/0 conn(0x55d10f91d800 :6789 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg: got bad authorizer

Monitor pillar file:

include:
  - ceph/clusters/cluster-01/common
ceph:
  mon:
    enabled: true
    keyring:
      mon:
        key: [pgp monitor key here]
        caps:
          mon: "allow *"
      admin:
        key: [pgp admin key here]
        caps:
          mds: "allow *"
          mgr: "allow *"
          mon: "allow *"
          osd: "allow *"
  mgr:
    enabled: true
    dashboard:
      enabled: true
      port: 7000

Common pillar file

#!jinja|yaml|gpg
ceph:
  common:
    version: luminous
    fsid: 1390641e-39d6-4000-9761-928429d7d7c2
    members:
    - name: ip-172-16-12-99
      host: 172.16.12.99
    - name: ip-172-16-13-217
      host: 172.16.13.217
    - name: ip-172-16-14-152
      host: 172.16.14.152
    keyring:
      admin:
        key: [pgp admin key here]
        caps:
          mds: "allow *"
          mgr: "allow *"
          mon: "allow *"
          osd: "allow *"

[etfeet]

i was able to get this working by replacing the mon_keyring jinja template that uses salt mine with one that uses the pillar data.

{%- from "ceph/map.jinja" import mon with context %}
{%- for key in mon.keyring %}
{%- set keyring = mon['keyring'][key] %}
{% if key == 'mon' %}
[{{key}}.]
    key = {{ keyring.key  }}
    {%- for cap_name, cap in  keyring.caps.iteritems() %}
    caps {{ cap_name }} = "{{ cap }}"
    {%-endfor %}

{%- endif %}
{%- endfor %}

[dvandok]

The documentation seems to leave out what functions to expose to the salt mine, but it suggests that grains.items is one of them. The way the key material is looked up by grain makes this very insecure, if I understand this correctly it exposes the grains to all minions. I.e. it takes only a single security compromised minion to leak your ceph keys. Oops!
The more secure setup is to keep key material in the pillar which is only exposed to the minion it is for. Thanks @etfeet for that reworked version.