Filtering with Ansible’s selectattr()/rejectattr() when the tested attribute can be absent

Ansible’s selectattr and rejectattr allow filtering of a list of dictionaries based on a specific test being executed against each dictionary’s keys and values. But what if the dictionary is not guaranteed to always have the key being tested actually defined?

Welp, I didn’t really mean to post another post on what I originally considered to be Ansible 101 level knowledge that just I seem to be missing to have readily available (in spite of having written complex roles by now) but seeing that I apparently was wrong, here we are and it feels useful to write it down.

For the example scenario in this post imagine the following data structure representing fruit from a European perspective:

    fruit:
      - name: apple
        origin: local
      - name: pear
      - name: kiwi
        origin: exotic

We’re too lazy to repeat specifying origin explicitly for each element here and that’s a good thing: it’s not lazyness, it’s “optimize for the common case”! Sounds way better, right?

Now suppose we need a list of exotic fruits only. Usually we could use a filter expression like

  {{ fruit | selectattr('origin', 'equalto', 'exotic') }}

which equals Python code such as this

  domestic_fruits = [ f for f in fruit if f.origin == 'exotic' ]

But as just mentioned we don’t want to have to always specify origin explicitly and with the fruit list as shown above we’d get a 'dict object' has no attribute 'origin' error very quickly. What we’d want instead, of course, is that the test should simply pretend that exotic had been specified (because we’re spoiled brats and know more about exotic than local fruit). In Python we could simply write:

  domestic_fruits = [ f for f in fruit if not hasattr(f, "origin") or f.origin == "exotic" ]

But with Jinja filters this is not as easy and all of the following ideas that you could come up won’t help:

  # default() makes no sense since it is applied to the generator returned by selectattr():
  {{ fruit | selectattr('origin', 'equalto', 'exotic') | default('yes') }}

  # default() makes no sense since it would be applied to the string 'origin':
  {{ fruit | selectattr('origin' | default('exotic'), 'equalto', 'exotic') }}

  # This will prevent the error message but not include fruit that has 'exotic' left off in its definition
  {{ fruit | selectattr('origin', 'defined') | selectattr('origin', 'equalto', 'exotic') }}

  # selectattr/rejectattr do not support and/or operations, also: how should it know, which string
  # is still an argument to 'origin' and which is the second test?
  {{ fruit | selectattr('or', 'origin', 'not', 'defined', 'origin', 'equalto', 'exotic') }}

  # selectattr/rejectattr do not support brackets either
  {{ fruit | selectattr('or'(('origin', 'not', 'defined'), ('origin', 'equalto', 'exotic'))) }}

One of the key takeaways here is that selectattr and rejectattr simply take the first string argument and assume it’s a Jinja test to be called with all other strings being arguments to that test. Or in other words: what you specify in the brackets is a list of strings that get turned into an expression, not the expression to be applied itself.

Instead, a solution is to use union to combine two lists together: the list of fruit that does not have origin defined with the list of fruit that does have origin both defined and set to exotic. Because union acts on lists and both selectattr and rejectattr returns generators we also need to call list to have these generators generate actual lists:

  {{ fruit | rejectattr('origin', 'defined') | list |
     union( fruit | selectattr('origin', 'defined') | selectattr('origin', 'equalto', 'exotic') | list ) }}"

If you think that looks horribly complicated, have a look at this Foreman tasks file and say Hello to Evgeni from me ;) At some point you should of course consider writing a dedicated test in Python, though.