Puppet – Aggregate, resolve, validate and filter IP addresses

## Allow inbound traffic from these hosts
mymodule::firewall::allow:
    - 192.0.2.5/24
    - 192.0.2.0/24
    - 192.0.2.55

That list can be condensed into a single rule for only 192.0.2.0/24.

## Example of IPv6 addresses that are all equivalent (not an exhaustive list)
mymodule::firewall::ipv6_example:
    - 2001:0DB8:0:0:0:0:0:0/128
    - 2001:0db8:0:0:0:0:0:0/128
    - 2001:0DB8::0/128
    - 2001:0db8::0/128
    - 2001:0DB8::0
    - 2001:0db8::0
    - 2001:0DB8::0:0
    - 2001:0db8::0:0
    - 2001:0DB8::0:0:0
    - 2001:0db8::0:0:0

Most modules, especially firewall modules, will treat each of the IP’s above as unique IP’s. This results in unnecessary rules being added and could have a significant performance penalty.

• You may be working with modules that only support IPv4 and/or IPv6 IP’s but you would like to provide one or more hostnames instead and have the IP’s resolved with DNS; it is much more convenient to automatically have firewall rules updated if the IP changes rather than having to replace any occurrences of the IP to the new one.
• You may be working with modules that only support IPv4 or only support IPv6 IP’s but the list of IP’s you want to use as a source contains both. This means you will need to filter out any IPv4 or IPv6 IP’s (and handle DNS resolution if required).
• You may want to ensure that IP’s are valid and in the case of hostnames resolve to an IPv4/IPv6 address.

All of these problems can be solved in a resuable manner without having to write handlers inside your modules using a Puppet function written in Ruby. For my use case, I have a common module that all nodes include named “core”. I have put these functions inside that module so that it is available everywhere. The files must be in the “<module>/lib/puppet/functions/” directory, in my case this is:

/etc/puppet/code/environments/production/modules/core/lib/puppet/functions

I am far from a Ruby expert so there is more than likely a neater/better way to do this, if you have any suggestions please leave a comment or contact me.

normalize_prefix

The normalize_prefix function accepts an array of prefixes/hostnames, loop over each one and do the following:

• Test if the supplied IP/hostname is a valid IPv4 or IPv6 address. If it is, the address is normalized (ensuring that single IP’s in the format 192.0.2.0 get formatted as 192.0.2.0/32 and IPv6 IP’s are compacted and lowercased).
• If the supplied address is not an IP address, it will attempt to be resolved as a hostname. If the filter option was set to a specific address family, it will only be resolved to that address family.
• All IP’s set to lowercase.
• All results will be combined into an array and any duplicates removed.
• All remaining results will then be checked to see if there is an overlapping prefix in place (eg. if the array contains 192.0.2.4/32 and 192.0.2.0/24 it will result in 192.0.2.4/32 being removed).

# frozen_string_literal: true

# Require resolv library
require 'resolv'

# Create the normalize_prefix Puppet function
Puppet::Functions.create_function(:normalize_prefix) do # rubocop:disable Metrics/BlockLength
  # Normalize an array of IP addresses/hostnames/prefixes.
  #
  # @param list
  #   The list of IP addresses/hostnames/prefixes to normalize.
  #
  # @optional_param strict
  #   Enable/disable strict mode. Strict mode will raise an error if there is validation
  #   issues instead of ignoring the item.
  #
  # @optional_param filter
  #   Filter to ipv4 or ipv6
  #
  # @return [Array] Returns the list of IP addresses/hostnames normalized
  # @example Normalize a list of IP's (example.com resolves to 192.0.2.255
  # and 2001:db8::f)
  #   normalize_prefix([ '192.0.2.4/24', 'example.com', '2001:db8::0003' ]) =>
  #     [ '192.0.2.0/24', '2001:db8::3/128', '2001:db8::f/128' ]

  # Basic normalize function; only array of IP's provided but no filtering and strict mode enabled
  dispatch :normalize_basic do
    required_param 'Array', :list
    return_type 'Array'
  end

  # Normalize function with ability to set strict mode
  dispatch :normalize_strict do
    required_param 'Array', :list
    required_param 'Boolean', :strict
    return_type 'Array'
  end

  # Normalize function with ability to filter to an address family
  dispatch :normalize_filter do
    required_param 'Array', :list
    required_param 'Enum[ ipv4, ipv6 ]', :filter
    return_type 'Array'
  end

  # Normalize function with ability to set strict mode and filter to an address family
  dispatch :normalize_strict_filter do
    required_param 'Array', :list
    required_param 'Boolean', :strict
    required_param 'Enum[ ipv4, ipv6 ]', :filter
    return_type 'Array'
  end

  # Handlers
  def normalize_basic(list)
    normalize(list)
  end

  def normalize_strict(list, strict)
    normalize(list, strict: strict)
  end

  def normalize_filter(list, filter)
    normalize(list, filter: filter)
  end

  def normalize_strict_filter(list, strict, filter)
    normalize(list, strict: strict, filter: filter)
  end

  # Normalize set of prefixes
  def normalize(list, strict: true, filter: nil) # rubocop:disable Metrics/MethodLength,Metrics/AbcSize,Metrics/CyclomaticComplexity,Metrics/PerceivedComplexity
    # If the list of IP's to normalize is empty and strict mode is enabled
    # an error will be raised as there is nothing to do.
    if list.empty? && strict == true
      raise Puppet::ParseError, "Empty array of IP's/hostnames/prefixes was provided to \
      normalize_prefix; there must be at least one IP, prefix or hostname to normalize."
    end

    # Return empty list if not using strict mode but the list is empty
    return [] if list.empty?

    # Create empty list of addresses that will be used to store
    # post-processed items
    processed = []

    # Loop over each IP/hostname/prefix in list
    list.each do |host|
      # Create empty array of IP's for this run
      ips = []

      # If the host is an IP address and matches filter then no further processing needed
      validate_ip(ip: host, filter: filter) do |ip|
        ips << ip
      end

      # If the IP was not validated, try resolving it instead
      if ips.empty?
        resolve_ip(host: host, filter: filter) do |ip|
          (ips << ip).flatten!
        end
      end

      # Add IP's to processed list
      processed.push(*ips) unless ips.empty?
    end

    # Check if list to return is empty
    if processed.empty? && strict
      raise Puppet::ParseError, "Could not parse the IP/prefix list '#{list}' to a set of prefixes. \
      Strict mode is enabled which makes this a fatal error."
    end

    return [] if processed.empty?

    # Return the cleaned process list
    clean_prefixes(list: processed)
  end

  # Check if the IP is IP4 or IPv6
  def iptype(ip:)
    begin
      address = IPAddr.new ip
    rescue # rubocop:disable Style/RescueStandardError
      raise Puppet::ParseError, "Could not parse IP '#{ip}' to determine type"
    end
    yield 'ipv4' if address.ipv4?
    yield 'ipv6' if address.ipv6?
  end

  # Validate an IP address is actually an IP address of the correct type
  def validate_ip(ip:, filter: nil)
    # Attempt to create IPAddr object; if this fails it means its not an IP
    begin
      ip = IPAddr.new ip
    rescue # rubocop:disable Style/RescueStandardError
      return
    end

    # Stop function if type is incorrect
    return if filter.eql?('ipv6') && ip.ipv4?
    return if filter.eql?('ipv4') && ip.ipv6?

    # Return normalized IP address
    yield [ip.to_s, '/', ip.prefix].join.downcase
  end

  # Attempt to resolve a hostname
  def resolve_ip(host:, filter: nil) # rubocop:disable Metrics/MethodLength,Metrics/AbcSize
    # Create empty list of results
    results = []

    # Get the IP's
    begin
      # Resolve the addresses
      addresses = case filter
                  when 'ipv4'
                    Resolv::DNS.new.getresources(host, Resolv::DNS::Resource::IN::A)
                  when 'ipv6'
                    Resolv::DNS.new.getresources(host, Resolv::DNS::Resource::IN::AAAA)
                  else
                    Resolv::DNS.new.getresources(host, Resolv::DNS::Resource::IN::A) + \
                    Resolv::DNS.new.getresources(host, Resolv::DNS::Resource::IN::AAAA)
                  end
      # Convert the results to an IP address string
      addresses.each do |address|
        ip = IPAddr.new address.address.to_s
        results << [ip.to_s, '/', ip.prefix].join.downcase
      end
    rescue # rubocop:disable Style/RescueStandardError
      return
    end

    # If there are no results then return
    return if results.empty?

    # Pass the array back to requesting method
    yield results
  end

  # Clean a list of prefixes by removing all overlapping prefixes
  def clean_prefixes(list:) # rubocop:disable Metrics/AbcSize,Metrics/CyclomaticComplexity,Metrics/MethodLength,Metrics/PerceivedComplexity
    # Make list unique
    list = list.uniq

    ## Loop over each prefix in the list and check if it is part of another prefix; this is used to deduplicate entries
    list.each do |prefix|
      ## Loop over each prefix again for testing/comparison
      list.each do |compare|
        ## Skip self comparison
        prefix.to_s == compare.to_s && next
        ## Test if prefix is contained with in the comparison prefix
        if IPAddr.new(compare).include? IPAddr.new(prefix).to_s
          ## Comparison prefix contains test prefix; test prefix should be removed from list
          list -= [prefix]
        end
      end
    end

    ## Split into IPv4 and IPv6 to make more visually pleasing sort
    ipv4 = []
    ipv6 = []
    list.each do |prefix|
      begin
        address = IPAddr.new prefix
      rescue # rubocop:disable Style/RescueStandardError
        raise Puppet::ParseError, "Could not parse '#{prefix}' to determine type"
      end
      ipv4 << prefix if address.ipv4?
      ipv6 << prefix if address.ipv6?
    end

    ## Sort and return
    ipv4.sort + ipv6.sort
  end
end

Usage

To use the function in Puppet modules:

## The list of prefixes to normalize
$original_prefix_list = [ '192.0.2.0', '192.0.2.0/24', 'example.com', '2001:db8::0003', '2001:0DB8:0000:0000:0000:0000:0000:0003/128' ]

## Return the normalized prefix list
## example.com must resolve to an IP address (IPv4 and/or IPv6)
## In this case, example.com resolves to 192.0.2.50
$normalized_prefix_list_resolved = normalize_prefix($original_prefix_list) ## Returns [ '192.0.2.0/24', '2001:db8::3/128' ]

## Return the normalized prefix list
## example.com must resolve to an IP address (IPv4 and/or IPv6)
## In this case, example.com does not resolve to any IP address
$normalized_prefix_list_notresolved = normalize_prefix($original_prefix_list) ## Raises Parser Error

## Return the normalized prefix list
## In this case, example.com does not resolve to an IP address it will be ignored
$normalized_prefix_list_loose = normalize_prefix($original_prefix_list, false) ## Returns [ '192.0.2.0/24', '2001:db8::3/128' ]

## Filter the list to only IPv4 prefixes
$normalized_ipv4 = normalize_prefix($original_prefix_list, 'ipv4') ## Returns [ '192.0.2.0/24' ]

## Filter the list to only IPv6 prefixes
$normalized_ipv6 = normalize_prefix($original_prefix_list, 'ipv6') ## Returns [ '2001:db8::3/128' ]

## Create two separate prefix lists
$list_one = [ '192.0.2.0/29', '192.0.2.0/28', '192.0.2.50/32', '192.0.2.250', '2001:dB8:0::/64', '2001:0db8::/32' ]
$list_two = [ '192.0.2.0/24', '2001:db8:ffff::/48', '2001:db8:fafa:fafa:fafa:fafa:fafa:fafa' ]

## Merge the two prefix lists into a single normalized list
$merged = normalize_prefix(concat($list_one, $list_two)) ## Returns [ '192.0.2.0/24', '2001:db8::/32' ]

To use the function in Puppet templates (erb) it can be executed using scope.call_function:

## Return the normalized prefix list
<% prefixes = scope.call_function('normalize_prefix', [ @original_prefix_list ]) -%>

## Return the normalized prefix list filtered for only IPv4 prefixes
<% prefixes = scope.call_function('normalize_prefix', [ @original_prefix_list, 'ipv4' ]) -%>

## Return the normalized prefix list filtered for only IPv6 prefixes
<% prefixes = scope.call_function('normalize_prefix', [ @original_prefix_list, 'ipv6' ]) -%>

## Return the normalized prefix list with strict mode disabled (allow cases where hostnames may not resolve to IP addresses and allow empty lists to be returned)
<% prefixes = scope.call_function('normalize_prefix', [ @original_prefix_list, false ]) -%>

## Return the normalized prefix list with strict mode disabled filtered for IPv4
<% prefixes = scope.call_function('normalize_prefix', [ @original_prefix_list, false, 'ipv4' ]) -%>

# frozen_string_literal: true

# Require resolv library
require 'resolv'

# Create the normalize_ip Puppet function
Puppet::Functions.create_function(:normalize_ip) do # rubocop:disable Metrics/BlockLength
  # Normalize an array of IP addresses/hostnames/prefixes.
  #
  # @param list
  #   The list of IP addresses/hostnames to normalize.
  #
  # @optional_param strict
  #   Enable/disable strict mode. Strict mode will raise an error if there is validation
  #   issues instead of ignoring the item.
  #
  # @optional_param filter
  #   Filter to ipv4 or ipv6
  #
  # @return [Array] Returns the list of IP addresses/hostnames normalized
  # @example Normalise a list of IP's (example.com resolves to 192.0.2.255
  # and 2001:db8::f)
  #   normalize_ip([ '192.0.2.0/32', 'example.com', '2001:db8::0003' ]) =>
  #     [ '192.0.2.0', '192.0.2.255', '2001:db8::3', '2001:db8::f' ]

  # Basic normalize function; only array of IP's provided but no filtering and strict mode enabled
  dispatch :normalize_basic do
    required_param 'Array', :list
    return_type 'Array'
  end

  # Normalize function with ability to set strict mode
  dispatch :normalize_strict do
    required_param 'Array', :list
    required_param 'Boolean', :strict
    return_type 'Array'
  end

  # Normalize function with ability to filter to an address family
  dispatch :normalize_filter do
    required_param 'Array', :list
    required_param 'Enum[ ipv4, ipv6 ]', :filter
    return_type 'Array'
  end

  # Normalize function with ability to set strict mode and filter to an address family
  dispatch :normalize_strict_filter do
    required_param 'Array', :list
    required_param 'Boolean', :strict
    required_param 'Enum[ ipv4, ipv6 ]', :filter
    return_type 'Array'
  end

  # Handlers
  def normalize_basic(list)
    normalize(list)
  end

  def normalize_strict(list, strict)
    normalize(list, strict: strict)
  end

  def normalize_filter(list, filter)
    normalize(list, filter: filter)
  end

  def normalize_strict_filter(list, strict, filter)
    normalize(list, strict: strict, filter: filter)
  end

  # Normalize set of prefixes
  def normalize(list, strict: true, filter: nil) # rubocop:disable Metrics/MethodLength,Metrics/AbcSize,Metrics/CyclomaticComplexity,Metrics/PerceivedComplexity
    # If the list of IP's to normalize is empty and strict mode is enabled
    # an error will be raised as there is nothing to do.
    if list.empty? && strict == true
      raise Puppet::ParseError, "Empty array of IP's was provided to \
      normalize_ip; there must be at least one IP or hostname to normalize."
    end

    # Return empty list if not using strict mode but the list is empty
    return [] if list.empty?

    # Create empty list of addresses that will be used to store
    # post-processed items
    processed = []

    # Loop over each IP/hostname in list
    list.each do |host|
      # Create empty array of IP's for this run
      ips = []
      # If the host is an IP address and matches filter then no further processing needed
      validate_ip(ip: host, filter: filter) do |ip|
        ips << ip
      end

      # If the IP was not validated, try resolving it instead
      if ips.empty?
        resolve_ip(host: host, filter: filter) do |ip|
          (ips << ip).flatten!
        end
      end

      # Add IP's to processed list
      processed.push(*ips) unless ips.empty?
    end

    # Check if list to return is empty
    if processed.empty? && strict
      raise Puppet::ParseError, "Could not parse the IP/prefix list '#{list}' to a set of IP's. \
      Strict mode is enabled which makes this a fatal error."
    end

    return [] if processed.empty?

    # Make sure the processed list is unique
    processed = processed.uniq

    ## Split into IPv4 and IPv6 to make more visually pleasing sort
    ipv4 = []
    ipv6 = []
    processed.each do |addr|
      begin
        address = IPAddr.new addr
      rescue # rubocop:disable Style/RescueStandardError
        raise Puppet::ParseError, "Could not parse '#{addr}' to determine type"
      end
      ipv4 << addr if address.ipv4?
      ipv6 << addr if address.ipv6?
    end

    ## Sort and return
    ipv4.sort + ipv6.sort
  end

  # Validate an IP address is actually an IP address of the correct type
  def validate_ip(ip:, filter: nil)
    # Attempt to create IPAddr object; if this fails it means its not an IP
    begin
      ip = IPAddr.new ip
    rescue # rubocop:disable Style/RescueStandardError
      return
    end

    # Stop function if type is incorrect
    return if filter.eql?('ipv6') && ip.ipv4?
    return if filter.eql?('ipv4') && ip.ipv6?

    # Return normalized IP address
    yield [ip.to_s].join.downcase
  end

  # Attempt to resolve a hostname
  def resolve_ip(host:, filter: nil) # rubocop:disable Metrics/MethodLength,Metrics/AbcSize
    # Create empty list of results
    results = []

    # Get the IP's
    begin
      # Resolve the addresses
      addresses = case filter
                  when 'ipv4'
                    Resolv::DNS.new.getresources(host, Resolv::DNS::Resource::IN::A)
                  when 'ipv6'
                    Resolv::DNS.new.getresources(host, Resolv::DNS::Resource::IN::AAAA)
                  else
                    Resolv::DNS.new.getresources(host, Resolv::DNS::Resource::IN::A) + \
                    Resolv::DNS.new.getresources(host, Resolv::DNS::Resource::IN::AAAA)
                  end
      # Convert the results to an IP address string
      addresses.each { |address| results << address.address.to_s.downcase }
    rescue # rubocop:disable Style/RescueStandardError
      return
    end

    # If there are no results then return
    return if results.empty?

    # Pass the array back to requesting method
    yield results
  end
end

## The list of IP's to normalize
$original_ip_list = [ '192.0.2.0', '192.0.2.50', '192.0.2.51', 'example.com', '2001:db8::0003', '2001:0DB8:0000:0000:0000:0000:0000:0003' ]

## Return the normalized IP list
## example.com must resolve to an IP address (IPv4 and/or IPv6)
## In this case, example.com resolves to 192.0.2.50 and 2001:db8::3
$normalized_ip_list_resolved = normalize_prefix($original_ip_list) ## Returns [ '192.0.2.0', '192.0.2.50', '192.0.2.51', '2001:db8::3' ]