Puppet – Aggregate, resolve, validate and filter IP addresses

By gbe0Posted Networking, Puppet/Ruby

When working with lists of IP’s with Puppet there are some common tasks that you will most likely encounter:

  • Aggregate IP’s – As an example, if you have a firewall rule and the source list is like this:
## Allow inbound traffic from these hosts
mymodule::firewall::allow:
    - 192.0.2.5/24
    - 192.0.2.0/24
    - 192.0.2.55

That list can be condensed into a single rule for only 192.0.2.0/24.

  • You may also be working with hostnames which need to be resolved to IP addresses (optionally IPv4 or IPv6 only). These hostnames also may have multiple IP’s associated with them, some modules do not take this into account and only add a single IP. This results either in each Puppet run resulting in a configuration change as the IP gets swapped or the additional IP address(es) completely being ignored.
  • You may be working with IPv6 IP’s which can be specified in many different possible formats. As an example, these IP’s are all equal:
## Example of IPv6 addresses that are all equivalent (not an exhaustive list)
mymodule::firewall::ipv6_example:
    - 2001:0DB8:0:0:0:0:0:0/128
    - 2001:0db8:0:0:0:0:0:0/128
    - 2001:0DB8::0/128
    - 2001:0db8::0/128
    - 2001:0DB8::0
    - 2001:0db8::0
    - 2001:0DB8::0:0
    - 2001:0db8::0:0
    - 2001:0DB8::0:0:0
    - 2001:0db8::0:0:0

Most modules, especially firewall modules, will treat each of the IP’s above as unique IP’s. This results in unnecessary rules being added and could have a significant performance penalty.

  • You may be working with modules that only support IPv4 and/or IPv6 IP’s but you would like to provide one or more hostnames instead and have the IP’s resolved with DNS; it is much more convenient to automatically have firewall rules updated if the IP changes rather than having to replace any occurrences of the IP to the new one.
  • You may be working with modules that only support IPv4 or only support IPv6 IP’s but the list of IP’s you want to use as a source contains both. This means you will need to filter out any IPv4 or IPv6 IP’s (and handle DNS resolution if required).
  • You may want to ensure that IP’s are valid and in the case of hostnames resolve to an IPv4/IPv6 address.

All of these problems can be solved in a resuable manner without having to write handlers inside your modules using a Puppet function written in Ruby. For my use case, I have a common module that all nodes include named “core”. I have put these functions inside that module so that it is available everywhere. The files must be in the “<module>/lib/puppet/functions/” directory, in my case this is:

/etc/puppet/code/environments/production/modules/core/lib/puppet/functions

I am far from a Ruby expert so there is more than likely a neater/better way to do this, if you have any suggestions please leave a comment or contact me.

normalize_prefix

The normalize_prefix function accepts an array of prefixes/hostnames, loop over each one and do the following:

  • Test if the supplied IP/hostname is a valid IPv4 or IPv6 address. If it is, the address is normalized (ensuring that single IP’s in the format 192.0.2.0 get formatted as 192.0.2.0/32 and IPv6 IP’s are compacted and lowercased).
  • If the supplied address is not an IP address, it will attempt to be resolved as a hostname. If the filter option was set to a specific address family, it will only be resolved to that address family.
  • All IP’s set to lowercase.
  • All results will be combined into an array and any duplicates removed.
  • All remaining results will then be checked to see if there is an overlapping prefix in place (eg. if the array contains 192.0.2.4/32 and 192.0.2.0/24 it will result in 192.0.2.4/32 being removed).
# frozen_string_literal: true

# Require resolv library
require 'resolv'

# Create the normalize_prefix Puppet function
Puppet::Functions.create_function(:normalize_prefix) do # rubocop:disable Metrics/BlockLength
  # Normalize an array of IP addresses/hostnames/prefixes.
  #
  # @param list
  #   The list of IP addresses/hostnames/prefixes to normalize.
  #
  # @optional_param strict
  #   Enable/disable strict mode. Strict mode will raise an error if there is validation
  #   issues instead of ignoring the item.
  #
  # @optional_param filter
  #   Filter to ipv4 or ipv6
  #
  # @return [Array] Returns the list of IP addresses/hostnames normalized
  # @example Normalize a list of IP's (example.com resolves to 192.0.2.255
  # and 2001:db8::f)
  #   normalize_prefix([ '192.0.2.4/24', 'example.com', '2001:db8::0003' ]) =>
  #     [ '192.0.2.0/24', '2001:db8::3/128', '2001:db8::f/128' ]

  # Basic normalize function; only array of IP's provided but no filtering and strict mode enabled
  dispatch :normalize_basic do
    required_param 'Array', :list
    return_type 'Array'
  end

  # Normalize function with ability to set strict mode
  dispatch :normalize_strict do
    required_param 'Array', :list
    required_param 'Boolean', :strict
    return_type 'Array'
  end

  # Normalize function with ability to filter to an address family
  dispatch :normalize_filter do
    required_param 'Array', :list
    required_param 'Enum[ ipv4, ipv6 ]', :filter
    return_type 'Array'
  end

  # Normalize function with ability to set strict mode and filter to an address family
  dispatch :normalize_strict_filter do
    required_param 'Array', :list
    required_param 'Boolean', :strict
    required_param 'Enum[ ipv4, ipv6 ]', :filter
    return_type 'Array'
  end

  # Handlers
  def normalize_basic(list)
    normalize(list)
  end

  def normalize_strict(list, strict)
    normalize(list, strict: strict)
  end

  def normalize_filter(list, filter)
    normalize(list, filter: filter)
  end

  def normalize_strict_filter(list, strict, filter)
    normalize(list, strict: strict, filter: filter)
  end

  # Normalize set of prefixes
  def normalize(list, strict: true, filter: nil) # rubocop:disable Metrics/MethodLength,Metrics/AbcSize,Metrics/CyclomaticComplexity,Metrics/PerceivedComplexity
    # If the list of IP's to normalize is empty and strict mode is enabled
    # an error will be raised as there is nothing to do.
    if list.empty? && strict == true
      raise Puppet::ParseError, "Empty array of IP's/hostnames/prefixes was provided to \
      normalize_prefix; there must be at least one IP, prefix or hostname to normalize."
    end

    # Return empty list if not using strict mode but the list is empty
    return [] if list.empty?

    # Create empty list of addresses that will be used to store
    # post-processed items
    processed = []

    # Loop over each IP/hostname/prefix in list
    list.each do |host|
      # Create empty array of IP's for this run
      ips = []

      # If the host is an IP address and matches filter then no further processing needed
      validate_ip(ip: host, filter: filter) do |ip|
        ips << ip
      end

      # If the IP was not validated, try resolving it instead
      if ips.empty?
        resolve_ip(host: host, filter: filter) do |ip|
          (ips << ip).flatten!
        end
      end

      # Add IP's to processed list
      processed.push(*ips) unless ips.empty?
    end

    # Check if list to return is empty
    if processed.empty? && strict
      raise Puppet::ParseError, "Could not parse the IP/prefix list '#{list}' to a set of prefixes. \
      Strict mode is enabled which makes this a fatal error."
    end

    return [] if processed.empty?

    # Return the cleaned process list
    clean_prefixes(list: processed)
  end

  # Check if the IP is IP4 or IPv6
  def iptype(ip:)
    begin
      address = IPAddr.new ip
    rescue # rubocop:disable Style/RescueStandardError
      raise Puppet::ParseError, "Could not parse IP '#{ip}' to determine type"
    end
    yield 'ipv4' if address.ipv4?
    yield 'ipv6' if address.ipv6?
  end

  # Validate an IP address is actually an IP address of the correct type
  def validate_ip(ip:, filter: nil)
    # Attempt to create IPAddr object; if this fails it means its not an IP
    begin
      ip = IPAddr.new ip
    rescue # rubocop:disable Style/RescueStandardError
      return
    end

    # Stop function if type is incorrect
    return if filter.eql?('ipv6') && ip.ipv4?
    return if filter.eql?('ipv4') && ip.ipv6?

    # Return normalized IP address
    yield [ip.to_s, '/', ip.prefix].join.downcase
  end

  # Attempt to resolve a hostname
  def resolve_ip(host:, filter: nil) # rubocop:disable Metrics/MethodLength,Metrics/AbcSize
    # Create empty list of results
    results = []

    # Get the IP's
    begin
      # Resolve the addresses
      addresses = case filter
                  when 'ipv4'
                    Resolv::DNS.new.getresources(host, Resolv::DNS::Resource::IN::A)
                  when 'ipv6'
                    Resolv::DNS.new.getresources(host, Resolv::DNS::Resource::IN::AAAA)
                  else
                    Resolv::DNS.new.getresources(host, Resolv::DNS::Resource::IN::A) + \
                    Resolv::DNS.new.getresources(host, Resolv::DNS::Resource::IN::AAAA)
                  end
      # Convert the results to an IP address string
      addresses.each do |address|
        ip = IPAddr.new address.address.to_s
        results << [ip.to_s, '/', ip.prefix].join.downcase
      end
    rescue # rubocop:disable Style/RescueStandardError
      return
    end

    # If there are no results then return
    return if results.empty?

    # Pass the array back to requesting method
    yield results
  end

  # Clean a list of prefixes by removing all overlapping prefixes
  def clean_prefixes(list:) # rubocop:disable Metrics/AbcSize,Metrics/CyclomaticComplexity,Metrics/MethodLength,Metrics/PerceivedComplexity
    # Make list unique
    list = list.uniq

    ## Loop over each prefix in the list and check if it is part of another prefix; this is used to deduplicate entries
    list.each do |prefix|
      ## Loop over each prefix again for testing/comparison
      list.each do |compare|
        ## Skip self comparison
        prefix.to_s == compare.to_s && next
        ## Test if prefix is contained with in the comparison prefix
        if IPAddr.new(compare).include? IPAddr.new(prefix).to_s
          ## Comparison prefix contains test prefix; test prefix should be removed from list
          list -= [prefix]
        end
      end
    end

    ## Split into IPv4 and IPv6 to make more visually pleasing sort
    ipv4 = []
    ipv6 = []
    list.each do |prefix|
      begin
        address = IPAddr.new prefix
      rescue # rubocop:disable Style/RescueStandardError
        raise Puppet::ParseError, "Could not parse '#{prefix}' to determine type"
      end
      ipv4 << prefix if address.ipv4?
      ipv6 << prefix if address.ipv6?
    end

    ## Sort and return
    ipv4.sort + ipv6.sort
  end
end
normalize_prefix.rbDownload

Usage

To use the function in Puppet modules:

## The list of prefixes to normalize
$original_prefix_list = [ '192.0.2.0', '192.0.2.0/24', 'example.com', '2001:db8::0003', '2001:0DB8:0000:0000:0000:0000:0000:0003/128' ]

## Return the normalized prefix list
## example.com must resolve to an IP address (IPv4 and/or IPv6)
## In this case, example.com resolves to 192.0.2.50
$normalized_prefix_list_resolved = normalize_prefix($original_prefix_list) ## Returns [ '192.0.2.0/24', '2001:db8::3/128' ]

## Return the normalized prefix list
## example.com must resolve to an IP address (IPv4 and/or IPv6)
## In this case, example.com does not resolve to any IP address
$normalized_prefix_list_notresolved = normalize_prefix($original_prefix_list) ## Raises Parser Error

## Return the normalized prefix list
## In this case, example.com does not resolve to an IP address it will be ignored
$normalized_prefix_list_loose = normalize_prefix($original_prefix_list, false) ## Returns [ '192.0.2.0/24', '2001:db8::3/128' ]

## Filter the list to only IPv4 prefixes
$normalized_ipv4 = normalize_prefix($original_prefix_list, 'ipv4') ## Returns [ '192.0.2.0/24' ]

## Filter the list to only IPv6 prefixes
$normalized_ipv6 = normalize_prefix($original_prefix_list, 'ipv6') ## Returns [ '2001:db8::3/128' ]

## Create two separate prefix lists
$list_one = [ '192.0.2.0/29', '192.0.2.0/28', '192.0.2.50/32', '192.0.2.250', '2001:dB8:0::/64', '2001:0db8::/32' ]
$list_two = [ '192.0.2.0/24', '2001:db8:ffff::/48', '2001:db8:fafa:fafa:fafa:fafa:fafa:fafa' ]

## Merge the two prefix lists into a single normalized list
$merged = normalize_prefix(concat($list_one, $list_two)) ## Returns [ '192.0.2.0/24', '2001:db8::/32' ]

To use the function in Puppet templates (erb) it can be executed using scope.call_function:

## Return the normalized prefix list
<% prefixes = scope.call_function('normalize_prefix', [ @original_prefix_list ]) -%>

## Return the normalized prefix list filtered for only IPv4 prefixes
<% prefixes = scope.call_function('normalize_prefix', [ @original_prefix_list, 'ipv4' ]) -%>

## Return the normalized prefix list filtered for only IPv6 prefixes
<% prefixes = scope.call_function('normalize_prefix', [ @original_prefix_list, 'ipv6' ]) -%>

## Return the normalized prefix list with strict mode disabled (allow cases where hostnames may not resolve to IP addresses and allow empty lists to be returned)
<% prefixes = scope.call_function('normalize_prefix', [ @original_prefix_list, false ]) -%>

## Return the normalized prefix list with strict mode disabled filtered for IPv4
<% prefixes = scope.call_function('normalize_prefix', [ @original_prefix_list, false, 'ipv4' ]) -%>

normalize_ip

The normalize_ip function behaves in a similar way to the normalize_prefixes function except it is for IP addresses not prefixes.

# frozen_string_literal: true

# Require resolv library
require 'resolv'

# Create the normalize_ip Puppet function
Puppet::Functions.create_function(:normalize_ip) do # rubocop:disable Metrics/BlockLength
  # Normalize an array of IP addresses/hostnames/prefixes.
  #
  # @param list
  #   The list of IP addresses/hostnames to normalize.
  #
  # @optional_param strict
  #   Enable/disable strict mode. Strict mode will raise an error if there is validation
  #   issues instead of ignoring the item.
  #
  # @optional_param filter
  #   Filter to ipv4 or ipv6
  #
  # @return [Array] Returns the list of IP addresses/hostnames normalized
  # @example Normalise a list of IP's (example.com resolves to 192.0.2.255
  # and 2001:db8::f)
  #   normalize_ip([ '192.0.2.0/32', 'example.com', '2001:db8::0003' ]) =>
  #     [ '192.0.2.0', '192.0.2.255', '2001:db8::3', '2001:db8::f' ]

  # Basic normalize function; only array of IP's provided but no filtering and strict mode enabled
  dispatch :normalize_basic do
    required_param 'Array', :list
    return_type 'Array'
  end

  # Normalize function with ability to set strict mode
  dispatch :normalize_strict do
    required_param 'Array', :list
    required_param 'Boolean', :strict
    return_type 'Array'
  end

  # Normalize function with ability to filter to an address family
  dispatch :normalize_filter do
    required_param 'Array', :list
    required_param 'Enum[ ipv4, ipv6 ]', :filter
    return_type 'Array'
  end

  # Normalize function with ability to set strict mode and filter to an address family
  dispatch :normalize_strict_filter do
    required_param 'Array', :list
    required_param 'Boolean', :strict
    required_param 'Enum[ ipv4, ipv6 ]', :filter
    return_type 'Array'
  end

  # Handlers
  def normalize_basic(list)
    normalize(list)
  end

  def normalize_strict(list, strict)
    normalize(list, strict: strict)
  end

  def normalize_filter(list, filter)
    normalize(list, filter: filter)
  end

  def normalize_strict_filter(list, strict, filter)
    normalize(list, strict: strict, filter: filter)
  end

  # Normalize set of prefixes
  def normalize(list, strict: true, filter: nil) # rubocop:disable Metrics/MethodLength,Metrics/AbcSize,Metrics/CyclomaticComplexity,Metrics/PerceivedComplexity
    # If the list of IP's to normalize is empty and strict mode is enabled
    # an error will be raised as there is nothing to do.
    if list.empty? && strict == true
      raise Puppet::ParseError, "Empty array of IP's was provided to \
      normalize_ip; there must be at least one IP or hostname to normalize."
    end

    # Return empty list if not using strict mode but the list is empty
    return [] if list.empty?

    # Create empty list of addresses that will be used to store
    # post-processed items
    processed = []

    # Loop over each IP/hostname in list
    list.each do |host|
      # Create empty array of IP's for this run
      ips = []
      # If the host is an IP address and matches filter then no further processing needed
      validate_ip(ip: host, filter: filter) do |ip|
        ips << ip
      end

      # If the IP was not validated, try resolving it instead
      if ips.empty?
        resolve_ip(host: host, filter: filter) do |ip|
          (ips << ip).flatten!
        end
      end

      # Add IP's to processed list
      processed.push(*ips) unless ips.empty?
    end

    # Check if list to return is empty
    if processed.empty? && strict
      raise Puppet::ParseError, "Could not parse the IP/prefix list '#{list}' to a set of IP's. \
      Strict mode is enabled which makes this a fatal error."
    end

    return [] if processed.empty?

    # Make sure the processed list is unique
    processed = processed.uniq

    ## Split into IPv4 and IPv6 to make more visually pleasing sort
    ipv4 = []
    ipv6 = []
    processed.each do |addr|
      begin
        address = IPAddr.new addr
      rescue # rubocop:disable Style/RescueStandardError
        raise Puppet::ParseError, "Could not parse '#{addr}' to determine type"
      end
      ipv4 << addr if address.ipv4?
      ipv6 << addr if address.ipv6?
    end

    ## Sort and return
    ipv4.sort + ipv6.sort
  end

  # Validate an IP address is actually an IP address of the correct type
  def validate_ip(ip:, filter: nil)
    # Attempt to create IPAddr object; if this fails it means its not an IP
    begin
      ip = IPAddr.new ip
    rescue # rubocop:disable Style/RescueStandardError
      return
    end

    # Stop function if type is incorrect
    return if filter.eql?('ipv6') && ip.ipv4?
    return if filter.eql?('ipv4') && ip.ipv6?

    # Return normalized IP address
    yield [ip.to_s].join.downcase
  end

  # Attempt to resolve a hostname
  def resolve_ip(host:, filter: nil) # rubocop:disable Metrics/MethodLength,Metrics/AbcSize
    # Create empty list of results
    results = []

    # Get the IP's
    begin
      # Resolve the addresses
      addresses = case filter
                  when 'ipv4'
                    Resolv::DNS.new.getresources(host, Resolv::DNS::Resource::IN::A)
                  when 'ipv6'
                    Resolv::DNS.new.getresources(host, Resolv::DNS::Resource::IN::AAAA)
                  else
                    Resolv::DNS.new.getresources(host, Resolv::DNS::Resource::IN::A) + \
                    Resolv::DNS.new.getresources(host, Resolv::DNS::Resource::IN::AAAA)
                  end
      # Convert the results to an IP address string
      addresses.each { |address| results << address.address.to_s.downcase }
    rescue # rubocop:disable Style/RescueStandardError
      return
    end

    # If there are no results then return
    return if results.empty?

    # Pass the array back to requesting method
    yield results
  end
end
normalize_ip.rbDownload

Usage

Usage of the script is the same as with the normalize_prefix function:

## The list of IP's to normalize
$original_ip_list = [ '192.0.2.0', '192.0.2.50', '192.0.2.51', 'example.com', '2001:db8::0003', '2001:0DB8:0000:0000:0000:0000:0000:0003' ]

## Return the normalized IP list
## example.com must resolve to an IP address (IPv4 and/or IPv6)
## In this case, example.com resolves to 192.0.2.50 and 2001:db8::3
$normalized_ip_list_resolved = normalize_prefix($original_ip_list) ## Returns [ '192.0.2.0', '192.0.2.50', '192.0.2.51', '2001:db8::3' ]

Leave a Reply

Your email address will not be published. Required fields are marked *