OER/Pfr ….Watching like hell……

Today we will discuss not so explored cisco outstanding feature OER/Pfr and see how it can monitor and make decisions on its own and route  traffic on the best available path in terms of jitter,delay,bandwidth consumption,etc..

I have taken a simple GNS3 topology,here R1 is serving as Master Controller(MC) the core brain of the OER/Pfr that makes routing decisions the key is there needs to be at least two or more exit points, and R2 and R4 serving as Border Routers(BR) ,here R2 is connected to ISP-1 and R4 is connected to ISP-2 , i have also taken HOST-1 and HOST-2 and disable routing there to see the traffic flow from the hosts perspective.The topology is shown below.

TOPOLOGY Overview:-

Here R1,R2 and R4 are in AS 64511 and Confederation  identifier is 124.The IGP running between R1,R2 and R4 is EIGRP 124 and they are peered with there loopback0 addresses for R1 loopback0 is 1.1.1.1/32 likewise on every router.

R1#sh ip bgp summ | b Neigh
Neighbor        V    AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
2.2.2.2         4 64511      10      11        7    0    0 00:06:05        3
4.4.4.4         4 64511      10      11        7    0    0 00:06:00        3

R2 and R4 peered with ISP-1(AS-1) and ISP-2(AS-2).

R2#sh ip bgp summ | b Neigh
Neighbor        V    AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
1.1.1.1         4 64511      11      10        7    0    0 00:06:34        1
21.21.21.1      4     1      10      10        7    0    0 00:06:29        3

R4#sh ip bgp summ | b Nei
Neighbor        V    AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
1.1.1.1         4 64511      12      11        7    0    0 00:07:00        3
42.42.42.2      4     2      11      11        7    0    0 00:07:02        3

On HOST-1 and HOST-2 the ip routing is disabled and Default gateway provided to R1 FastEthernet0/0 interface.

HOST-1#sh ip route
Default gateway is 1.1.124.1

OER/Pfr Configuration:-

We start with R1(MC) , we first create a key chain for communication between R1(MC) and R2,R4(BR’s).

R1(config)#key chain OER
R1(config-keychain)#key 1
R1(config-keychain-key)#key-string cisco

Next we start configuring the Master Controller portion on R1, almost all the OER related configs are done on MC, here we enable logging,specify port to communicate(we have taken port 1790) , specify who the Border Routers are , along with the Exit interfaces on BR’s to the ISP-1 and ISP-2.

R1(config)#oer master
R1(config-oer-mc)#logging
R1(config-oer-mc)#port 1790
R1(config-oer-mc)#border 2.2.2.2 key-chain OER
R1(config-oer-mc-br)#interface fa0/0 internal
R1(config-oer-mc-br)#interface s0/0 external
R1(config-oer-mc-br-if)#exi
R1(config-oer-mc-br)#exi
R1(config-oer-mc)#border 4.4.4.4 key-chain OER
R1(config-oer-mc-br)#interface fa0/0 internal
R1(config-oer-mc-br)#interface s0/0 external
R1(config-oer-mc-br-if)#exit

NOTE:-Here under Border Sub-config mode the interfaces for the BR’s are the interfaces on BR’s not on MC.

So the basic config part for MC is done above , now we configure the BR’s.Here on BR’s we create key chain , then tell BR’s who is the MC , we will also tell BR to use its Loopback0 to send active probes.Active probes are one method (available with OER) to determine delay.

key chain OER
key 1
key-string cisco
oer border
local Loopback0
port 1790
master 1.1.1.1 key-chain OER
active-probe address source interface Loopback0

Same config done on R4 also.

As soon as we enter above configs on R2 and R4 we will see the OER process starts on MC.

R1#
*Mar  1 00:30:43.303: %OER_MC-5-NOTICE: BR 2.2.2.2 UP
*Mar  1 00:30:43.315: %OER_MC-5-NOTICE: BR 2.2.2.2 IF Se0/0 UP
*Mar  1 00:30:43.339: %OER_MC-5-NOTICE: BR 2.2.2.2 IF Fa0/0 UP
*Mar  1 00:30:43.339: %OER_MC-5-NOTICE: BR 2.2.2.2 Active
R1#
*Mar  1 00:30:50.263: %OER_MC-5-NOTICE: BR 4.4.4.4 UP
*Mar  1 00:30:50.287: %OER_MC-5-NOTICE: BR 4.4.4.4 IF Se0/0 UP
*Mar  1 00:30:50.295: %OER_MC-5-NOTICE: BR 4.4.4.4 IF Fa0/0 UP
*Mar  1 00:30:50.295: %OER_MC-5-NOTICE: BR 4.4.4.4 Active
*Mar  1 00:30:50.295: %OER_MC-5-NOTICE: MC Active

We can also verify with the command “sh oer master” on R1 and “show oer border” on R2 and R4.

R1#sh oer master
OER state: ENABLED and ACTIVE
Conn Status: SUCCESS, PORT: 1790
Version: 2.1
Number of Border routers: 2
Number of Exits: 2
Number of monitored prefixes: 0 (max 5000)
Max prefixes: total 5000 learn 2500
Prefix count: total 0, learn 0, cfg 0

Border           Status   UP/DOWN             AuthFail  Version
4.4.4.4          ACTIVE   UP       00:00:23          0  2.1
2.2.2.2          ACTIVE   UP       00:00:30          0  2.1

R2#sh oer border
OER BR 2.2.2.2 ACTIVE, MC 1.1.1.1 UP/DOWN: UP 00:05:14,
Auth Failures: 0
Conn Status: SUCCESS, PORT: 1790
Version: 2.1  MC Version: 2.1
Exits
Fa0/0           INTERNAL
Se0/0           EXTERNAL

Next on the controller, we  specify that we will learn prefixes based on throughput and delay, set time periods regarding learning, and tell the controller the network lengths to learn. In this example, due to the small network, we are going to have the controller learn /32 routes(here i have injected two same routes from ISP-1 and ISP-2 both with 6.6.6.6/32 and 7.7.7.7/32) . We could also specify a maximum number of prefixes to learn as well. (Note: In a production network, you would not want thousands of static routes for sites visited by users. There is a default maximum, and the default aggregation is /24)..

ISP-2#sh ip bgp | b Netw
Network          Next Hop            Metric LocPrf Weight Path
*> 6.6.6.6/32       0.0.0.0                  0         32768 ?
*> 7.7.7.7/32       0.0.0.0                  0         32768 ?
ISP-1#sh ip bgp | B Netw
Network          Next Hop            Metric LocPrf Weight Path
*> 6.6.6.6/32       0.0.0.0                  0         32768 ?
*> 7.7.7.7/32       0.0.0.0                  0         32768 ?
R1(config)#oer master
R1(config-oer-mc)#learn
R1(config-oer-mc-learn)#throughput
R1(config-oer-mc-learn)#delay
R1(config-oer-mc-learn)#periodic-interval 1
R1(config-oer-mc-learn)#monitor-period 2
R1(config-oer-mc-learn)#expire after time 30
*Mar  1 00:42:01.443: %OER_MC-5-NOTICE: Prefix Learning STARTED
R1(config-oer-mc-learn)#aggregation-type prefix-length 32
R1(config-oer-mc-learn)#exit

Next, now that we have exited out of “learn mode”, we can configure some of the other properties of the controller, such as how often to make prefix decisions (the backoff period), allow it to put routes in the routing table (mode route control), and choose the best exit interface (mode select-exit best). OER will create /32 static routes that use that use the “best” interface to reach the destination network.

R1(config-oer-mc)#backoff 180 360
R1(config-oer-mc)#mode route control
*Mar  1 00:44:41.683: %OER_MC-5-NOTICE: Prefix Learning WRITING DATA
R1(config-oer-mc)#mode select-exit best

A few additional options include how often to make policy decisions, (periodic command), and which policy items are the most important regarding making those policy decisions (resolve commands).  Highest priority is 1, and lowest is 10.

R1(config-oer-mc)#periodic 180
R1(config-oer-mc)#resolve loss priority 1 variance 1
R1(config-oer-mc)#resolve delay priority 2 variance 1
R1(config-oer-mc)#resolve utilization priority 3 variance 1
R1(config-oer-mc)#resolve range priority 4

As we have configured different parameters to use with OER , we will issue some Show commands to verify that our configurations are in place.

R1#sh oer master
OER state: ENABLED and ACTIVE
Conn Status: SUCCESS, PORT: 1790
Version: 2.1
Number of Border routers: 2
  Number of Exits: 2
Number of monitored prefixes: 0 (max 5000)
Max prefixes: total 5000 learn 2500
Prefix count: total 0, learn 0, cfg 0

Border           Status   UP/DOWN             AuthFail  Version
4.4.4.4          ACTIVE   UP       00:18:03          0  2.1
2.2.2.2          ACTIVE   UP       00:18:10          0  2.1

Global Settings:
max-range-utilization percent 20 recv 0
mode route metric bgp local-pref 5000
mode route metric static tag 5000
trace probe delay 1000
logging

Default Policy Settings:
backoff 180 360 180
delay relative 50
holddown 300
periodic 180
probe frequency 56
mode route control
mode monitor both
mode select-exit best
loss relative 10
jitter threshold 20
mos threshold 3.60 percent 30
unreachable relative 50
resolve loss priority 1 variance 1
  resolve delay priority 2 variance 1
  resolve utilization priority 3 variance 1
  resolve range priority 4 variance 0

Learn Settings:
current state : STARTED
time remaining in current state : 168 seconds
throughput
  delay
no inside bgp
no protocol
monitor-period 2
periodic-interval 1
aggregation-type prefix-length 32
prefixes 100
expire after time 30

Now generate some traffic in the background and allow OER to dynamically learn the specific routes , as well as learn best path to those routes added to the routing table of R2 and R4 (BR). We will create two SLA’s from R1 directed to the Loopback interfaces of ISP-1 and ISP-2,(6.6.6.6/32 and 7.7.7.7/32).
R1(config)#ip sla 1
R1(config-ip-sla)#tcp-connect 6.6.6.6 4000
R1(config-ip-sla-tcp)#time
R1(config-ip-sla-tcp)#timeout 100
R1(config-ip-sla-tcp)#frequency 1
R1(config-ip-sla-tcp)#exit
R1(config)#ip sla schedule 1 start-time now life forever
R1(config)#ip sla 2
R1(config-ip-sla)#tcp-connect 7.7.7.7 5000
R1(config-ip-sla-tcp)#timeout 100
R1(config-ip-sla-tcp)#frequency 1
R1(config-ip-sla-tcp)#exit
R1(config)#ip sla 2
*Mar  1 00:59:46.751: %OER_MC-5-NOTICE: Prefix Learning WRITING DATA
R1(config)#ip sla schedule 2 start-time now life forever

On R2 and R4 we will set up “ip sla responder” , so that they will correctly respond to the tcp-connect requests sourced from R1. ON R2 and R4.

R2(config)#ip sla responder
R2(config)#exit

On R2 and R4, lets see what networks have been learned. (Note: Based on the timers, it may take a ~minute or so for the /32 networks of 6.6.6.6 and 7.7.7.7 to be learned.

R2#sh oer border passive cache prefix

OER Passive Prefix Cache, State: enabled, 278544 bytes
2 active, 4094 inactive, 26 added
1361 ager polls, 0 flow alloc failures
Active flows timeout in 1 minutes
Inactive flows timeout in 15 seconds
IP Sub Flow Cache, 25800 bytes
6 active, 1018 inactive, 78 added, 26 added to flow
0 alloc failures, 0 force free
1 chunk, 1 chunk added

Prefix              NextHop         Src If         Dst If          Flows
Pkts   B/Pk  Active   sDly   #Dly  PktLos  #UnRch
————————————————————————
7.7.7.7/32          21.21.21.1      Fa0/0          Se0/0              7
10      80    64.7      0      0       0       0
6.6.6.6/32          21.21.21.1      Fa0/0          Se0/0              7
10      80    64.1      0      0       0       0

Here 6.6.6.6/32 and 7.7.7.7/32 both are learning from ISP-1 , lets trace from HOST-1 and HOST-2 to confirm our statistics.

HOST-1#trace 6.6.6.6

Type escape sequence to abort.
Tracing the route to 6.6.6.6

1 1.1.124.1 1004 msec 4 msec 4 msec
2 21.21.21.1 1008 msec 4 msec

HOST-2#trace 7.7.7.7

Type escape sequence to abort.
Tracing the route to 7.7.7.7

1 1.1.124.1 1004 msec 4 msec 4 msec
2 21.21.21.1 1008 msec 4 msec
So now the path is confirmed as we traced the same Loopback interfaces on two different ISP’s from our HOSTs however the path is taken best according to the configured parameters on R1(MC).

Lets tweek some parameters on R2 and make it worse so that both the /32 destinations will be reachable from R4 that is ISP-2.

So now that OER has decided the best path, using R2 Serial0/0 the best path for those 2 networks, lets tip the scale in the other direction by using generic  traffic shaping on Serial0/0 of R2, so that the delay there will be worse than R4, and watch the results. (Note: in addition to the generic traffic shaping on R2 Serial0/0, I also ran an extended ping sourced from 6.6.6.6 and 7.7.7.7 to increase the overall delay).

R2(config)#int s0/0
R2(config-if)#traffic-shape rate 10000 1000 0
R2#sh traffic-shape
Interface   Se0/0
Access Target    Byte   Sustain   Excess    Interval  Increment Adapt
VC     List   Rate      Limit  bits/int  bits/int  (ms)      (bytes)   Active
–             10000     125    1000      0         100       125       –

R2#debug oer border active-probes detail

probeType = echo, probeTarget = 6.6.6.6, probeTargetPort = 0
probeSource = Default, probeSourcePort = 0, probeNextHop = Default
probeIfIndex = 2 probeToS = 0 policy_seq = 0
*Mar  1 01:38:03.131: OER BR APE detail: Completed retrieving Probe Statistics.
probeType = echo, probeTarget = 6.6.6.6, probeTargetPort = 0
probeSource = 2.2.2.2, probeSourcePort = 0, probeNextHop = 21.21.21.1
probeIfIndex = 2, SAA index = 14 probeToS = 0 policy_seq = 0
*Mar  1 01:38:03.131: OER BR APE detail: Completions 2, Sum of rtt 77, Max rtt 77, Min rtt 7
R4#debug oer border active-probes detail
*Mar  1 01:26:28.555: OER BR APE detail: Attempting to retrieve Probe Statistics.
probeType = echo, probeTarget = 6.6.6.6, probeTargetPort = 0
probeSource = Default, probeSourcePort = 0, probeNextHop = Default
probeIfIndex = 2 probeToS = 0 policy_seq = 0
*Mar  1 01:26:28.555: OER BR APE detail: Completed retrieving Probe Statistics.
probeType = echo, probeTarget = 6.6.6.6, probeTargetPort = 0
probeSource = 4.4.4.4, probeSourcePort = 0, probeNextHop = 42.42.42.2
probeIfIndex = 2, SAA index = 12 probeToS = 0 policy_seq = 0
*Mar  1 01:26:28.555: OER BR APE detail: Completions 2, Sum of rtt 9, Max rtt 8, Min rtt 1

R2#
*Mar  1 01:27:14.167: OER BR APE detail: Attempting to retrieve Probe Statistics.
probeType = echo, probeTarget = 7.7.7.7, probeTargetPort = 0
probeSource = Default, probeSourcePort = 0, probeNextHop = Default
probeIfIndex = 2 probeToS = 0 policy_seq = 0
*Mar  1 01:27:14.167: OER BR APE detail: Completed retrieving Probe Statistics.
probeType = echo, probeTarget = 7.7.7.7, probeTargetPort = 0
probeSource = 2.2.2.2, probeSourcePort = 0, probeNextHop = 21.21.21.1
probeIfIndex = 2, SAA index = 13 probeToS = 0 policy_seq = 0
*Mar  1 01:27:14.171: OER BR APE detail: Completions 2, Sum of rtt 88, Max rtt 88, Min rtt 8
R4#
*Mar  1 01:27:14.727: OER BR APE detail: Attempting to retrieve Probe Statistics.
probeType = echo, probeTarget = 7.7.7.7, probeTargetPort = 0
probeSource = Default, probeSourcePort = 0, probeNextHop = Default
probeIfIndex = 2 probeToS = 0 policy_seq = 0
*Mar  1 01:27:14.727: OER BR APE detail: Completed retrieving Probe Statistics.
probeType = echo, probeTarget = 7.7.7.7, probeTargetPort = 0
probeSource = 4.4.4.4, probeSourcePort = 0, probeNextHop = 42.42.42.2
probeIfIndex = 2, SAA index = 13 probeToS = 0 policy_seq = 0
*Mar  1 01:27:14.727: OER BR APE detail: Completions 2, Sum of rtt 8, Max rtt 4, Min rtt 4

The RTT of R4 shown above from the debug output is less than R2 for those two networks so R4 has chosed best path now,we can see following messages on R1 (MC) also to confirm the same.

R1#
*Mar  1 01:27:17.499: %OER_MC-5-NOTICE: Route changed Prefix 7.7.7.7/32, BR 4.4.4.4, i/f Se0/0, Reason Range, OOP Reason Timer Expired
R1#
*Mar  1 01:27:17.499: %OER_MC-5-NOTICE: Route changed Prefix 6.6.6.6/32, BR 4.4.4.4, i/f Se0/0, Reason Range, OOP Reason Timer Expired

Also on R4 .

R4#sh oer border passive cache prefix

OER Passive Prefix Cache, State: enabled, 278544 bytes
2 active, 4094 inactive, 41 added
2320 ager polls, 0 flow alloc failures
Active flows timeout in 1 minutes
Inactive flows timeout in 15 seconds
IP Sub Flow Cache, 25800 bytes
6 active, 1018 inactive, 123 added, 41 added to flow
0 alloc failures, 0 force free
1 chunk, 1 chunk added

Prefix              NextHop         Src If         Dst If          Flows
Pkts   B/Pk  Active   sDly   #Dly  PktLos  #UnRch
————————————————————————
6.6.6.6/32          42.42.42.2      Fa0/0          Se0/0              4
6      80    41.4      0      0       0       0
7.7.7.7/32          42.42.42.2      Fa0/0          Se0/0              5
7      80    48.0      0      0       0       0

The trace from HOST-1 and HOST-2 confirms the same.

HOST-1#trace 6.6.6.6

Type escape sequence to abort.
Tracing the route to 6.6.6.6

1 1.1.124.4 4 msec 4 msec 4 msec
2 42.42.42.2 4 msec *  36 msec

HOST-2#trace 7.7.7.7

Type escape sequence to abort.
Tracing the route to 7.7.7.7

1 1.1.124.4 4 msec 4 msec 4 msec
2  *
42.42.42.2 36 msec 8 msec

So that was OER , we will be discussing more OER/Pfr scenarios in upcoming post.

Latest posts by Afroz Ahmad (see all)

One comment

Leave a Reply

Your email address will not be published. Required fields are marked *