Small fixes and hacks for NeDi 1.8.100p3

[sergrok]

I have been using NeDi for a long time and it helps me a lot. There are about 500 SNMP devices in our network and for the best operation of NeDi in our environment, I made small changes to the code. I hope that it can be useful to someone else.

 

1) By default, MySQL is case-insensitive and PostgreSQL is case-sensitive. We can change "~" to "~*" AND "LIKE" to "ILIKE" in DB queries for case-insensitive search in PostgreSQL
BTW, in my tests between MySQL and PostgreSQL on the same servers, PostgreSQL shows significantly better performance when working with NeDi

Code: [Select]

diff -r ./html/inc/libmisc.php /var/nedi/html/inc/libmisc.php
413c413,414
<               $options = array("~"=>"~","!~"=>"!~","LIKE"=>"like","NOT LIKE"=>"!like",">"=>">","="=>"=","!="=>"!=",">="=>">=","<"=>"<","<="=>"<=","&"=>"and","|"=>"or");
---
>               $options = array("~*"=>"~","!~*"=>"!~","ILIKE"=>"like","NOT ILIKE"=>"!like",">"=>">","="=>"=","!="=>"!=",">="=>">=","<"=>"<","<="=>"<=","&"=>"and","|"=>"or");
695c696,697
<       }elseif( $op[0] and !( preg_match('/~|LIKE$/i',$op[0]) and $st[0] === '') ){                    # process normally unless empty regexp/like in 1
---
>       }elseif( $op[0] and !( preg_match('/~\*|ILIKE$/i',$op[0]) and $st[0] === '') ){                 # process normally unless empty regexp/like in 1
698c700,701
<               if($nco and $op[1] and !( preg_match('/~|LIKE$/i',$op[1]) and $st[1] === '') ){         # subcondition 2 unless empty regexp/like
---
>               if($nco and $op[1] and !( preg_match('/~\*|ILIKE$/i',$op[1]) and $st[1] === '') ){              # subcondition 2 unless empty regexp/like
708c711,712
<               }elseif($op[2] and !( preg_match('/~|LIKE$/i',$op[2]) and $st[2] === '') ){             # process normally unless empty regexp/like in 3
---
>               }elseif($op[2] and !( preg_match('/~\*|ILIKE$/i',$op[2]) and $st[2] === '') ){          # process normally unless empty regexp/like in 3
711c715,716
<                       if($nco > 3 and $op[2] and !( preg_match('/~|LIKE$/i',$op[3]) and $st[3] === '') ){# subcondition 4 unless empty regexp/like
---
>                       if($nco > 3 and $op[2] and !( preg_match('/~\*|ILIKE$/i',$op[3]) and $st[3] === '') ){# subcondition 4 unless empty regexp/like


2) Most cisco devices update SNMP interface counters every 10 seconds. This is useful for viewing interface traffic in real time.

Code: [Select]

diff -r ./html/inc/rt-popup.php /var/nedi/html/inc/rt-popup.php
13c13
< $r    = isset($_GET['r']) ? $_GET['r'] : 5;
---
> $r    = isset($_GET['r']) ? $_GET['r'] : 10;
52a53
>                       <option value="10" <?= ($r == 10)?"selected":"" ?>>10


3) When using the HSRP, warnings about duplication of the IP address are displayed  (https://forum.nedi.ch/index.php?topic=1459.msg5688). To avoid this, we can get a table of HSRP IP addresses from the cisco device and do not display warnings for it.

Code: [Select]

diff -r ./inc/libsnmp.pm /var/nedi/inc/libsnmp.pm
1924,1925c1924,1925
<       $session->close;
---
> #     $session->close;  #We will close SNMP session later
1961a1962,1989
> #get active HSRP vIP's for device
>               misc::Prt("IFIP:Walking Cisco HSRP table ");
>               my $match_hsrp_ip = 0;
>               $r   = $session->get_table("1.3.6.1.4.1.9.9.106.1.2.1.1.11");
>               $err = $session->error;
>               if($err){
>                       misc::Prt("ERR :Get Cisco HSRP table $err\n");
>               }else{
>                       while( my($key, $val) = each(%{$r}) ) {
>                               misc::Prt("$val, ") if $main::opt{'d'};
>                               $match_hsrp_ip = 1 if($ip eq $val);
>                       }
>               }
>               misc::Prt("\n");
>                                   if($match_hsrp_ip){
>                                       misc::Prt("IFIP:$ip/$main::net{$na}{$ip}{pfx} on HSRP IF $main::net{$na}{$ip}{ifn} is configured on " . join(', ', keys %{$misc::ifip{$ip}}).". Skipping.\n");
>                                   }else{
1967a1998
>                                   }
2000c2040
<
---
>       $session->close;        #we will move close SNMP session here


4) There is a problem in determining the primary IP address of the device. The availability of the device IP address is checked only using ping. But there are situations when the IP address is available for ping, but SNMP is not available for this IP address. For example, if the interface uses ACL or VRF. To avoid this problem, we must additionally check for the availability of SNMP for the IP address.

Code: [Select]

diff -r ./inc/libsnmp.pm /var/nedi/inc/libsnmp.pm
222c222
<                       return;
---
>                       return $na;             #return device name for Identify function
1984a2016,2023
>               $misc::doip{$ip} = $ip;
>               my $temp_opt_value = $main::opt{'t'};           #backup opt{'t'} value
>               $main::opt{'t'} = 'a';                          #ugly hack. Force the "Identify" function not to add a new device to DB
>               my $dv = snmp::Identify($ip,"s");               #Try to snmp connect to device
>               $main::opt{'t'} = $temp_opt_value;              #recover opt{'t'} value
>               delete $misc::doip{$ip};
>               if( $dv !~ /^(|noSuchObject)$/ ){               #If device name not empty and not "noSuchObject"
1987a2027
>                                               }else{  $misc::mq += mon::Event('i',150,'nedj',$na,$na,"$ip on $main::net{$na}{$ip}{ifn} is SNMP unreachable, but chosen by useip policy ($usip)"); }


5) There is a problem when using Cisco DMVPN. A multipoint GRE (mGRE) tunnel interface can have many connections to remote devices. And NeDi checks that the interface can have only one CDP/LLDP neighborhood. We will remove this check for tunnel interfaces.

Code: [Select]

diff -r ./inc/libsnmp.pm /var/nedi/inc/libsnmp.pm
2520c2560
<                               }elsif( $neb{$i}{$n}{'dp'} ne 'UBIQ' and $misc::portprop{$na}{$lif}{lnk} and !$misc::portprop{$na}{$lif}{mcf} ){# Avoid duplicates (several discovery protocols or static links) except if macflood is set or ubiquity APs
---
>                               }elsif( $neb{$i}{$n}{'dp'} ne 'UBIQ' and $misc::portprop{$na}{$lif}{typ} ne '131' and $misc::portprop{$na}{$lif}{lnk} and !$misc::portprop{$na}{$lif}{mcf} ){#Scan for multiple CDP on 131 type IFs (DMVPN tunnels)  #Avoid duplicates (several discovery protocols or static links) except if macflood is set or ubiquity APs


6) Some routers with built-in Wi-Fi, such as the Cisco 881/891, warn of the presence of an CDP loop between the router and the built-in Wi-Fi access point.

Code: [Select]

diff -r ./inc/libsnmp.pm /var/nedi/inc/libsnmp.pm
2557c2597
<                                       if( $id eq $neb{$i}{$n}{'id'} or $na eq  $neb{$i}{$n}{'na'} ){  # Seeing myself?
---
>                                       if(($id eq $neb{$i}{$n}{'id'} or $na eq  $neb{$i}{$n}{'na'}) and ($lif !~ /^wl/ )){     # Seeing myself?   #skip loop detect for wl* interfaces (bug on Cisco 881/891 with integrated AP)


7) It's useful to see which user made changes to the configuration in config diff. Like this:

Code: [Select]

3- ! Last configuration change at 11:52:51 EKT Wed Aug 14 2019 by netops_1
4- ! NVRAM config last updated at 12:54:09 EKT Wed Aug 14 2019 by netops_1
3+ ! Last configuration change at 14:48:52 EKT Fri Dec 20 2019 by netops_3
4+ ! NVRAM config last updated at 14:57:10 EKT Fri Dec 20 2019 by netops_3

But these lines are skipped during configuration backup.

Code: [Select]

diff -r ./inc/libcli.pm /var/nedi/inc/libcli.pm
266c266,267
< $cmd{'IOS'}{'cfst'} = '^version';
---
> $cmd{'IOS'}{'cfst'} = '^Current';
289c290,291
< $cmd{'IOS-old'}{'cfst'} = '^version';
---
> $cmd{'IOS-old'}{'cfst'} = '^Current';
301c303,304
< $cmd{'IOS-rtr'}{'cfst'} = '^version';
---
> $cmd{'IOS-rtr'}{'cfst'} = '^Current';


8 ) Fix for cisco wireless controllers  with non default device name

Code: [Select]

diff -r ./inc/libcli.pm /var/nedi/inc/libcli.pm
317c320,321
< $cmd{'IOS-wlc'}{'ropr'} = '\(Cisco Controller\) >$';
---
> $cmd{'IOS-wlc'}{'ropr'} = '\([\w+().-]+\) >$';


9) Allow to use legacy SSH ciphers

Code: [Select]

diff -r ./inc/libcli.pm /var/nedi/inc/libcli.pm
727c735
<               my $known = "-o 'StrictHostKeyChecking no'";
---
>               my $known = "-o 'StrictHostKeyChecking no' -o KexAlgorithms=+diffie-hellman-group1-sha1 -o HostKeyAlgorithms=+ssh-dss -o Ciphers=+aes256-cbc,3des-cbc";


10) We don’t want the script to crash when errors (out of range) occur with inserting to the "interfaces" table.  TODO: Make checks of each value before inserting into the table.

Code: [Select]

diff -r ./inc/libdb.pm /var/nedi/inc/libdb.pm
1352a1353,1356
>
> local $sth->{RaiseError};
>


11) The default bandwidth for Vlan interfaces is 1 Gb, and for Loopback 8 Gb. But in fact, much more traffic can pass through such interfaces and this causes notifications. We do not want such notifications.

Code: [Select]

diff -r ./inc/libmisc.pm /var/nedi/inc/libmisc.pm
614c614
<               if($trfele and $main::int{$dv}{$i}{'spd'} and $skip !~ /t/){                            # Ignore speed 0 and if traffic is skipped
---
>               if($trfele and $main::int{$dv}{$i}{'spd'} and $main::int{$dv}{$i}{typ} != 24 and $main::int{$dv}{$i}{typ} != 53 and $skip !~ /t/){                              # Ignore speed 0 and if traffic is skipped. Ignore Lo 8Gb (type=24) and Vlan 1Gb (type=53) IFs


12) Ignore "!Time" for Cisco Nexus and "# WLC Config" in config

Code: [Select]

diff -r ./inc/libmisc.pm /var/nedi/inc/libmisc.pm
1270c1270
<                               if( $l !~ /\#time:|ntp clock-period/){                                  # Ignore ever changing lines
---
>                               if( $l !~ /\#time:|\!Time:|^# WLC Config|ntp clock-period/){            # Ignore ever changing lines.   + ignore "!Time" for nexus and "# WLC Config"


13) Fix for line break in html emails

Code: [Select]

diff -r ./inc/libmon.pm /var/nedi/inc/libmon.pm
189a190
>       $l =~ s/\n/<br>\n/g;                                                                    #<br>\n instead of \n for html emails. diff config for example
191c192,193
<                                                       $smtp->datasend("$l\n");
---
>                                                       $smtp->datasend("$l<br>\n");
193c195,196
<                                                       $smtp->datasend("$ln) $l\n");
---
>                                                       $smtp->datasend("$ln) $l<br>\n");



Let me take this opportunity to offer a feature request: to collect information about MTU values ​​on deivce interfaces. It was also useful to display MTU information on a topology map.
Thank you.

[ascii]

i have 1 more
the bootimage might be longer than 64 chars.

and in the Database change the bootimage field size from 64 to 128 in device table

change line 1089 in /var/nedi/inc/libdb.pm
from

Code: [Select]

$main::dev{$dv}{bi}?substr($main::dev{$dv}{bi},0,64):'',to

Code: [Select]

$main::dev{$dv}{bi}?substr($main::dev{$dv}{bi},0,128):'',
the error accored  for me on Cisco IE-4000 switch

Bootimage changed from sdflash:/ie4000-universalk9-mz.152-7.E/ie4000-universalk9-mz.152 to sdflash:/ie4000-universalk9-mz.152-7.E/ie4000-universalk9-mz.152-7.E.bin

it was always cut to 64 now it's fine

[rickli]

sergok: Many thanks for sharing this. I'll look into it for 2.0 and get back to you...

ascii: Thanks as well, but I found that usually the path should be cut off from the bootimage. I don't want to deal with such long strings ;-)
        Looks like that sdcard: prevented it, so I've included it in the match function accordingly...

[ascii]

ok

i changed the line 484 in /var/nedi/inc/libsnmp.pm
from

Code: [Select]

484                         $bimg =~ s/^(\.|flash:|(slot|disk)\d:|(sup-)?boot(flash|disk):|FIRMWARE REVISION: )([-.\/\w]*\/)?//;
to

Code: [Select]

484                         $bimg =~ s/^(\.|(sd)?flash:|(slot|disk)\d:|(sup-)?boot(flash|disk):|FIRMWARE REVISION: )([-.\/\w]*\/)?//;

[rickli]

Exactly what I did :-D

[rickli]

Ok, I went through the list. Thanks again sergok!

1)   Done but in AdOpVal()
2)   Done
3)   Valid point and duly noted. If implemented, it should cover other vendors too (VRRP)
4)   SNMP check everytime is too "expensive" for me. It's duly noted though :-)
5)   Can be enabled in Devices-Interface now by setting Topology -> Multiple Neighbor on desired interfaces
6)   Done
7)   Done
   Has been fixed in 1.9
9)   I recommend adapting /etc/ssh/ssh_config instead
10)  I keep adding checks as they occur to avoid slowing down to code unnecessarily. On my machine, it doesn't die, even if a string is too long, so there must be a DB setting for this.
12)  Did not see "WLC config" lines, but "!Time:" is ignored now
13)  Mail formatting heavily depends on the mailservers, thus this can be configured in nedi.conf now
MTU) Noted for v2.1 (will check with my customers first)

[sergrok]

Hello

I am glad that some of these changes have become useful.
4) We can make SNMP verification optional in nedi.conf
11) How about this item in the list? )
13) In this case, I used the Microsoft Exchange server

Thank you rickli

[rickli]

Oh forgot 11 

Go to Devices-Interfaces and filter on type 53 and/or 24. Then enter traffic threshold 100 and click Update.

In NeDi 2.0 the thresholds will be set automatically for new devices 

[sergrok]

Hello, @rickli . I made one more little fix.

14) When we do a routing table discovery, we only check if the border has been reached, but not netfilter. Because of this, in some cases we get a lot of useless duplicated "TODO:" log entries. Especially if we reach a router that stores BGP fullview table.

Code: [Select]

diff -r ./inc/libsnmp.pm /var/nedi/inc/libsnmp.pm
2908,2909d3221
<                               }elsif($nh !~ /$misc::netfilter/){
<                                      misc::Prt("ROUT:$nh doesn't match netfilter /$misc::netfilter/\n") if $main::opt{'d'};

[rickli]

Good point, I'll add that! The alternative is to use border instead netfilter...