Welcome, Guest. Please login or register.

Author Topic: Fortigate backup config Stale  (Read 3294 times)

dcecchetto

  • Guest
Fortigate backup config Stale
« on: April 17, 2014, 02:10:46 PM »
Hi to all,

the problem appears when you have too much difference configuration and a file is too long, in the case of fortigate will easily exceed the 4000 rows.
I think I've solved the problem, with the change of CfgChanges on libmisc.pm and added a split function.
Please test if working.

sub CfgChanges{

   use Algorithm::Diff qw(diff);

   my $chg = '';
   my $row = 1000;
   my $accts_split_a = SplitArray(0,$row,@{$_[0]});
   my $accts_split_b = SplitArray(0,$row,@{$_[1]});
   my $i = 0;
   foreach (@$accts_split_a){
      my $diffs = diff(@$accts_split_a[$i], @$accts_split_b[$i]);
      return '' unless @$diffs;
      
      foreach my $chunk (@$diffs) {
         foreach my $line (@$chunk) {
            my ($sign, $lineno, $l) = @$line;
            if( $l !~ /\#time:|ntp clock-period/){
               $chg .= sprintf "%4d$sign %s\n", $lineno+1+($row*$i), $l;
            }
         }
      }
      $i++
   }
   return $chg;
}

=head2 FUNCTION SplitArray()

Split Array in more SubArray.

B<Options> pointer to use large arrays

B<Globals> -

B<Returns> splitted array

=cut
sub SplitArray {
   my ($start, $length, @array) = @_;
   my @array_split;
   my $count =  @array / $length;
    for (my $i=0; $i <= $count; $i++)
{
    my $end = ($i == 9) ? $#array : $start + $length - 1;
    @{$array_split[$i]} = grep defined,@array[$start .. $end];
    $start += $length;
}
    return \@array_split;
}

Davide

rickli

  • Administrator
  • Hero Member
  • *****
  • Posts: 2765
    • View Profile
    • NeDi
Re: Fortigate backup config Stale
« Reply #1 on: April 17, 2014, 10:17:39 PM »
Indeed I've come acrosse this problem as well. The first time works and then the diff fails. Have you tested if it works, when changes occur at the split? But I guess it makes sense to include your fix either way...tx!
Please consider Other-Invoices on your NeDi installation for an annual contribution, tx!
-Remo

dcecchetto

  • Guest
Re: Fortigate backup config Stale
« Reply #2 on: April 22, 2014, 09:23:49 AM »
The first time work because nedi read only the config and from the second make the diff.
This diff creates two arrays as long configuration lines present( in my case about 3000 row),
from the tests I've done seems to work.

Nedi is on github?

rickli

  • Administrator
  • Hero Member
  • *****
  • Posts: 2765
    • View Profile
    • NeDi
Re: Fortigate backup config Stale
« Reply #3 on: May 29, 2014, 01:53:05 AM »
I've added your contribution. Please test in upcoming beta..and many thanks!
Please consider Other-Invoices on your NeDi installation for an annual contribution, tx!
-Remo

SebastianB

  • Guest
Re: Fortigate backup config Stale
« Reply #4 on: July 08, 2015, 01:37:04 PM »
Hello,

sorry for reviving an old threat, but I ran into some issues with stale backups in 1.4.300 as well. Interestingly it only affects larger backups with multiple thousand lines and only when minor changes to the configuration of the device have been made.

I did a
Code: [Select]
./nedi.pl -a 1.2.3.4 -v -b and saw that NeDi was able to login to the device using SSH and to retrieve the configuration. However no changes where detected and therefore no new version was written to the database. When I cleared (deleted) the device's configuration using the web GUI, the device's configuration was successfully saved in the database during the next execution of nedi.pl.

I did some troubleshooting why this may happen and may have found the reason. I am by no means a Perl wizzard, so take my results with a grain of salt :)

When NeDi fetches a a large config from a device it is split into multiple 1000 line chunks. Each chunk is than compared to the corresponding chunk of the previous stored version in the database using the Diff function in libmisc.pm.

Now in my case only one of those chunks contains a change, all other chunks are identical with the previous versions. When the Diff function in libmisc.pm compares two identical blocks it prematurely exists the function returning '' (line 1033). This happens although there may have been changes in the previous chunks or in following chunks. The empty return code then makes the discovery process believe that no changes were made to the device's configuration and therefore storing the information in the database is skipped.

In our environment I was able to remedy the issue by changing the line 1033 from
Code: [Select]
return '' unless (at)$diffs;

to:

Code: [Select]
if(!(at)$diffs){
   $i++;
   next;
}

The forum does not allow me to use the (at)-sign, so please replace the (at) with the proper character (ASCII Code 64).

Maybe this helps someone else who has the same issue.

Regards,
Sebastian

rickli

  • Administrator
  • Hero Member
  • *****
  • Posts: 2765
    • View Profile
    • NeDi
Re: Fortigate backup config Stale
« Reply #5 on: July 08, 2015, 03:23:51 PM »
Excellent analysis and feedback, tx!

I cannot verify this at the moment as I don't have access to a Fortigate. Nevertheless I'll include your fix in the upcoming patch4 and of course NeDi 1.5...
Please consider Other-Invoices on your NeDi installation for an annual contribution, tx!
-Remo

SebastianB

  • Guest
Re: Fortigate backup config Stale
« Reply #6 on: July 09, 2015, 04:11:16 PM »
Glad I could help (at least a little bit)  :)

Just for your information, in my case the issue occurred when backing up a large config (8000+ lines) from a Cisco ASA. So the issue is probably not limited to Fortigate devices but may affect larger configs from all devices/vendors that are compared using the Diff function.

Looking forward to patch 4....

Regards,
Sebastian