Welcome, Guest. Please login or register.

Author Topic: Nedi 1.4 Parrallel Discoverys and Database Lock issues  (Read 1759 times)

ruehlb

  • Newbie
  • *
  • Posts: 16
    • View Profile
Nedi 1.4 Parrallel Discoverys and Database Lock issues
« on: March 18, 2015, 04:45:20 PM »
When using Nedi 1.0.8 and 1.0.9, I wrote my own start script that took the list of devices I have in the database, split them into 8 different seed files then ran 8 different simultaneous discoveries which dropped my discovery time from ~120 minutes to ~20 minutes. I have stood up a new 1.4 instance of Nedi and when I try to run the same script, I end up with two different database errors. Here are those two errors:

DBD::mysql::st execute failed: Deadlock found when trying to get lock; try restarting transaction at /var/nedi/inc/libdb.pm line 1541.
DBD::mysql::db do failed: Deadlock found when trying to get lock; try restarting transaction at /var/nedi/inc/libdb.pm line 1874.

When these errors occur, that specific discovery stops and the remaining devices do not get discovered during that time period.

I have tried to offset the discoveries so to limit the number of discoveries running at a time but it appears that if any two are running it could still cause issues.

This is the script that creates the 8 different seed files:
Code: [Select]
#!/bin/bash
count=1

while [ $count -le 8 ]
do
echo $count
mysql --user=<dbuser> --password=<dbpass> -D nedi -s -e 'select inet_ntoa(devip) from devices' | sort | xargs -n 8 | awk {'print $'$count''} > /var/nedi/seedlist$count
count=`expr $count + 1`
done

I run this once an hour which calls the script to create the seeds and starts each discovery:
Code: [Select]
#!/bin/bash

sh /var/nedi/createseed.sh

/var/nedi/nedi.pl -u /var/nedi/seedlist1 > /var/log/nedi/lastrun-seed1.log &
/var/nedi/nedi.pl -u /var/nedi/seedlist2 > /var/log/nedi/lastrun-seed2.log &
/var/nedi/nedi.pl -u /var/nedi/seedlist3 > /var/log/nedi/lastrun-seed3.log &
/var/nedi/nedi.pl -u /var/nedi/seedlist4 > /var/log/nedi/lastrun-seed4.log &
/var/nedi/nedi.pl -u /var/nedi/seedlist5 > /var/log/nedi/lastrun-seed5.log &
/var/nedi/nedi.pl -u /var/nedi/seedlist6 > /var/log/nedi/lastrun-seed6.log &
/var/nedi/nedi.pl -u /var/nedi/seedlist7 > /var/log/nedi/lastrun-seed7.log &
/var/nedi/nedi.pl -u /var/nedi/seedlist8 > /var/log/nedi/lastrun-seed8.log &

wait

As stated, this worked fine in 1.0.8 and 1.0.9. I would like to have some sort of parallel discovery take place because if not I can't get a full discovery of our network completed in 1 hour.

If there is a better way to do parallel discoveries, I am all ears as well. This is just what I have used in the past.

Thanks!

rickli

  • Administrator
  • Hero Member
  • *****
  • Posts: 2785
    • View Profile
    • NeDi
Re: Nedi 1.4 Parrallel Discoverys and Database Lock issues
« Reply #1 on: March 23, 2015, 10:44:42 PM »
It seems the link handling was a step back for parallel discoveries. I've changed it in the current 1.5 development and it seems to work so far...
Please consider Other-Invoices on your NeDi installation for an annual contribution, tx!
-Remo