Category Archives: Networking

Add offline devices to Observium

You might, like me, want to add devices to Observium before they are online, perhaps during daytime hours whilst you are writing configurations for those devices, rather than having to have to have a separate process to come back and add them to Observium later when the devices are online.

To that end, I wrote two scripts.

These scripts work for me. They are uploaded here in the hope they will be useful to others, but you have to understand what it is that they do and be satisfied that they do not pose a danger to you or your Observium installation. No warranty is expressed or implied as to their suitability for your needs.

Place both in the root of the Observium install.

add_offline_device.php – stealing most of the original add_device and instead of doing the normal add routine, instead using Observium’s dbInsert function to insert mostly dummy values that would normally be queried using SNMP when adding. As soon as the device is online, the dummy values will be replaced by the first discovery run.

#!/usr/bin/env php
<?php
/**
 *   This file is NOT part of Observium.
 *
 *   This file is Phil's fault - https://interpip.es/networking/monitoring/add-offline-devices-to-observium/
 *
 *   Use it to add a device "offline"
 *   Useful if you are preparing configuration for devices ahead of deployment
 *   and want them to be added to Observium at that time so that they automatically
 *   start getting monitored when they are plugged in.
 *
 *   NB that by adding offline, there is no way to tell if the hostname and/or
 *   SNMP details you are setting are valid. If you feed this script garbage
 *   that cannot work, then of course it will enter garbage into the database
 *   that will never work.
 *
 *   Devices are added 'ignored' so they won't show as down in your Observium UI
 *
 *   Cron unignore_now_online_devices.php to ensure that devices are
 *   unignored once they are successfully discovered.
 *
 *   This file was written by basically butchering add_device.php from
 *   Observium Feb 2026, (c) Adam Armstrong
 *
 *   It works for me, but no warranty expressed or implied: if it eats your
 *   turnips and sets fire to your hair, then that is at your own risk.
 *
 */

/// FIXME. This is mostly DERP arguments parsing

chdir(dirname($argv[0]));

$options = getopt("dhpt", [], $opt_index);

include("includes/observium.inc.php");
include("includes/discovery/functions.inc.php");

print_message("%R".OBS_SCRIPT_NAME."%g for " . OBSERVIUM_PRODUCT . " made by Phil\n%WAdd Offline Device(s) and %ymaybe%W not explode\n%yUse at own risk!%n\n", 'color');

if (OBS_DEBUG) {
    print_versions();
}

if (isset($options['h'])) {
    print_help(OBS_SCRIPT_NAME);
    exit;
}

$snmp_options = [];
// Remove options and script name from argv
$argv = array_slice($argv, $opt_index);

$added     = 0;
$add_array = [];
if (!empty($argv[0])) {
    if (is_file($argv[0])) {
        // Parse file into an array with devices to add
        foreach (new SplFileObject($argv[0]) as $line) {
            $d = preg_split('/\s/', $line, -1, PREG_SPLIT_NO_EMPTY);
            if (empty($d) || str_starts_with($d[0], '#')) {
                // Skip empty lines or commented
                continue;
            }
            $add_array[] = $d;
        }
    } else {
        $add_array[] = $argv;
    }

    // Save base SNMP v3 credentials and v2c/v1 community
    $snmp_config_v3        = $config['snmp']['v3'];
    $snmp_config_community = $config['snmp']['community'];

    foreach ($add_array as $add) {
        $snmp = get_device_snmp_argv($add, $snmp_options);
        if (!$snmp) {
            //print_error("Try to add $hostname:");
            continue;
        }
        $hostname       = $snmp['hostname'];
        $snmp_version   = $snmp['snmp_version'];
        $snmp_transport = $snmp['snmp_transport'];
        $snmp_port      = $snmp['snmp_port'];

        // FIXME. Still used hard set v2c/v3 auth by config
        if ($snmp_version === "v3") {
            // v3
            $config['snmp']['v3'] = $snmp['snmp_v3_auth'];
        } elseif (!empty($snmp_version)) {
            // v1 or v2c
            $config['snmp']['community'] = $snmp['snmp_community'];
        }

        print_message("Try to add $hostname:");

        // (Phil) Construct $device object to direct insert using Observium's dbInsert function
        //
        // FIXME: Defaults for timeout, retries and maxrep set as the Web UI indicated at the time
        // I'm sure there is an Observium official way to get the default/configured values for
        // these fields but I don't have time for figuring that out.
        //
        // Fields that would be queried on add are instead set to TBC
        // Device is inserted "ignored" to prevent it showing as down (it's supposed to be down)
        //
        // The "purpose" (Description in the Web UI) is set to "Added Offline" and this should
        // not be changed after adding the device before it comes online if you want the
        // unignore_now_online_devices.php script to work.

        $device = $snmp;
        $device['snmp_timeout'] = 1;
        $device['snmp_retries'] = 5;
        $device['snmp_maxrep'] = 10;
        $device['os'] = "generic";
        $device['sysObjectID'] = "TBC";
        $device['snmpEngineID'] = "TBC";
        $device['sysName'] = "TBC";
        $device['location'] = "TBC";
        $device['sysContact'] = "TBC";
        $device['snmp_community'] = $device['snmp_community'][0];
        $device['ignore'] = 1;
        $device['purpose'] = "Added Offline";

        if ($device_id = dbInsert($device, 'devices')) {
                $device = device_by_id_cache($device_id);
                print_success("Added offline device " . $device['hostname'] . " (" . $device_id . ").");
                $added++;
        }

        // Restore base SNMP v1/2c/3 credentials (need for add multiple devices)
        $config['snmp']['community'] = $snmp_config_community;
        $config['snmp']['v3']        = $snmp_config_v3;
    }
}

$count  = safe_count($add_array);
$failed = $count - $added;
if ($added) {
    print_message("\nDevices success: $added.");
    if ($failed) {
        print_message("Devices failed: $failed.");
    }
} else {
    if ($count) {
        print_message("Devices failed: $failed.");
    }
    print_help(OBS_SCRIPT_NAME);
}

function print_help($scriptname) {
    global $config;

    $snmp_version = get_versions('snmp');
    if (version_compare($snmp_version, '5.8', '<')) {
        $snmpv3_auth   = '[md5|sha]';
        $snmpv3_crypto = '[des|aes]';
    } else {
        $snmpv3_auth   = '[md5|sha|sha-224|sha-256|sha-384|sha-512]';
        $snmpv3_crypto = '[des|aes|aes-192|aes-192-c|aes-256|aes-256-c]';
    }
    $snmp_transports = '[' . implode("|", $config['snmp']['transports']) . ']';
    print_message("%n
USAGE:
$scriptname <hostname> [community] [v1|v2c] [port] $snmp_transports
$scriptname <hostname> [any|nanp|anp|ap] [v3] [user] [password] [enckey] $snmpv3_auth $snmpv3_crypto [port] $snmp_transports
$scriptname <filename>

EXAMPLE:
%WSNMPv1/2c%n:                    $scriptname <%Whostname%n> [community] [v1|v2c] [port] $snmp_transports [context]
%WSNMPv3%n   :         Defaults : $scriptname <%Whostname%n> any v3 [user] [port] $snmp_transports [context]
           No Auth, No Priv : $scriptname <%Whostname%n> nanp v3 [user] [port] $snmp_transports [context]
              Auth, No Priv : $scriptname <%Whostname%n> anp v3 <user> <password> $snmpv3_auth [port] $snmp_transports [context]
              Auth,    Priv : $scriptname <%Whostname%n> ap v3 <user> <password> <enckey> $snmpv3_auth $snmpv3_crypto [port] $snmp_transports [context]
%WFILE%n     :                    $scriptname <%Wfilename%n>

ADD FROM FILE:
 To add multiple devices, create a file in which each line contains one device with or without options.
 Format for device options, the same as specified in USAGE.

OPTIONS:

DEBUGGING OPTIONS:
 -d                                          Enable debugging output.
 -dd                                         More verbose debugging output.", 'color', FALSE);
}

// EOF

Then, unignore_devices_added_offline.php, intended to be run in cron to update devices added by the first script to be “not ignored” and clear the purpose string when the discovery has run for the first time and the snmpEngineID for the device is no longer “TBC”.

I run this every 15 minutes but I’m not the boss of you, so choose whatever frequency makes sense for your environment.

#!/usr/bin/env php
<?php

/* unignore_now_online_devices.php
 *
 *   This file is NOT part of Observium.
 *
 *   This file is Phil's fault - https://interpip.es/networking/monitoring/add-offline-devices-to-observium/
 *
 *   This script is to be used with an observium install which has been
 *   used with Phil's add_offline_device.php script to add offline
 *   devices in "ignore" state with the "purpose" ("Description" in
 *   the web UI) of "Added Offline".
 *
 *   For any device with the purpose of "Added Offline" but where the
 *   snmpEngineID is no longer "TBC" (indicating it was successfully polled),
 *   we set ignored=0 and clear the purpose string.
 *
 *   It works for me, but no warranty expressed or implied: if it eats your
 *   turnips and sets fire to your hair, then that is at your own risk.
 *
 * FIXME: Fairly trivial to instead make the cron job search for purposes that /start/
 * with Added Offline and just strip that from the purpose to allow users to add
 * whatever description they want in the meantime in the UI. I don't use the purpose
 * field myself, so it's not a priority for me.
 */

require("config.php");

$db = new mysqli($config['db_host'], $config['db_user'], $config['db_pass'], $config['db_name']);

if ($db->connect_errno)
{
        print "Failed to connect to MySQL: " . $db->connect_error;
        exit();
}
else
{
        if ($db->query("UPDATE `devices` SET `ignore` = 0, `purpose` = '' WHERE `purpose` LIKE 'Added Offline' AND `snmpEngineID` NOT LIKE 'TBC'") === TRUE)
        {
                print "Updated " . $db->affected_rows . " devices.\n";
        }
        else
        {
                print "Something went wrong with the mysql query " . $db->error . "\n";
        }
        $db->close();
}

// EOF

pfSense: Suricata Sync Causes XMLRPC Failures and carp backup events

Observed with pfSense 2.4.5p1 and Suricata 5.0.3 (and presumably older versions of both)

Once you enable Suricata config sync, any configuration changes take *ages* to save because Syncs basically start failing to complete – eventually falling through to timeouts.

You might start to see synchronise errors like this on the master (which will get flagged up as notifications):

/rc.filter_synchronize: A communications error occurred while attempting to call XMLRPC method host_firmware_version:

and/or

/suricata/suricata_logs_mgmt.php: A communications error occurred while attempting to call XMLRPC method exec_php:

and/or

/rc.filter_synchronize: New alert found: A communications error occurred while attempting to call XMLRPC method restore_config_section:

You might also notice that CARP is – to put it mildly – freaking out:

Carp backup event
Carp backup event
Carp backup event

And that OVPN / other packages likewise are having problems, stopping/starting/restarting because it thinks the WAN IP has changed as the CARP state flaps back and forth, with the system logging stuff like:

/rc.newwanip: rc.newwanip: Info: starting on ovpns1.
/rc.newwanip: rc.newwanip: on (IP address: ) (interface: []) (real interface: ovpns1).
/rc.newwanip: rc.newwanip called with empty interface.
/rc.newwanip: pfSense package system has detected an IP change or dynamic WAN reconnection - -> - Restarting packages.

The issue is likely that you have promiscuous mode enabled on your Suricata interfaces (because it is the *default* to enable it).

The kernel disabling and enabling promiscuous mode off and on as Suricata reloads during sync causes carnage with the sync TCP connection, CARP, and in turn, everything else.

Promiscuous mode should not be required if you are using Suricata in-line at layer 3 (i.e. on the firewall which is hosting your default gateway which is probably why CARP is running to begin with).

Simply disable promiscuous mode (at the very *least* on any interfaces you’re running CARP on which is probably all of them in an HA setup) and you’ll find things behave much better, and config syncs complete nice and fast again.

Configuring TACACS+ authentication and accounting on IOS 15

Just the bare minimum:

! you probably have this already, if you don't; you should read up on it first
aaa new-model

! use local users, and then all tacacs+ servers, to authenticate logins 
aaa authentication login default local group tacacs+ 

! give enable to tacacs+ users 
aaa authentication enable default group tacacs+ 

! send accounting records for when logins ('exec mode') begin and end 
aaa accounting exec default start-stop group tacacs+
 
! send accounting records for config commands 
aaa accounting commands 15 default stop-only group tacacs+ 

! send accounting records for outgoing connections made to other systems 
aaa accounting connection default start-stop group tacacs+ 

! send system event account records (reloads etc) 
aaa accounting system default start-stop group tacacs+ 

! OPTIONAL: On a router with multiple interfaces that could be chosen to
! reach the TACACS server it is best to specify one; we use Loopback addresses
! for iBGP peering, so it makes sense to use them here too
ip tacacs source-interface Loopback0 

! define at least one tacacs server with some friendly $SERVERNAME 
tacacs server $SERVERNAME
   ! Set the TACACS+ server's ipv4 $ADDRESS (or ipv6, adjust accordingly)
   address ipv4 $ADDRESS
   ! Set the encryption $KEY to match the key configured on the TACACS+ server for this device
   key $KEY
!

Now: BEFORE you log off, try to log in again and make sure you can still log in with your original local credentials.

If you can no longer login after making the above changes, you’ll need to fix that first before you disconnect to prevent you locking yourself out.