Explore directory renaming in Splunk Enterprise 9.0 Indexer Clustering

With the introduction of Splunk Enterprise 9.0, Splunk has changed the language used for some directories in indexer-clustering. Let’s explore how this works in practice.

A bit of history and background

In versions of Splunk prior to 9.0, configurations that would be sent to indexers would be stored in the $SPLUNK_HOME/etc/master-apps directory on the cluster manager. Configurations pushed to pre-9.0 indexers would end up in the $SPLUNK_HOME/etc/slave-apps directory. This changes with Splunk Enterprise version 9.0.

There is now two possible locations to store applications on a cluster-manager: master-apps and manager-apps. The manager-apps location is the preferred location for new installations, and the master-apps directory is retained for backward compatibility. Only one of these slots can be used at a time.

On Splunk Enterprise 9.0 indexers, applications received from a cluster manager will end up in the $SPLUNK_HOME/etc/peer-apps directory. This change is automatic and only handled by the cluster manager, so you shouldn’t need to take anything into account when upgrading to Splunk 9.0.

However, since choosing where to place apps on the CM is something that is controlled by the user, I had to find out: what happens if someone tries to push the configuration to from the wrong location?

Push bad config

An immediate question I had when reading about the config change: what if someone accidentally puts a new app in the wrong directory and tries to push a bundle to the indexers?

The good news: Splunk thought someone would mess this up, and there’s error checking in place to prevent a push from happening if there are apps in the manager-apps and master-apps directories at the same time. time :

Copy to clipboard

Location of populated manager apps found, but main apps are also populated. There can only be one. This needs to be fixed before continuing
WARNING: Server certificate hostname validation is disabled. Please see server.conf/[sslConfig]/cliVerifyServerName for details.

Some errors encountered while applying the bundle.
both management apps and core apps are populated. There can only be one. Bundle push blocked until all bundles are either in management apps (preferred) or core apps.

To solve this problem, the administrator simply needs to ensure that there is only one directory (master-apps or manager-apps) containing the configuration files. If this happens in your environment, you’ll hopefully catch it on the first cluster bundle push after the error occurs.

Below is a demo showing how it works in practice.

Migration of master applications to management applications

To migrate from the old master-apps directory to the new manager-apps directory on your cluster manager, simply move all applications to the new manager-apps directory and push a bundle to the indexers as you normally would. Your cluster manager will now use the new path for applications. Also, be sure to update your documentation to point to the new location, and make sure that any mechanisms outside of Splunk that would write to the old manager-apps directory (such as configuration management tools) also point to the new location.

Below is a demo showing the migration process.


Overall, this is a relatively minor change to Splunk’s overall admin workflow. If you’re used to the pre-9.0 way, this will be a change you’ll have to get used to, but you don’t have to worry about the process accidentally breaking your Splunk environment.