Splunk® Enterprise

Release Notes

Splunk Enterprise version 8.1 will no longer be supported as of April 19, 2023. See the Splunk Software Support Policy for details. For information about upgrading to a supported version, see How to upgrade Splunk Enterprise.
This documentation does not apply to the most recent version of Splunk® Enterprise. For documentation on the most recent version, go to the latest release.

Known issues

The following are issues and workarounds for this version of Splunk Enterprise.

Issues are listed in all relevant sections. Some issues appear more than once. To check for additional security issues related to this release, visit the Splunk Security Portal.

Refer to System requirements in the Installation Manual for a list of supported platforms and architectures.

For a list of deprecated features and platforms, refer to Deprecated features and removed features in this manual.

Upgrade issues

Date filed Issue number Description
2020-12-05 SPL-198311 KVstore failed to start after offline upgrade of SHC from Splunk Enterprise 7.x to 8.x

Workaround:
1) Use Searchable Rolling Upgrade rather than Offline upgrade for SHC nodes migration/upgrade.

2) If this issue is encountered after offline upgrade, please run the following steps on problematic SH nodes. - Stop the search head that has the stale KV store member - Run the command splunk clean kvstore --cluster - Restart the search head. This triggers the initial synchronization with other KV store members. - Run the command splunk show kvstore-status to verify synchronization has succeeded.

2020-11-30 SPL-198052, SPL-206073, SPL-206074, SPL-206076 Upgrading From 8.0.6 to 8.1.0.1 Using a DEB package results in a No such file or directory message.
2020-11-09 SPL-197140, SPL-234386 UF failed to start on Solaris 11.3 with error: "symbol in6addr_any: referenced symbol not found"

Workaround:
1. Do not upgrade past Splunk 8.0.5 on Solaris 11.3

OR 2. Upgrade to Solaris 11.4

2020-08-31 SPL-194426 External search command chunked v2 python SDK fails with multibyte result data under python 3.

Workaround:
Apps may experience this issue if they: implement a custom search command using the Splunk Enterprise SDK for Python between versions 1.6.5 and 1.6.13; are executed by Splunk Enterprise or Splunk Cloud using Python 3; and are sent events with multi-byte characters.

App developers whose apps implement a custom search command using a version of the Splunk Enterprise SDK for Python must update to version 1.6.14 or higher and release new versions of their apps.

Splunk Enterprise and Splunk Cloud administrators who are using apps impacted by this issue must update to app versions that use the Splunk Enterprise SDK for Python version 1.6.14 or higher. If this is not possible, administrators are encouraged to either: allow these apps to be executed using Python 2; or cease usage of impacted apps until updated versions are available.

2020-07-10 SPL-191850 The .deb installation package will fail if dpkg version doesn't support an .xz compressed control file.

Workaround:
Update dpkg to version 1.17.6 or later.
2018-04-13 SPL-153403 After running the "clean userdata" command, admin is unable to login with msg "No users exist. Please set up a new user."

Workaround:
Create a $SPLUNK_HOME/etc/system/local/user-seed.conf and restart Splunk

[user_info]
PASSWORD = <yourpassword>


2017-03-20 SPL-139019 Possible compatibility issues between Python / SDK clients and new 6.6 and later default sslVersions, cipherSuites

Workaround:
Users can do either of the following:

1. Overwrite the new Splunk 6.6 server.conf [sslConfig] sslVersions, cipherSuites with your own settings that are compatible with your version of OpenSSL, e.g. the previous defaults from 6.5.x are compatible with OpenSSL 0.9.8 on Mac OSX:

[sslConfig]
sslVersions = *,-ssl2
sslVersionsForClient = *,-ssl2
cipherSuite = TLSv1+HIGH:TLSv1.2+HIGH:@STRENGTH

2. For some more up-to-date clients, it is possible to enforce TLS1.2 (e.g. --tlsv1.2 for curl) in order to connect successfully.

3. Upgrade OpenSSL on your platform and link it with your client (e.g. Python, curl, etc..). For example, OpenSSL 1.0.2 is currently available on Mac OSX via Homebrew (see https://brew.sh) and is compatible with the new Splunk 6.6 default sslVersions, cipherSuites.

2017-03-13 SPL-138647 Possible compatibility issues between new 6.6 and later default sslVersions, cipherSuites and external services, e.g. e-mail, LDAP

Workaround:
If security is not a significant concern, simply revert back to the 6.5.x SSL/TLS defaults, e.g. for e-mail, add to $SPLUNK_HOME/etc/system/local/alert_actions.conf

[email]
sslVersions = *,-ssl2
cipherSuite = TLSv1+HIGH:TLSv1.2+HIGH:@STRENGTH


To configure LDAP with the same settings used by e-mail alerts: $SPLUNK_HOME/etc/openldap/ldap.conf

TLS_PROTOCOL_MIN 3.1
TLS_CIPHER_SUITE TLSv1+HIGH:TLSv1.2+HIGH:@STRENGTH


To completely revert the LDAP configuration to the 6.5.x SSL/TLS defaults, comment out TLS_PROTOCOL_MIN and TLS_CIPHER_SUITE


If you would like to retain the more secure 6.6.x defaults, but prefer to add an exception for your less secure external services, follow the procedure below:

1. To determine what sslVersions and cipherSuites are supported by a server, run splunk cmd openssl s_client -connect hostname:port | awk '/Protocol/ || /Cipher/ || /Verify/'.

The example below is for a Postfix SMTP server:

eserv@indexer01:~$ splunk cmd openssl s_client -connect smtp-server01:465 | awk '/Protocol/ || /Cipher/ || /Verify/'
depth=1 C = US, O = Example Customer, OU = IT, CN = Example Customer IT CA, emailAddress = [email protected] verify error:num=19:self signed certificate in certificate chain New, TLSv1/SSLv3, Cipher is DHE-RSA-AES256-SHA

   Protocol : TLSv1 
   Cipher : DHE-RSA-AES256-SHA 
   Verify return code: 19 (self signed certificate in certificate chain)

2. Check the OpenSSL output for Protocol and Cipher. In the example above, Protocol = TLSv1 and Cipher = DHE-RSA-AES256-SHA

3. Update Splunk's relevant sslVersions and/or cipherSuite. In the example above, sslVersions should be set to tls (allows TLSv1, TLSv1.1, TLSv1.2) and DHE-RSA-AES256-SHA should be appended to the end of the default cipherSuites definition, e.g. add
$SPLUNK_HOME/etc/system/local/alert_actions.conf
:

[email]
sslVersions = tls

cipherSuites = ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA

2014-08-20 SPL-89640 When running Splunk on Linux as non-root user and using RPM to upgrade, the RPM writes $SPLUNK_HOME/var/log/introspection as root, causing errors upon restarts

Workaround:
Chown the $SPLUNK_HOME/var/log/introspection directory to the user Splunk Enterprise runs as after upgrading and before restarting Splunk Enterprise.
2013-08-19 SPL-73386 Users are not allowed to run historical scheduled search

Workaround:
1. Create a special power/admin user who can run scheduled searches.

2. Assign this user ownership of the scheduled searches.

3. Share the searches at the app level and grant read/write permission to the correct set of users.

Data input issues

Date filed Issue number Description
2022-02-23 SPL-219631 Memory Leak at indexer level with IngestEval lookups
2022-01-26 SPL-217790 INGEST_EVAL using numbers adds up to license

Workaround:
Use field casting and define int or string:

INGEST_EVAL = len[int]=len(_raw) or INGEST_EVAL = field_xyz[int]=100

2022-01-07 SPL-217087, SPL-205550 PDF generation process stuck and consuming CPU
2021-07-29 SPL-209519, SPL-213290, SPL-213416 dedicatedIOthreads is not respected, causing HEC performance problems

Workaround:
Scale HEC by adding HFs/indexers instead of tuning dedicatedIOthreads

Or stay on 7.3.x/8.0.x until you can upgrade to version with fix

2020-09-29 SPL-195635, SPL-202178, SPL-206477, SPL-202163, SPL-206534 Splunkd increased memory usage over time when monitoring UDP port(s) with in inputs.conf
2020-09-15 SPL-195024, SPL-195482, SPL-200120, SPL-202918, SPL-202919, SPL-202917 sometimes a HEC bundle reload can introduce HEC server DOS.

Workaround:
"splunk _internal call /services/data/inputs/http/_reload" on affected indexers

if that doesnt work, do: restart affected indexers

(also a potential workaround, not recommended:) turn off the reload feature for http inputs "reload.inputs.http = never"

2015-11-12 SPL-109362 When the disk runs out of space for the limit set in the server.conf, add data workflow gets stuck with "Uploading file" message modal in the review stage
2015-05-22 SPL-101981 Field extractions do not work when sourcetypes use quotes in the Getting Data In interface.
2015-03-17 SPL-98163 INDEXED_EXTRACTIONS=W3C is truncating field cs_uri_stem when spaces are present in URL

Workaround:
Create a separate extraction in props.conf where defined w3c extraction method:

EXTRACT-cs_uri_stem1 = (GET|POST) (?<cs_uri_stem1>[^-]++)

Search issues

Date filed Issue number Description
2022-08-29 SPL-229278 Search crashes with "StatsBuffer found inconsistent row" after upgrading

Workaround:
Use '| noop feature_flag=stats:allow_stats_v2:false' in a search to use StatsV1 to avoid this issue for a single search. Alternatively, add '[stats] use_stats_v2 = false' in the limits.conf file to globally configure this setting for all searches.

Both workarounds impact the performance for searches using the 'stats' family of functions such as 'chart', 'timechart', 'eventstats', 'tstats', 'prestats', 'mstats' and 'streamstats' because performance enhancements added with StatsV2 are not used. As a result, upgrading to Splunk Enterprise version 9.0.1 or higher is preferred.

2021-12-13 SPL-216365 On Windows, warning messages concerning "larger than MAX_PATH" appear in splunkd.log when attempting to send email using sendemail.py.

Workaround:
Workaround: These warning messages are benign, and can be ignored. "WARN Pathname - Pathname 'C:\Program Files\Splunk\bin\Python3.exe C:\Program Files\Splunk\etc\apps\search\bin\sendemail.py"
2021-08-19 SPL-210674, SPL-211145, SPL-211146 Zero length content in double quotes for NOT search incorrectly excludes fields which contain values.
2021-08-10 SPL-210070, SPL-209599 Searches with hundreds of search commands can crash the main Splunk server.
2021-06-02 SPL-206635, SPL-210969 tstats "fillnull_value" only works for results from tsidx (accelerated DM) but not from unaccelerated results (fallback search)

Workaround:
Use eval (calculated field) in datamodel to fill the null values in the definition itself, with something like this
if(isnull(status), "NULL",status)
2021-05-11 SPL-205542, SPL-205545, SPL-205978, SPL-205975, SPL-205976 SPL command "iplocation" info is out of date

Workaround:
From http://dev.maxmind.com/geoip/geoip2/geolite2/, download the binary gzipped version of the GeoLite2 City database file.

Copy the file to the search head on your Splunk Enterprise instance. Expand the GZ file. Stop any real-time searches that are running. Copy the GeoLite2-City.mmdb file to the $SPLUNK_HOME/share/ directory to overwrite the file there. Restart the real-time searches.

2021-05-07 SPL-205436, SPL-208943 in batch mode, on distributed search, addtotals with wildcard that matches more than 50 fields populates the _time field with other field's value (like the sourcetype, index, splunk_server) for result above 50

Workaround:
Use a transforming command (i.e. |stats) after |addtotals

Add |streamstats before |addtotals to force a cursored search

Disable batch mode, limits.conf on the search head:

[search]
allow_batch_mode = false
2021-05-05 SPL-205362, SPL-206450 No events are returned when data with preceding wildcards(*) is ingested

Workaround:
Switch back to search evaluator v1

$ cat $SPLUNK_HOME/etc/system/local/limits.conf [search] use_search_evaluator_v2=false

2021-04-27 SPL-204889 Round function no longer rounds to the left of the decimal, for example "n=round(155, -1)"
2021-04-26 SPL-204793, SPL-205516, SPL-206367 REST API search fails on 8.1.3, completes on 7.2.6
2021-03-27 SPL-203092, SPL-203190, SPL-204188 Setting server.socket_host in web.conf to 127.0.0.1 on version 8.1.x fail with WARNING: web interface does not seem to be available!

Workaround:
If possibly use the other option ...

https://docs.splunk.com/Documentation/Splunk/8.1.3/Admin/Splunk-launchconf SPLUNK_BINDIP = <ip address>

2020-12-06 SPL-198314, SPL-233681, SPL-233762 Exporting _time field applies user timezone offset but contains the server's timezone (usually +0000)

Workaround:
Force a specific time format by using strftime in an eval command.

for example, add

 | convert timeformat="%FT%T.%3Q%z" ctime(_time)

to the end of your search

2020-12-04 SPL-198284, SPL-231587 Crash in search process in PrecacheUsersThread when max_searches_per_process is set lower than default

Workaround:
Set limits.conf back to default, by removing any override of max_searches_per_process.

For example:

[search]
max_searches_per_process=1

to

[search]
2020-12-01 SPL-198149, SPL-198866, SPL-199358 KVStore lookup indexing leads to slow search performance and intermittent errors in searches

Workaround:
If you encounter this problem, change the enable_splunkd_kv_lookup_indexing parameter to true in the [lookup] stanza of limits.conf in the $SPLUNK_HOME/etc/system/local directory on your search peers.
2020-10-07 SPL-196040, SPL-212495 Excessive logging of 'Unable to parse site_label, label=invalid due to err="Invalid site id: invalid"' for SearchResultsFiles

Workaround:
Increase the minimum logging level for SearchResultsFiles:
#log-local.cfg

[splunkd]
category.SearchResultsFiles=ERROR

https://docs.splunk.com/Documentation/Splunk/latest/Troubleshooting/Enabledebuglogging#log-local.cfg

2020-09-18 SPL-195181, SPL-176333 Lookups may return incorrect results due to internal caching

Workaround:
Add

allow_caching=f to the lookup command:

| lookup <name> allow_caching=f ...

On 7.3+: Add allow_caching=f to the lookup definition on the search head

transforms.conf:
[<lookup name>]
allow_caching = f

To check if you might be running into this issue, you'll need to enable debug on the search in question by adding:

| noop log_DEBUG=CachedProvider
<pre>
If you have hits for the cached lookup, like in the sample log below, you can hit this issue.

<pre>
DEBUG CachedProvider - Cached provider metrics: lookup=<lookup name> hits=67064 misses=321 total=67385
2020-09-01 SPL-194461, SPL-194199 |fieldformat in |foreach statement doesn't work

Workaround:
Either use eval or fieldformat outside of foreach

instead of

... | foreach field [| eval "<<FIELD>>"=... ] 

use something like this

... | fieldformat "field"=... 
2020-08-31 SPL-194426 External search command chunked v2 python SDK fails with multibyte result data under python 3.

Workaround:
Apps may experience this issue if they: implement a custom search command using the Splunk Enterprise SDK for Python between versions 1.6.5 and 1.6.13; are executed by Splunk Enterprise or Splunk Cloud using Python 3; and are sent events with multi-byte characters.

App developers whose apps implement a custom search command using a version of the Splunk Enterprise SDK for Python must update to version 1.6.14 or higher and release new versions of their apps.

Splunk Enterprise and Splunk Cloud administrators who are using apps impacted by this issue must update to app versions that use the Splunk Enterprise SDK for Python version 1.6.14 or higher. If this is not possible, administrators are encouraged to either: allow these apps to be executed using Python 2; or cease usage of impacted apps until updated versions are available.

2020-02-12 SPL-183259 When generating LISPY for field values that are numbers (""), the values aren't deduplicated, which can cause slowdowns in certain scenarios

Workaround:
Dedup values in search before, for example:

instead of

index="field_test" [search index="field_test" globalCallID_callId=1234* | fields globalCallID_callId]

add a stats or dedup in the subsearch:

index="field_test" [search index="field_test" globalCallID_callId=123* | stats values(globalCallID_callId) AS globalCallID_callId | mvexpand globalCallID_callId ]

If that list is still large and you're seeing the slowdown, consider moving the filtering to a | where after the initial search, for example:

index="field_test" globalCallID_callId=* | where [search index="field_test" globalCallID_callId=123* | stats values(globalCallID_callId) AS globalCallID_callId | mvexpand globalCallID_callId ]
2020-01-10 SPL-181573 geostats provides incorrect results for lower zoom levels when split BY has a higher cardinality than globallimit.

Workaround:
- Increase globallimit to the value of "unique values" number mentioned in the warning message:

"The split by field <field> has a large number of unique values <number>. Chart column set will be trimmed to 10. Use globallimit argument to control column count."

- Use very high globallimit in geostats and post process after if needed

- Don't use BY in geostats

- Use lower cardinality BY and/or higher globallimit in geostats

2019-02-05 SPL-166001 16MB+ events are not displayed on the search results, but they will be listed on the fields sidebar and in the timeline. search.log message: "SRSSerializer - max str len exceeded - probably corrupt"

Workaround:
Make sure fields are under 16777216 characters (or 16MB, usually _raw is the biggest)

OR

Revert back to the old serialization format (CSV), however, this applies to all searches, so you won't be getting the (performance) benefits of the new format.

$SPLUNK_HOME/etc/system/local/limits.conf: [search] results_serial_format=csv

2017-08-23 SPL-144350 Archived Index is created without error when the splunk index is invalid
2017-07-13 SPL-143111 "Splunkd daemon is not responding" when edit local windows event log collection
2017-04-04 SPL-140765 Splunk having problems extracting json file consisting of 68k plus key-value pairs
2016-11-29 SPL-133182 When two datasets have identical names but one is local (private) while the other is global, attempts to view or extend the global dataset use results from the local dataset instead.
2015-08-10 SPL-105061, SOLNESS-7274 Broken module prevents splunkweb from starting
2015-06-17 SPL-103247 Filtering on _time uses different semantics for the "=" operator on microseconds depending on whether the value is quoted.
2014-12-22 SPL-94910 The replace function does not apply to fields names with an underscore in them.

Workaround:
Rename the fields before the replace.

... | rename *_* AS *-* | replace "something" by "somethingelse"

2014-11-13 SPL-93039 The relevancy search command does not work, always returning 0 or -inf.
2014-10-02 SPL-91638, SPL-107375 For scheduled searches in a search head cluster, empty search jobs may appear in the job inspector for a cluster member.
2014-09-15 SPL-90861, SPL-90396, SPL-90886 If search encounters invalid offsets or invalid rawdata at TSIDX offsets, it skips reading any number of events from that bucket. No message is displayed, though the information is added to search.log.
2014-04-16 SPL-83129 Eval function strptime does not return results when 1970 date is used.
2014-04-04 SPL-82650 A report created and scheduled by admin cannot be embedded by a power user.
2014-03-27 SPL-82357 The splunk clean all -f CLI command doesn't remove data from the main index on Windows systems.
2014-03-15 SPL-81934 For clusters, may be unable to open search results output file for search results in a cluster.

Workaround:
Write to a temp file and rename to the target file.
2014-02-21 SPL-80942 Flashtimeline: 500 Internal Server Error when pasting long URL into panel name.
2013-12-18 SPL-78179 REST /saved/searches App names with special characters have invalid links.
2013-08-19 SPL-73386 Users are not allowed to run historical scheduled search

Workaround:
1. Create a special power/admin user who can run scheduled searches.

2. Assign this user ownership of the scheduled searches.

3. Share the searches at the app level and grant read/write permission to the correct set of users.

Saved search, alerting, scheduling, and job management issues

Date filed Issue number Description
2021-06-21 SPL-207588, SPL-195481, SPL-207587 Users are not able to open 'Advance Edit' feature while editing Alerts
2021-06-16 SPL-207377, SPL-207697, SPL-208513 E-mail-Addresses get moved from TO to BCC in Alert notification

Workaround:
Configure the emails in the alert followed by the symbol "," without typing any manual new line (Ej. Ctrl + Enter).
2021-05-06 SPL-205405 8.1.3 Username containing special character is not able to open 'Advance Edit' feature while editing Alerts / saved searches
2021-04-14 SPL-204072, SPL-207674, SPL-207675 Using a subsearch on an accelerated datamodel leads to incorrect results

Workaround:
limits.conf on the SH:

[search] phased_execution_mode = singlethreaded

2021-04-12 SPL-203954 realtime searches triggering alerts infinitely by mistake

Workaround:
Remove commands such as "fillnull" and "sort" from the first pipe of the search
2019-09-20 SPL-176812 Multiple SH Clustering with single deployer can't use datamodel summary sharing
2018-09-19 SPL-160286 The data preview for the Add Data workflow does not display for Log to Metrics source types
2017-11-29 SPL-146802 Distributed environment requires index defined on search head for log event alerts
2015-11-15 SPL-109471 For Real Time Scheduled Search in search head cluster, alerts are triggered twice when members cannot HB to captain
2015-04-09 SPL-99421 Long name of app causes accelerated search to not complete normally and shows invalid results on Windows 2008 R2

Workaround:
Reduce length of name of the app and report acceleration searches will run properly within the context of the app.
2014-08-15 SPL-89332 Report acceleration summaries do not show in Settings when you have hundreds of reports accelerated.
2014-08-05 SPL-88396 After configuring a client name for a deployment client, the name is not shown in the Forwarder Management UI

Workaround:
Create a server class, where you can see the client name, and use that group when you add data.
2014-05-01 SPL-83686 Data Model Pivot: Extra NULL column displays in Pivot with big data and Numbered Attribute in Split Columns.

Workaround:
The workaround is to add filter status=*, or make a more refined Data Model that has an object for events with status.
2014-03-24 SPL-82262, SPL-82241 Pivot search command fails for an admin trying to pivot on a Private Data Model created by a User.
2014-03-20 SPL-82164 Migrating invalid data models from 6.0 to 6.x fails.
2014-03-19 SPL-82133 Data model allows users to upload a JSON file which has Field names with spaces but will not validate it.
2014-03-10 SPL-81637 Splunkd preview runs indefinitely on any file preview with "DATETIME_CONFIG=none".
2014-03-10 SPL-81645 Creating data model with root transaction name starting with root event name fails
2013-11-26 SPL-77054, SPL-77055 Data model objects that have names starting with an underscore character ("_") do not work correctly and cannot be used in Pivot.

Charting, reporting, and visualization issues

Date filed Issue number Description
2016-09-15 SPL-128819, SPL-130243, SPL-130245 Editing panel in dashboard removes charting.legend.masterlegend option

Workaround:
Use <option name="charting.legend.masterLegend">null</option>
2016-04-27 SPL-118911 In SHC, referenced saved real-time searches in a dashboard do not stream results.

Workaround:
See Troubleshoot referenced real-time searches for workaround details.


2015-02-23 SPL-97193 The initial value for Multiselect input does not display properly in Visualizations Editor if input has empty string.

Distributed search and search head clustering issues

Date filed Issue number Description
2022-02-09 SPL-218862 Unable to see sourcetypes in introspection index

Workaround:
None
2022-02-01 SPL-218169, SPL-241835, SPL-241836 For alerts with per-result throttling (suppression) in SHC, sometimes, based on timing, the originating SH that ran the seach will show different results (suppressed) than the replicated artefacts on other SHs (unsuppressed)
2021-07-07 SPL-208356, SPL-217021, SPL-217022 Jobs Feed logs "Invalid custom progress ratio" for floating point progress ratios
2021-06-30 SPL-208136, SPL-212205, SPL-214177 Multisite indexer cluster - duplicated events returned when using assign_primaries_to_all_sites=false

Workaround:
In server.conf set:

[clustering] assign_primaries_to_all_sites = true

2021-05-22 SPL-206123, SPL-206757, SPL-206758 The authorize.conf setting srchIndexesDisallowed is ignored if search heads or peers are running version 8.0.x or lower in a mixed version environment

Workaround:
All search heads and search peers must be running Splunk Enterprise version 8.1.0 or higher.
2021-05-21 SPL-206067 With large KVstore temporal lookups that are replicated to indexers, turning ON enable_splunkd_kv_lookup_indexing may lead to indexer crash

Workaround:
By default enable_splunkd_kv_lookup_indexing is OFF. Do not turn ON enable_splunkd_kv_lookup_indexing configuration in limits.conf on Indexers.
2021-04-17 SPL-204230, SPL-206646, SPL-214045, SPL-214046 Crashing thread: TcpChannelThread and DispatchReaper Received fatal signal 11 - On Search Head Cluster.

Workaround:
Set enable_jobs_data_lite flag to false in server.conf.
2021-03-26 SPL-203060 The splunkd process changes the local distsearch.conf on service start

Workaround:
There is no workaround. After upgrading to Splunk Enterprise 8.x, the splunkd process checks and modifies the local/distsearch.conf on each service start. The process will:
  • Remove any settings that define default values already set in the /default/distsearch.conf file.
  • Removes comments preceded by a hash.
  • Reorders the KV pairs alphanumerically within a stanza.
  • Reorders stanzas within the file.


2021-02-01 SPL-200032, SPL-201499 Incorrect behavior of deployer_push_lookup_mode: Lookup isn't overwritten if global setting is always_preserve, local app setting is always_overwrite and the app on the deployer hasn't changed

Workaround:
Change any configuration option or the lookup itself on the deployer, for example, increment the version in app.conf:
/opt/splunk/etc/shcluster/apps/<myapp>/local/app.conf
[launcher]
version = 1.0.1
2018-03-14 SPL-152148 KV store replication fails on the upgrade search head during SHC member-by-member upgrade.

Workaround:
To ensure there is no kvstore activity during upgrade, perform an offline upgrade as follows:
  1. Shutdown all cluster members.
  2. Upgrade all members.
  3. Start the member


2017-11-29 SPL-146802 Distributed environment requires index defined on search head for log event alerts
2017-03-13 SPL-138654 Splunk searches fail when filepath gets too long on Windows
2016-07-12 SPL-124085 On Search Head Cluster It is not possible to remove an App from the SHs once it has been disabled.
2015-11-15 SPL-109471 For Real Time Scheduled Search in search head cluster, alerts are triggered twice when members cannot HB to captain
2015-09-23 SPL-106978 Failed SHC captain election causes unnecessary change in server.conf
2015-02-26 SPL-97385 $SPLUNK_HOME/var/run/splunk/snapshot contains large tarballs in the presence of large ES lookup table files.

Workaround:
The allowable size of the download can be increased by setting the following in server.conf.

[httpServer] max_content_length = 1500MB

The other option is to disable the search which controls the generation of the large lookup file. In this case, the search is:

[Endpoint - Local Processes Tracker - Lookup Gen]

2014-08-25 SPL-90028 Using "inputcsv dispatch=true" to read a CSV from a dispatch directory may not work on search head cluster members that have a replica of the desired artifact.
2014-08-14 SPL-89131 In a search head cluster, the search Job management page on cluster member doesn't immediately reflect 'isSaved' state after you click Save.
2014-08-02 SPL-88228 When user clicks on the RSS feed for an alert, search pool information is not displayed. Individual pool member information is displayed, however.

Data model and pivot issues

Date filed Issue number Description
2021-04-14 SPL-204072, SPL-207674, SPL-207675 Using a subsearch on an accelerated datamodel leads to incorrect results

Workaround:
limits.conf on the SH:

[search] phased_execution_mode = singlethreaded

2019-09-20 SPL-176812 Multiple SH Clustering with single deployer can't use datamodel summary sharing
2014-12-08 SPL-94047, SPL-98628 While creating a Pivot and using the _time column as a Split column, the table columns aren't formatted in a human readable way, but displayed with the epoc timestamp.It works when using _time as a 'Split Row' column.
2014-05-01 SPL-83686 Data Model Pivot: Extra NULL column displays in Pivot with big data and Numbered Attribute in Split Columns.

Workaround:
The workaround is to add filter status=*, or make a more refined Data Model that has an object for events with status.
2014-03-24 SPL-82262, SPL-82241 Pivot search command fails for an admin trying to pivot on a Private Data Model created by a User.
2014-03-20 SPL-82164 Migrating invalid data models from 6.0 to 6.x fails.
2014-03-19 SPL-82133 Data model allows users to upload a JSON file which has Field names with spaces but will not validate it.
2014-03-11 SPL-81701 Data Model Pivot, "Legend Position" and "Stack Mode" change to default settings if you change the X/Y-Axis more than once.
2014-03-10 SPL-81645 Creating data model with root transaction name starting with root event name fails
2014-03-07 SPL-81538 When using Pivot, stack mode is lost when "Scatter Chart" is selected.
2013-11-26 SPL-77054, SPL-77055 Data model objects that have names starting with an underscore character ("_") do not work correctly and cannot be used in Pivot.

Indexer and indexer clustering issues

Date filed Issue number Description
2023-02-02 SPL-235761, SPL-207384 Searches return incomplete results during addPeer/BatchAdding after CM restart.
2023-01-18 SPL-235074, SPL-207384 Searches return incomplete results during addPeer/BatchAdding after Cluster Manager restart.
2021-08-18 SPL-210523, SPL-212851, SPL-214213 Splunk search with 'delete' command on multisite indexer cluster cannot delete events.

Workaround:
Run delete command on every indexer instance.

Please try running your delete search on each indexer to see if you can then fully remove the events. You can do this from the CLI of the indexer (if you have disabled web UI on the indexers): $ splunk add user admin_delete -password <password> -full-name "admin_delete" -role admin -role can_delete

Example delete search: $ splunk search 'index=<INDEX> sourcetype=<SOURCETYPE> "<SEARCH_PHRASE>" earliest="02/25/2022:16:30:29" latest="02/25/2022:17:00:29" |delete'

Sample output: INFO: 1000 events successfully deleted INFO: Your timerange was substituted based on your search string splunk_server index deleted errors


------- ------- ------

idx1 __ALL__ 1000 0 idx1 <INDEX> 1000 0

Then repeat on the other indexers. Once done, check if all the events have been deleted by running your search on the search head again.

2021-06-30 SPL-208136, SPL-212205, SPL-214177 Multisite indexer cluster - duplicated events returned when using assign_primaries_to_all_sites=false

Workaround:
In server.conf set:

[clustering] assign_primaries_to_all_sites = true

2021-06-01 SPL-206510, SPL-213903, SPL-213901, SPL-213902 CM issues fixup tasks for "frozen in cluster" clustered buckets

Workaround:
Workaround 1:

Restart CM

Workaround 2: Use a REST search on the CM to only show fixup for non-frozen bucket. In versions lower than 8.2.0, the endpoint name is master, not manager.

| rest splunk_server=local /services/cluster/manager/fixup level=replication_factor 
| table index title initial.reason latest.reason level 
| append 
    [| rest splunk_server=local /services/cluster/manager/fixup level=search_factor 
    | table index title initial.reason latest.reason level ] 
| join title 
    [| rest splunk_server=local /services/cluster/manager/buckets f=title f=frozen search="frozen=0"] 
| stats values(level) AS level values(frozen) AS frozen values(initial.reason) AS initial.reason values(latest.reason) AS latest.reason BY index title
2021-05-20 SPL-205969, SPL-206226, SPL-207672 Linux 8.1.3 IDX clustering on Google Cloud - frequent crashes on CPs with Assertion `_chunk.size() == ((file_offset_t) -1) || n <= (size_t) _chunk.size()' failed.

Workaround:
No workarounds currently known.
2021-05-07 SPL-205462, SPL-210602, SPL-211950, SPL-214324 Indexers running out of space - Splunk not honoring the 'maxVolumeDataSizeMB' setting

Workaround:
  1. Use a lower value for maxVolumeDataSizeMB config - check the difference between disk usage and size reported by VolumeManger over a time period. Take average of this difference values (say N Mbs). Set maxVolumeDataSizeMB = maxVolumeDataSizeMB - N.

Search to see the configured and reported size per volume for an instance:

index=_introspection source=*disk_objects* component=Volumes host="<indexer in question>"
| timechart max(data.max_size) max(data.total_size) BY data.name
  1. In addition to 1, review the values of homePath.maxDataSizeMB for your indexers - this config parameter can also be used to provide limit on disk usage. Details on configuring this parameter are available in this documentation topic: https://docs.splunk.com/Documentation/Splunk/8.1.3/Indexer/Configureindexstoragesize
2020-11-20 SPL-197930 Splunk 8.1.0 post-upgrade indexer high memory usage with tsidxWritingLevel=4

Workaround:
Set tsidxWritingLevel to less than 4.
2020-10-07 SPL-196040, SPL-212495 Excessive logging of 'Unable to parse site_label, label=invalid due to err="Invalid site id: invalid"' for SearchResultsFiles

Workaround:
Increase the minimum logging level for SearchResultsFiles:
#log-local.cfg

[splunkd]
category.SearchResultsFiles=ERROR

https://docs.splunk.com/Documentation/Splunk/latest/Troubleshooting/Enabledebuglogging#log-local.cfg

2016-08-25 SPL-127353 Data rebalance finishes early when one peer is the source for all buckets

Workaround:
when only one indexer in a cluster indexed data (has all the searchable copies), rebalance once before adding the new indexer, and then rebalance a second time
2015-05-08 SPL-101184 Rolling restart in an Indexer Cluster may not be successful on a peer if a oneshot command is also running on that peer. Perform a manual restart to revive the peer.
2014-10-13 SPL-91861 On Windows indexer on an ec2 instance, splunk-optimize main thread can crash on buckets on the temporary drive z:\>.
2014-09-29 SPL-91432 On Windows when the master is down, the CLI command splunk offlinehangs when run from one of the streaming target peers.
2014-09-08 SPL-90630 On a multisite cluster, no warning is given when search head names are the same.
2014-07-29 SPL-87816 When implementing an indexer cluster or search head cluster, you cannot set pass4SymmKey in the general stanza. The system default values in the clustering and shclustering stanzas override any user-provided values in the general stanza.

Workaround:
Set the value in the [clustering] or [shclustering] stanza, depending on the type of cluster you're implementing.
2014-07-14 SPL-86799 After adding a new license to the clustering search head, splunkd on restart cannot be reached by splunkweb.
2014-04-29 SPL-83636 When configuring a multi-site cluster using cluster-config, the error messages are incorrect if the SF/RF was previously set.
2014-03-18 SPL-82038 Cluster-config does not work if a parameter value includes a space character.
2014-03-17 SPL-81955 Multisite: Peer takes approximately 6 minutes to restart when its site configuration is changed.
2014-01-06 SPL-78688 Peer is able to change to an invalid (empty) replication port
2013-08-06 SPL-72484 You cannot use the CLI to delete an index with a capital letter in its name.

Data Fabric Search issues

Date filed Issue number Description
2019-09-02 SPL-175783 Search results for the stats function perc95() are different for DFS and Splunk Enterprise

Workaround:
Use exactpercX(Y) function instead of perc95( ) to get more accurate results.
2019-07-24 SPL-173766 Search heads in a distributed search environment are unable to sync on available Spark resources

Universal forwarder issues

Date filed Issue number Description
2021-09-15 SPL-212200, SPL-214114, SPL-214937 Heavy Forwarder crashed with Crashing thread TcpOutEloop : Splunk version 8.1.4

Workaround:
Following configuration should reduce the frequency of crashes:

limits.conf [input_channels] max_inactive = 50000 inactive_eligibility_age_seconds = 36000

2021-04-28 SPL-204906, SPL-207677, SPL-208018 UF has problems recognizing the DST changes
2021-03-20 SPL-202721 UF 8.1.2/8.1.3 rpm -i returns cp: cannot stat '/opt/splunkforwarder/etc/regid.2001-12.com.splunk-UniversalForwarder.swidtag': No such file or directory complete
2020-11-30 SPL-198052, SPL-206073, SPL-206074, SPL-206076 Upgrading From 8.0.6 to 8.1.0.1 Using a DEB package results in a No such file or directory message.
2020-11-09 SPL-197140, SPL-234386 UF failed to start on Solaris 11.3 with error: "symbol in6addr_any: referenced symbol not found"

Workaround:
1. Do not upgrade past Splunk 8.0.5 on Solaris 11.3

OR 2. Upgrade to Solaris 11.4

2020-09-29 SPL-195635, SPL-202178, SPL-206477, SPL-202163, SPL-206534 Splunkd increased memory usage over time when monitoring UDP port(s) with in inputs.conf
2019-05-28 SPL-171178, SPL-167307, SPL-202078 Indexer Acknowledgement causes metric index events that do not have "_raw" fields to be duplicated

Workaround:
Indexer acknowledgement is a feature that helps prevent loss of data when forwarders send data to an indexer. Indexer acknowledgement is controlled by the Boolean useACK setting in inputs.conf and outputs.conf.

Indexer acknowledgement uses the _raw field to track completeness of delivery for each event. In some cases, when an event does not contain a valid _raw field, Splunk servers fail to determine whether the event is completely delivered and do not return acknowledgement for it, even when the event is processed successfully. As a result, the forwarder sends the same event again, leading to duplication of indexed data. This can affect metric indexes, where events with the JSON source type will not have _raw fields.

When this issue occurs, the workaround is to set useACK=false to disable indexer acknowledgement. You may want to set up multiple forwarding/HEC channels or ports with two useACK settings, to meet the needs of both kinds of source events: those that contain the _raw field and those that do not.

2018-04-10 SPL-153251 Universal Forwarder txz package cannot be installed on FreeBSD 11.1

Workaround:
1. Use pkg install instead of pkg add

OR 2. Install package by untarring tgz file to /opt/splunkforwarder

2017-03-20 SPL-139019 Possible compatibility issues between Python / SDK clients and new 6.6 and later default sslVersions, cipherSuites

Workaround:
Users can do either of the following:

1. Overwrite the new Splunk 6.6 server.conf [sslConfig] sslVersions, cipherSuites with your own settings that are compatible with your version of OpenSSL, e.g. the previous defaults from 6.5.x are compatible with OpenSSL 0.9.8 on Mac OSX:

[sslConfig]
sslVersions = *,-ssl2
sslVersionsForClient = *,-ssl2
cipherSuite = TLSv1+HIGH:TLSv1.2+HIGH:@STRENGTH

2. For some more up-to-date clients, it is possible to enforce TLS1.2 (e.g. --tlsv1.2 for curl) in order to connect successfully.

3. Upgrade OpenSSL on your platform and link it with your client (e.g. Python, curl, etc..). For example, OpenSSL 1.0.2 is currently available on Mac OSX via Homebrew (see https://brew.sh) and is compatible with the new Splunk 6.6 default sslVersions, cipherSuites.

2017-03-14 SPL-138731 New 6.6 and later default SHA256/2048-bit key certificates are not compatible with previous versions SHA1/1024-bit key certificates if cert verification is enabled

Workaround:
Users can do any of the following:

1. Disable certificate verification - the same root certificate is available with every Splunk download so enabling certificate verification while using the default certificates provides very little additional security.

2. Generate new SHA256/2048-bit key certificates using the new 6.6 root certificate and distribute to older versions of Splunk

3. Generate SHA1/1024-bit key certificates using the old root certificate to use with your new 6.6 instance. For convenience, the old root certificate is included in 6.6 in $SPLUNK_HOME/etc/auth/prev_release/

2015-06-10 SPL-103010 Indexing throughput on a forwarder with four pipelinesets drops 30% compared to a forwarder with two pipelinesets.
2015-04-14 SPL-99687, SPL-129637 Splunk universal forwarder is 7-10 days behind recent Windows Security and system log events.

Workaround:
To mitigate this, edit the following stanza in inputs.conf: [WinEventLog://Security] evt_resolve_ad_obj = 0.
2015-04-07 SPL-99316 Universal Forwarders stop sending data repeatedly throughout the day

Workaround:
In limits.conf, try changing file_tracking_db_threshold_mb in the [inputproc] stanza to a lower value.
2015-03-25 SPL-98594 Routing events to two different groups not working as expected.

Workaround:
1 On the original UF, instead of configuring 1 s2s and 1 syslog group, configure 2 s2s groups.

2 Setup a proxy UF which takes input from the original UF and send input out syslog server. This solution only requires config change and no patch release is required.

2014-08-05 SPL-88396 After configuring a client name for a deployment client, the name is not shown in the Forwarder Management UI

Workaround:
Create a server class, where you can see the client name, and use that group when you add data.

Distributed deployment, forwarder, deployment server issues

Date filed Issue number Description
2014-10-02 SPL-91648, SPL-91358 Forwarder unable to push scripted inputs to a Linux deployment client from a Windows deployment server.

Monitoring Console issues

Date filed Issue number Description
2021-03-29 SPL-203100 Summary page on monitoring console doesn't show correct RF/SF when not running on the CM.
2021-01-19 SPL-199534, SPL-201388, SPL-201386, SPL-201838 "DMC Alert - Total License Usage Near Daily Quota" does not work in case of fixed-sourcetype license
2020-11-26 SPL-198032, SPL-201347, SPL-201348 MC - Daily License Usage reports pool size incorrectly when filtered by pool

Workaround:
Step 1: Search and update text from

`$daily_usage_search$($splunk_server$, $size_search$, $host$, "$split_by_field_name$", "$pool$")`

to

`$daily_usage_search$($splunk_server$, $size_search$, $host$, "$pool$", "$split_by_field_name$")`

in the following file

$SPLUNK_HOME/etc/apps/splunk_monitoring_console/default/data/ui/views/license_usage_historic.xml

Step 2: Restart splunk

2019-11-13 SPL-179528 The splunktcp and splunktcp-ssl stanzas are not reloadable in inputs.conf
2017-08-18 SPL-144193 Bundle validation errors prevent future app deployment to indexer cluster
2017-08-14 SPL-143981 Uninstall app dialog does not show the app name correctly when the app doesn't have the label
2017-08-04 SPL-143664 Uploaded apps page makes two calls to packages endpoint
2017-05-24 SPL-141982 Upload modal should use size=large File element
2017-04-19 SPL-141274 Clicking Install multiple times in Install dialog causes error
2017-04-19 SPL-141273 Task endpoint fetch once even when there's no last deploy task id
2017-03-07 SPL-138351, SPL-172626 The role change of DMC via UI does not reflect to distsearch.conf

Workaround:
As a workaround can the customer manually modify the distsearch.conf.
2016-11-14 SPL-132151 XML error when trying to download uninstalled app

Splunk Web and interface issues

Date filed Issue number Description
2022-01-07 SPL-217087, SPL-205550 PDF generation process stuck and consuming CPU
2021-06-21 SPL-207573 The "../" characters are removed from the search string when used through Splunk Web UI.

Workaround:
escape the ../, so instead of searching for
/this/../something

you can search for

/this/.\./something

Alternatively, you can use REST API or CLI search instead when needing to search for ../

2021-03-31 SPL-203321 Splunk 8.1.* UI Focus jumps to the "Health Status of Splunkd" icon/link
2021-03-27 SPL-203092, SPL-203190, SPL-204188 Setting server.socket_host in web.conf to 127.0.0.1 on version 8.1.x fail with WARNING: web interface does not seem to be available!

Workaround:
If possibly use the other option ...

https://docs.splunk.com/Documentation/Splunk/8.1.3/Admin/Splunk-launchconf SPLUNK_BINDIP = <ip address>

2021-03-01 SPL-201713, SPL-206026, SPL-202687 Main splunkd crashing - thread: TcpChannelThread - Assertion `_redirectReply == REPLY_EATING_NORMAL' failed.

Workaround:
Do app updates manually via upload file or disable proxy
2020-12-04 SPL-198305 In Internet Explorer 11, using a "Favorite" link to access SplunkWeb fails

Workaround:
Register either one of the below URLs.

(1) For log in page, use the following URL.

e.g.) http://IP:Port/en-US/account/login

(2) Register non-login page such as Home, Search etc.

2019-07-11 SPL-173061 UI exposes a nonfunctional option for modifying permissions on custom search commands
2017-08-23 SPL-144350 Archived Index is created without error when the splunk index is invalid
2017-07-13 SPL-143111 "Splunkd daemon is not responding" when edit local windows event log collection
2016-11-14 SPL-132133 App Browser filtering of the apps does not work
2015-11-09 SPL-109165 Interactive Field Extractor hangs when using "^" as delimiter.

Workaround:
Use props and transforms to specify the delimiter of your choice.
2015-08-10 SPL-105061, SOLNESS-7274 Broken module prevents splunkweb from starting
2015-06-30 SPL-103701 Actions links should be removed for "Apps Browser"
2014-07-16 SPL-87015 chart count by source and *| cluster showcount=t | table cluster_count _raw) no metadata/ result is available when user drills down on Count and Percent columns.
2014-04-04 SPL-82650 A report created and scheduled by admin cannot be embedded by a power user.
2014-02-26 SPL-81103 Username surrounded by dollar signs cannot create saved searches.
2013-08-19 SPL-73386 Users are not allowed to run historical scheduled search

Workaround:
1. Create a special power/admin user who can run scheduled searches.

2. Assign this user ownership of the scheduled searches.

3. Share the searches at the app level and grant read/write permission to the correct set of users.

Windows-specific issues

Date filed Issue number Description
2021-12-13 SPL-216365 On Windows, warning messages concerning "larger than MAX_PATH" appear in splunkd.log when attempting to send email using sendemail.py.

Workaround:
Workaround: These warning messages are benign, and can be ignored. "WARN Pathname - Pathname 'C:\Program Files\Splunk\bin\Python3.exe C:\Program Files\Splunk\etc\apps\search\bin\sendemail.py"
2020-12-04 SPL-198305 In Internet Explorer 11, using a "Favorite" link to access SplunkWeb fails

Workaround:
Register either one of the below URLs.

(1) For log in page, use the following URL.

e.g.) http://IP:Port/en-US/account/login

(2) Register non-login page such as Home, Search etc.

2015-04-14 SPL-99687, SPL-129637 Splunk universal forwarder is 7-10 days behind recent Windows Security and system log events.

Workaround:
To mitigate this, edit the following stanza in inputs.conf: [WinEventLog://Security] evt_resolve_ad_obj = 0.
2015-04-01 SPL-98978 On differing versions of Splunk Enterprise indexer (5.0.1) and universal forwarder (6.2.2), collection of the Security Event log can take increasingly longer over time.

Workaround:
To fix the problem, restart Windows on the forwarder.


2014-10-31 SPL-92596 After upgrade from Splunk Enterprise 6.1 or earlier to 6.4.x on Windows, splunkweb service does not start automatically. Attempts to start it manually show "Error 1053: The service did not respond to the start or control request in a timely fashion."

Workaround:
This is expected behavior. See the Splunk Answers post: http://answers.splunk.com/answers/177187/why-is-the-splunk-web-service-not-running-after-an.html
2014-09-25 SPL-91279 Splunk Universal Forwarder on Windows (specifically, the splunk-perfmon.exe process) does not release key handles.

Workaround:
See "Handle leak when an application collects performance data in Windows Vista, in Windows 7, in Windows Server 2008 or in Windows Server 2008 R2" on the Microsoft Support website for a hotfix download.
2013-10-11 SPL-75116 The UI does not show configured items of some newly converted windows modular inputs that contain the name "default" in the stanza

Workaround:
Edit inputs.conf: in stanzas that contain WinRegMon://default, replace "default" with something else, then restart splunk.

REST, Simple XML, and Advanced XML issues

Date filed Issue number Description
2021-11-03 SPL-214703, SPL-213950 EPS drops after upgrade as a result of default 50k export cap in limits.conf
2021-07-02 SPL-208295, SPL-208430, SPL-211259, SPL-211262 Can't reassign objects with colons on All Configurations page

Workaround:
|rest /services/configs/conf-props |fields id eai:acl.owner eai:acl.app eai:appName
2021-04-08 SPL-203763, SPL-204069, SPL-204488 savedsearches.conf is not updated after upgrading to 8.1.0 because REST API endpoint is not reloaded.
2020-07-28 SPL-192792 tsidxWritingLevel and other fields are set empty after updating index in UI
2017-07-13 SPL-143111 "Splunkd daemon is not responding" when edit local windows event log collection
2016-10-31 SPL-131072 Datamodel backend allows invalid time values
2013-05-15 SPL-67453 When sending the following XML data as a GET or POST param to a custom splunkd endpoint: <dashboard>&lt;foo&gt;</dashboard>, the endpoint actually receives:<dashboard><foo></dashboard>.

PDF issues

Date filed Issue number Description
2022-01-07 SPL-217087, SPL-205550 PDF generation process stuck and consuming CPU
2016-11-23 SPL-132925 Table data rows generated with the addcoltotals command do not show up in PDF

Workaround:
If you are using addcoltotals to generate a totals data row, renaming the _time field can cause PDF generation issues.

Remove the label and labelfield or change the label to a number to generate the PDF as expected.

2015-03-31 SPL-98890 Maps printed from Report page do not honor custom zoom and center.
2014-06-16 SPL-85497 Unable to save generated PDFs using Chrome internal PDF viewer.

Workaround:
Workaround: Enable Adobe Acrobat or Acrobat Reader as the default PDF viewer in Chrome. For more information, seehttps://support.google.com/chrome/answer/142056.


Admin and CLI issues

Date filed Issue number Description
2021-07-02 SPL-208295, SPL-208430, SPL-211259, SPL-211262 Can't reassign objects with colons on All Configurations page

Workaround:
|rest /services/configs/conf-props |fields id eai:acl.owner eai:acl.app eai:appName
2021-04-28 SPL-204953, SPL-209393, SPL-212181 A vCPU license is subject to conditional license enforcement
2021-04-08 SPL-203821, SPL-182510, SPL-204154 Splunk anonymize will crash in certain Python 2 and Python 3 environments.

Workaround:
Use an older Splunk instance to anonymize.
2021-03-26 SPL-203060 The splunkd process changes the local distsearch.conf on service start

Workaround:
There is no workaround. After upgrading to Splunk Enterprise 8.x, the splunkd process checks and modifies the local/distsearch.conf on each service start. The process will:
  • Remove any settings that define default values already set in the /default/distsearch.conf file.
  • Removes comments preceded by a hash.
  • Reorders the KV pairs alphanumerically within a stanza.
  • Reorders stanzas within the file.


2020-07-28 SPL-192792 tsidxWritingLevel and other fields are set empty after updating index in UI
2020-04-14 SPL-186365 Users are able to create/clone knowledge objects into apps where they lack permissions
2019-08-05 SPL-174406, SPL-109254 Root unable to run splunk cli if SPLUNK_OS_USER is set
2018-08-13 SPL-158658 A timeout or slow response when accessing Splunk Web Licensing page

Workaround:
A timeout or slow performance of the license management page is caused by a build-up of historical license warning messages, which are processed every time the page is accessed. Can be verified by running this search on the License Manager:

| rest splunk_server=local /services/licenser/messages

If a high value is returned for that end point, you are likely affected. Log a support ticket with Splunk to obtain a license reset key, and apply the key to clear out any historical license warning messages. After the reset license is applied, the license management pages should load normally.

2017-11-29 SPL-146820 Unable to access some settings/manager pages (data model editor) if starting from the setup page of a non-visible app

Workaround:
Navigate to a visible app, such as the search and reporting app, and access the Splunk settings pages from that app context.
2017-04-03 SPL-140747 SSL connection in Python when using new ciphers may be slow.
2016-11-09 SPL-131880 Reports/Alerts owned by the deleted user cannot be found in the Orphaned filter for the Reassign Knowledge Objects page
2015-09-23 SPL-106978 Failed SHC captain election causes unnecessary change in server.conf
2015-03-11 SPL-97942 Capability defined in an app does not take effect when assigned to a role

Workaround:
The workaround is to change the ui-prefs in ./etc/users/username/local/ui-prefs.conf to look like this:

[search] display.events.fields = ["description","except_extract_1","except_extract_2","except_extract_3","sap_order_status","sourcetype","source","status","request_mode","request_id","request_status_id","object_id","BillToCity_","Airline_","BillToName_","BillToCountry_","City_"] display.events.type = table

2014-04-07 SPL-82699 SSO: Acceleration icon fails to display in Searches, Reports, and Alerts page.
2013-05-25 SPL-68010 The error thrown when your Splunk instance cannot connect to splunkbase/.../checkforupdate is not an ERROR, should be lowered to INFO.

Workaround:
Set server.conf [applicationsManager] allowInternetAccess = false
2013-05-02 SPL-66511 If $SPLUNK_HOME/etc is located on a case-insensitive filesystem, creating a new view with the same name as an existing view but with different case (capital letters vs lowercase, etc) silently overwrites the existing view.

Uncategorized issues

Date filed Issue number Description
2022-04-14 SPL-222658 List of third-party software incorrectly specifies zlib version 1.2.8 instead of version 1.2.11
2022-02-24 SPL-219715, SPL-225376, SPL-225374, SPL-225375 Workload Management fails to enable on restart if a rule contains a role that is missing on the platform
2022-02-07 SPL-218453, SPL-216068 Can not use field alias when searching virtual index
2021-12-01 SPL-215691, SPL-216068 Can not use field alias when searching virtual index
2021-11-29 SPL-215556, SPL-216996, SPL-217246 Splunk Enterprise - when httpout is configured on Splunk Enterprise, splunkd should return a failure

Workaround:
There is no workaround apart from realising that HFW cannot be used.
2021-07-05 SPL-208338, SPL-208754, SPL-210528, SPL-210529 When using a License Manager that has both ITSI and Hunk license installed, all connected Splunk instances are showing Hunk branding
2021-06-07 SPL-206856, SPL-205891 8.1 Export button is a half visible in report
2021-06-03 SPL-206757, SPL-206123 The authorize.conf setting srchIndexesDisallowed is ignored if search heads or peers are running version 8.0.x or lower in a mixed version environment

Workaround:
All search heads and search peers must be running Splunk Enterprise version 8.1.0 or higher.
2021-06-03 SPL-206758, SPL-206123 The authorize.conf setting srchIndexesDisallowed is ignored if search heads or peers are running version 8.0.x or lower in a mixed version environment

Workaround:
All search heads and search peers must be running Splunk Enterprise version 8.1.0 or higher.
2021-05-26 SPL-206233, SPL-208289, SPL-206946, SPL-207301 Tsidx Reduction creates very large amount of "minify-tsidx fsck" and repeatedly for the same bucket at the same causing large amount of memory usage and OoM in extreme cases

Workaround:
disable tsidx reduction until you upgrade to 8.1.5+ or 8.2.2+

indexes.conf

enableTsidxReduction = false
2021-05-26 SPL-206283, SPL-204489 Many events from internal logs are directed to malformedEventIndex
2021-05-10 SPL-205482, SPL-206086, SPL-206095 Post 8.1.3 upgrade: existing SAML configuration ceased to function. Configuration UI displays "Method not allowed" message
2021-05-04 SPL-205256, SPL-204519 SAML Scripted Auth Extensions login() function uses wrong assertion attribute
2021-05-04 SPL-205249, SPL-203901 Settings in Authentication Extension cause SH to crash

Workaround:
Do not use the Authentication Extension
2021-05-04 SPL-205247, SPL-204519 SAML Scripted Auth Extensions login() function uses wrong assertion attribute
2021-05-04 SPL-205248, SPL-204519 SAML Scripted Auth Extensions login() function uses wrong assertion attribute
2021-05-02 SPL-205109 Excessive WARN ScopedLDAPConnection "Converting non-UTF-8 value to" in the splunkd.log file
2021-04-28 SPL-204962, SPL-205729, SPL-207105 Scheduled Jobs sendemail csv file Generated Extra Blank Rows

Workaround:
change python back to version 2

etc\apps\search\default\commands.conf [sendemail] python.version = python2

2021-04-21 SPL-204497, SPL-207696, SPL-209394, SPL-209395 httpout on UF behaves different than tcpout when there is no TZ in the event
2021-04-15 SPL-204166 OEM customer can't elevate user privileges to configure SAML
2021-04-15 SPL-204162, SPL-207380, SPL-207438 Splunk diag: ERROR Cannot process directory path in Windows after upgrade to 8.1.2
2021-04-10 SPL-203946, SPL-204620, SPL-204420, SPL-204691 Extremely huge increase in LDAP query sent from Splunk SHs after upgrade from 8.0.7 to 8.1.3
2021-04-09 SPL-203922, SPL-204668, SPL-204351 Admission rule search_time_range=alltime does not respect time modifiers in search query and will filter all searches done via API

Workaround:
Disable or delete the admission rule with search_time_range=alltime to allow API searches. The preferred way to do this is via the user interface but an alternate is to disable in the workload_rules.conf.
2021-03-19 SPL-202682 The license usage report tab name is Previous 60 days, but the reports run over the last 30 days
2021-01-21 SPL-199630, SPL-198659 License warning '1 pool warning reported by 1 indexer' comes up on Licensing page
2020-10-01 SPL-195810 Using CLI command to stop migration of KVstore on a SHC running on Windows OS can cause the SHC captain to reach an invalid state

Workaround:
Restart the SHC captain
2020-08-10 SPL-193389 Parallel upload is not supported in gcp-sse-kms encryption mode

Workaround:
In the volumes using gcp-sse-kms encryption mode, specify "remote.gs.upload_chunk_size = 0" to disable parallel upload.
2020-07-30 SPL-192936 Subsecond search - When you update metric.timestampResolution via the UI, it is not updated on the search head index.conf file. This does not affect search functionality.
2020-07-30 SPL-193702 When SmartStore is enabled, having 'always_include_indexedfield_lispy=true' in limits.conf causes reduced search performance

Workaround:
Having always_include_indexedfield_lispy=true in limits.conf when you have SmartStore enabled can cause unnecessary cache churn, resulting in increased load on the network due to buckets being unnecessarily downloaded from remote storage

To resolve this issue, set always_include_indexedfield_lispy to false.

2020-05-06 SPL-188800 Starting Splunk software with incorrect KV store storage engine causes KV store to crash

Workaround:
In the [kvstore] stanza of your server.conf file, set the storageEngine setting to match the storage engine that you're using, either wiredTiger or mmapv1. To learn which storage engine you're using, check whether the file extensions in the var/lib/splunk/kvstore/mongo directory are *.wt for Wired Tiger or *.ns for Memory Mapped.
2019-10-03 SPL-177447 Bundle replication takes longer than expected time for indexers that have bundleEnforcerBlacklist configured
2019-09-26 SPL-177144, SPL-177326 Under heavy search workload, the search memory usage estimation may be higher than actual usage
2019-09-25 SPL-177008, SPL-176710, SPL-177009 Workload management fails to enable for addition of a pool with 1% cpu and 1% memory
2019-09-16 SPL-176514 Offline rebuild of unsearchable bucket may lead to stale information in dbinspect searches
2019-09-13 SPL-176447 SmartStore: Migration uploads of auto_high_volume buckets can fail indefinitely due to an XFS bug

Workaround:
Before migration, lower the max_concurrent_uploads setting in server.conf to 2.

After migration, revert the setting to the default of 8.

2019-07-19 SPL-173449, SPL-173259 timezone isn't stored for start_time/end_time of rule schedule every_day/every_week/every_month
2019-06-05 SPL-171553, SPL-171647 Smartstore: S3 GET is being done before S3 PUT for the receipt.json causing 404 errors (Source peer should not check if the bucket/receipt exists during uploads)
2019-03-26 SPL-168314 SmartStore standalone instance + Monitoring Console: Bootstrapping panel needs to reflect the standalone bootstrapping process
2018-10-17 SPL-161632 Can't install RPM Splunk 7.2+ file in Red Hat EL5
2018-09-04 SPL-159598 mongo 3.4 to 3.6 upgrade sometimes misses fcv document
2018-03-20 SPL-152330, SPL-151992 After installing Splunk on Windows using msiexec and the "GENRANDOMPASSWORD=1" option (and if generated password ends with backslash) admin is unable to login with msg "No users exist. Please set up a new user."

Workaround:
Create a $SPLUNK_HOME/etc/system/local/user-seed.conf and restart Splunk

[user_info]
PASSWORD = <yourpassword>


2018-01-25 SPL-148514 Splunk not starting on Linux kernel version 4.13.0-31

Workaround:
Do not upgrade kernel to version 4.13.0-31. Use either an older release or 4.13.0-32.35+
2017-05-09 SPL-141693 DataModel Editor - when child object has same name as inherited field, inherited field does not show in the inherited fields list.
2017-04-27 SPL-141478, SPL-237563 $_index_name does not resolve properly when used with the thawedPath pathname
2017-03-27 SPL-140442, SOLNESS-11786 In Splunk Enterprise 6.6.0 and later, with Enterprise Security 4.5.2 and 4.6.0, roles without "edit_roles" capability cannot perform operations on notable event review statuses.

Workaround:
If users cannot perform operations on notable event review statuses or have issues viewing "Edit all selected" links on Incident Review, user roles must be provided with the "edit_roles" capability.
2017-01-06 SPL-134707 Splunk restart does not create missing server.pem certificate on Windows

Workaround:
Use bin/splunk createssl server-cert -d etc/auth/ -n server to generate a new certificate.
2016-11-21 SPL-132670 Mac OS 10.11: disable boot-start doesn't remove the file /Library/LaunchAgents//com.splunk.plist by enabling boot-start in prior Splunk/UF
2016-08-31 SPL-127800 Opting in to data sharing on a monitoring console produces duplicate data
2016-07-26 SPL-125052 Sole Admin can demote themself to Power without path of recovery in GUI.

Workaround:
Through the command line, you can open notepad and modify the password file to regain 'Admin' status.
2016-06-21 SPL-123174 JSON indexed_extractions doesn't work for TCP inputs
2015-10-07 SPL-107606 Inconsistency between summary and datamodel_summary files.
2015-06-18 SPL-103302 Files ownership are failed to be changed when using debian package to install splunk and $SPLUNK_HOME is a symlink

Workaround:
Run a recursive chown from the command line on $SPLUNK_HOME manually, post install.
2015-05-24 SPL-102008 On Internet Explorer, a warning message does not display when you cannot log in due to a time zone difference.
2015-05-11 SPL-101289 When the number of indexing pipeline sets is greater than four, indexing throughput decreases.
2015-05-06 SPL-100980 Single indexer does not scale when receiving parsed data from multiple PipelineSets.
2015-05-04 SPL-100792 There are multiple group=thruput metrics lines in metrics.log. Searches that do not differentiate among them may get falsely high totals.

Workaround:
Searches that key off these lines need to select their desired name=x category in order to see a single thruput value.
2015-04-24 SPL-100322 A view gets stuck with "loading" due to problematic navigation (default.xml)

Workaround:
Workaround is to use label attribute for collection element.

<collection label="Others">

           <view source="unclassified" match="Dashboard"/>
     </collection>  
2015-03-26 SPL-98700 splunkd Indexer crashes in IndexerTPoolWorker due to duplicated bucket id.

Workaround:
The workaround is to remove the duplicated bucket.
2015-02-26 SPL-97389 When using timechart command, the embedded report shows different time format than the original report.
2015-01-08 SPL-95144, SPL-101986, SPL-101987, SPL-106884, SPL-107317, SPL-142789 Indexed message for Windows security event logs shows "FormatMessage error"

Workaround:
Splunk believes this was introduced in a Microsoft Windows patch. The workaround is to configure a delayed start of the Splunk service(s) so that it starts after the Windows Event Log service.
2014-11-10 SPL-92831 A mismatch of versions between the license-master and the license-slave is generating Warning messages like "WARN LMDirective - directive cmd=D_set_feature_state args='Acceleration,ENABLED' failed: reason='feature='Acceleration' is invalid' ."

Workaround:
The warnings can be ignored, the workaround is use same major versions (all on 6.2 or all on 6.1).


2014-10-24 SPL-92432, SPL-99583 Chart in dashboard panel does not honor interval settings.

Workaround:
In the panel XML, specify a larger height to use the correct interval settings.
2014-10-17 SPL-92162 Writing large amounts of data (> 20 GB) to KV store collections using outputlookup can result in high memory usage on the machine.
2014-09-11 SPL-90738 Monitoring a directory with an unknown sourcetype produces indexing errors.
2014-08-26 SPL-90139 <timestamp> does not display in the Patterns tab when searches are run in fast mode.
2014-04-22 SPL-83365 Splunk Enterprise on Windows does not show an error message when a user without the edit_license capability tries to add a license through the CLI.
2014-04-14 SPL-83068 Default index can be set to random index.
2014-04-01 SPL-82517, SPL-208875 Paper Size and Layout in PDF Schedule dialog do not respect Paper Size and Layout in Email Settings.
2014-03-23 SPL-82238 Datamodel fails to drill down further when the same attribute for Split Rows and Split Columns are selected.
2014-03-13 SPL-81856 Show all lines does not work in data model editor preview.
2014-03-12 SPL-81810 Licensing - license pool warning at license master keeps coming back after deleting it.

Workaround:
Delete the warnings on the peers first, then the License Manager.
2014-03-12 SPL-81781 In the Data Model Manager, "Acceleration Status" and "Access Count" fail to update when you click "Update".
2014-02-13 SPL-80568 Highcharts determines Y-axis values based on first point outside visible range.
2014-02-07 SPL-80285 In the Data Model Editor, the Edit Lookup page is blank if Lookup is shared only in Lookup Definitions.

Workaround:
For more information, see Add lookup files to Splunk.
2014-02-06 SPL-80187 In the Data Model Editor, lookup pages open with options displayed for other Lookup when the data model definition is private but the file is app or globally shared.

Workaround:
Share the definition. For more information, see Add lookup files to Splunk.
2014-01-31 SPL-79842 On Windows, Indexer doesnt accept new connections on splunktcpin port after queue blockage is resolved
2013-11-27 SPL-77139 Licenser pool usage gets reflected only after restarting splunkd.
2013-10-29 SPL-75764 Forwarder forwards duplicate data after props.conf is in place for cross platform scenario/when the forwarder is on Solaris and the indexer is on Linux.
2013-09-13 SPL-74337, BETA-496 You cannot specify a destination folder when installing on OSX.
2013-09-10 SPL-74209, SPL-74167 Persistent queues are not created on Windows for stanzas that contain unusual characters (such as < and >).

Workaround:
Specify the persistentQueue explicitly in the input definition.
2013-08-28 SPL-73826 Windows: hostname override not working properly
2012-02-22 SPL-48342 LDAP strategy host field cannot work with ipv6 format address but computer name is okay
2010-10-08 SPL-34347 wmi input default fields - with value including newlines doesn't search properly becasue of \r\n issue

Splunk Analytics for Hadoop

Date filed Issue number Description
2017-04-04 ERP-2040 Splunk archiving fails for large block sizes (buckets) due to HDFS write crashes for Hadoop version 2.8, 2.7.x

Workaround:
Upgrade Hadoop to 2.8.2 or higher.
2015-09-09 ERP-1650 timestamp data type not properly deserialized.
2015-08-05 ERP-1619 Searching on a newly created archive index before the bucket copy saved search is run causes a filenotfound exception.

Workaround:
Reenable the bucket copy saved search and let it run, or force the archiving to happen via | archivebuckets force=1 and then rerun the search.
2015-07-07 ERP-1598 minsplit rampup - splits generation takes too long.

Workaround:
Set minsplits=maxsplits
2015-05-12 ERP-1502 Non-accelerated pivot search on Pivot UI page waits for a long time to return result.
2015-01-08 ERP-1343, SPL-95174 Splunk Analytics for Hadoop searches fail on corrupted journal.gz files, although Splunk searches run without error.

Workaround:
Add the journal.gz to the input path's blacklist (vix.input.1.ignore = ....)
2014-10-27 ERP-1216 Data Explorer preview does not honor existing sourcetypes for big5/sjis files.
2014-10-03 ERP-1164 Report acceleration summary gets deleted when two Splunk Analytics for Hadoop instances point to the same Splunk working directory.

Workaround:
To mitigate this issue, make sure that vix.splunk.home.hdfs (or Working directory in the UI) is unique on both search heads that are not in a pool. To keep your instances in the same working directory, configure vix.splunk.search.cache.path to be unique on both search heads.
Last modified on 10 December, 2024
Welcome to Splunk Enterprise 8.1   Splunk Enterprise and anti-virus products

This documentation applies to the following versions of Splunk® Enterprise: 8.1.3


Was this topic useful?







You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters