Hi @gcusello  thank you for your prompt response, I have seen the link but it's bit complicated to understand , I have checked https://splunkbase.splunk.com/app/3249/?_gl=1*176xpt6*_ga*MTI2NzYyMjM0NC4xNjUwNTcxNDAz*_gid*MTc4MTA3NzI0Mi4xNjU4NDA3MTQz&_ga=2.108078146.1781077242.1658407143-1267622344.1650571403#/details But the steps are not clear for me, I would like your assistance if possible and provide me the steps to do on each of the listed components: SHs, manually ( Already done via GUI)  IDXs, manually, (steps to do this in CLI cause i'm using CentOS) HFs, manually. (steps?) Kindly help me with the steps to do this, Thank you in advance.   ... View more

Dear  @PickleRick , Search peer Splunkidx03 has the following message: Detecting bucket ID conflicts: idx=_internal, bid=_internal~518~B239BEEE-90FA-43C8-ADDA-620D3FACAB66, path1=/opt/splunk_data/indexes/_internaldb/db/518_B239BEEE-90FA-43C8-ADDA-620D3FACAB66, path2=/opt/splunk_data/indexes/_internaldb/db/db_1651988707_1651737547_518_B239BEEE-90FA-43C8-ADDA-620D3FACAB66. Temporally resolved by disabling the bucket: path=/opt/splunk_data/indexes/_internaldb/db/DISABLED-db_1651988707_1651737547_518_B239BEEE-90FA-43C8-ADDA-620D3FACAB66. Please check this disabled bucket for manual removal.   is the above error gives you more info on how you could advice me to fix the following error, I checked on splunkd.log   05-22-2022 19:54:03.001 +0200 WARN TcpOutputProc - Applying quarantine to ip=x.x.x.13 port=9997 _numberOfFailures=2 05-22-2022 19:54:03.004 +0200 ERROR X509Verify - X509 certificate (O=SplunkUser,CN=SplunkServerDefaultCert) failed validation; error=10, reason="certificate has expired" 05-22-2022 19:54:03.004 +0200 WARN SSLCommon - Received fatal SSL3 alert. ssl_state='SSLv3 read server certificate B', alert_description='certificate expired'. 05-22-2022 19:54:03.004 +0200 ERROR TcpOutputFd - Connection to host=x.x.x..14:9997 failed. sock_error = 0. SSL Error = error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed - please check the output of the `openssl verify` command for the certificates involved; note that if certificate verification is enabled (requireClientCert or sslVerifyServerCert set to "true"), the CA certificate and the server certificate should not have the same Common Name. 05-22-2022 19:54:03.004 +0200 WARN TcpOutputProc - Applying quarantine to ip=x.x.x.14 port=9997 _numberOfFailures=2 05-22-2022 19:54:03.007 +0200 ERROR X509Verify - X509 certificate (O=SplunkUser,CN=SplunkServerDefaultCert) failed validation; error=10, reason="certificate has expired" 05-22-2022 19:54:03.007 +0200 WARN SSLCommon - Received fatal SSL3 alert. ssl_state='SSLv3 read server certificate B', alert_description='certificate expired'. 05-22-2022 19:54:03.007 +0200 ERROR TcpOutputFd - Connection to host=x.x.x.15:9997 failed. sock_error = 0. SSL Error = error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed - please check the output of the `openssl verify` command for the certificates involved; note that if certificate verification is enabled (requireClientCert or sslVerifyServerCert set to "true"), the CA certificate and the server certificate should not have the same Common Name. 05-22-2022 19:54:03.008 +0200 WARN TcpOutputProc - Applying quarantine to ip=x.x.x.15 port=9997 _numberOfFailures=2 05-22-2022 19:54:08.090 +0200 WARN TcpOutputProc - The TCP output processor has paused the data flow. Forwarding to output group primary_indexers has been blocked for 11520 seconds. This will probably stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data. 05-22-2022 19:54:18.001 +0200 WARN TcpOutputProc - The TCP output processor has paused the data flow. Forwarding to output group primary_indexers has been blocked for 11530 seconds. This will probably stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data. 05-22-2022 19:54:22.247 +0200 INFO CMBucket - set bucket summary bid=nc_fw_sophos~1265~A91B9781-86B7-4ECC-9DF2-D6C6F6B75A08 summaryId=F237DE98-1722-40E2-AA0E-9964094F7F12_DM_Splunk_SA_CIM_Web peer=9F50A957-648B-40D7-8B1D-CB8E511C8EA5 type=data_model state=done modtime=1653242047.000000 checksum=7E3C17CAACC1DD664BD299E51A27F53D508F1B1B49BB18AB41F56995DA0ACA03 05-22-2022 19:54:26.759 +0200 INFO ClientSessionsManager:Listener_AppEvents - Received count=9 AppEvents from DC ip=x.x.x.12 name=EF0E61E4-E613-4114-8794-822E03173A9C   With the above info, may you help me to understand more about the error and hw to fix it? Thank you in advance! ... View more

Thank you so much for your kind help @PickleRick . ... View more

Thank you so much, Actually i wanted to see how much data is consumed on disk per each day. splunk btool indexes list | grep 'Path ='   The above command , which Path you mean here?   the second command gives me this in one indexer: [root@Splunkidx1 splunk]# du -ksx /opt/splunk/var/lib/splunk 47263660 /opt/splunk/var/lib/splunk [root@Splunkidx1 splunk]# Is the 47263660 , is the data consumed per day? if not how can i see the amount of data,  disk consumed per day? - I want to check all these and know , how much other storage to buy , so that i can be able to store more data for longtime? -Thank you in advance. ... View more

Thank you again @PickleRick  for your quick response,   - Is there any query that can show me the final volume of data indexed in indexers per day in GB? * in CLI / GUI? -Thank you in advance ... View more

Thank you so much @PickleRick for your kind support and response, I got now the point, may you please share/guide me how to use query or other way i can use to check on GUI/CLI the total data indexed per day in GB after this exercise (15% for compressed raw data, 35% for indexes)? Thank you in advance. ... View more

Hello dear @PickleRick , thank you for your feedback, But i didn't well understood, let me again explain well my question in details,  Actually I have a license of 200GB/per day? but when I check is license used previous 30 days, the average is between 36GB and 54GB , means per day i cannot pass 60gb,  I also have 5 indexers and each has the space of 2T , BUT I WANT TO KEEP MY DATA FOR LONGTIME FOR 5YEARS, HOW DO I CALCULATE THIS TO GET THE STORAGE FIT THE 5 YEARS?which query or formula can i use? HOW MANY Terabyte(TB) I WILL NEED TO KEEP MY DATA FOR 5YEARS in the scenario above, Thank you in advance! ... View more

index=_internal source=*license_usage.log type=Usage | stats sum(eval(b/1024/1024/1024)) AS volume_b by idx date_mday date_month date_year | stats max(volume_b) by idx date_month date_year   help me to edit this query to get the output in GB and also calculate the total number of all index in GB? Thank you in advance ... View more

How do I use this query to get the data in GB in readable format? index=_internal source=*license_usage.log type=Usage | timechart span=1d sum(eval(b/1024/1024/1024)) AS volume_b by idx   - I also want to see how much data stored in all indexers per day, per month and per year? ,what is the query i can use? Thank you in advance ... View more

Thank you dear @PickleRick for your quick response about this matter, I would also need to know the list of things/requirements will have to implement this TEST environment  to work same as the production env one.   1- which VM i will use? VirtualBox will work? 2- For OS is choose CentOS as the one i use in production? am i going to install other centos for each components(5 indexers, 1 SH ,1 Management node) to match the the numbers of the components in production?????? OR i will use one? 3- About Splunk enterprise software, am going to use this one for Download free- 60 trial (https://www.splunk.com/en_us/download/splunk-enterprise.html) OR i will use the one i used in production?????? 4- What about the indexers and how i will install and configure it, is there any link related or another guidance so that it can meet the production? 5- Kindly help me with the list of things i need to have in test environment to meet the production and what to do next? Thank you in advance   ... View more

Thank you @kkrises for your advice and guidance, When I have downloaded the version of Splunk ES https://splunkbase.splunk.com/app/xxx/, is this enough to upgrade splunk ES to 7.0.0 and Splunk Enterprise version 8.1.x- 8.2.4)???? or I have to also download other one for Splunk Enterprise version 8.1.x to 8.2.4 ?????   Another question, what command to use in CLI or use GUI to upgrade after getting upgrade version file splunk-enterprise-security_700.spl ??? Kindly help me, how to upgrate it after getting this upgrade version file?   Thank you in advance     ... View more

@Thank you dear @richgalloway for your guidance, The only problem, I still have, I have seen splunk ES has various versions of Splunk enterprise that is compatible to splunk ES v 7.0.0 - May you help me how i can upgrade splunk enterprise to the latest one which is 8.2.4 and Splunk ES to 7.00, is it possible to upgrade splunk enterprise from 7.2.6 to 8.2.4 ???? also possible to upgrade splunk ES from 5.3.0 to 7.0.0 ???? If possible may you please guide me how to upgrade them, what are the requirements??? Kindly help me on this cause in the documents is little bit confused, i don't well understand this, Thank you in advance.  ... View more

Dear @gcusello , is there any guidance on how to create test environment step by step or you can advise me , here i think we'll use a single instance deployment? - is it possible that you can do this test environment and work as production for free? - what splunk enterprise and splunk enterprise security we can use for testing environment? free splunk enterprise is limited to 500MB only and i don't what to use so that you can do the test environment which can be similar to the production? what are I required to have?and how to do it?  Kindly guide me on this, as i want to create test environment similar to the production? Thank you in advance!!     Thank you   ... View more

Thank you so muchMr @isoutamo  for your help, yes we actually use LVM, And as i trace all the indexers i found in : /dev/mapper/centos-opt , on some of indexer is almost full 92%, and on the other indexer is full 100%,  what are the way forward to resolve this, May you please guide me on how to fix this? there is no way to reduce this by delete maybe the old data instead of add other space? need your advice and guidance on this, Thank you in advance! ... View more