Analyzer - Log Files - FAQ
Content
How to gather the necessary information for a CLARiiON performance analysis
Page 1 of 4
EMC Knowledgebase
"How to gather the necessary information for a CLARiiON performance analysis"
ID: Usage: Date Created: Last Modified: STATUS: Audience: emc161922 1416 05/24/2007 05/12/2011 Approved Customer
Knowledgebase Solution Question: Question: Question: Question: Question: Question: Question: Environment: Environment: Environment: Environment: Environment: Environment: Problem: Problem: Change: Gathering Navisphere Analyzer .NAR files and configuration information for a CLARiiON performance case Information and data files that must be obtained for a Performance Analysis on all types of CLARiiON How to gather the necessary information for a CLARiiON performance analysis What logs do I collect for CLARiiON Performance issues? What logs to collect for CLARiiON performance troubleshooting? What data to collect for CLARiiON performance troubleshooting? What do I need to submit a performance problem? Product: CLARiiON Product: VNX Family EMC SW: Navisphere Analyzer EMC SW: Unisphere Analyzer Product: Celerra NS Network Server Product: EMC Disk Library (EDL) Poor read and write performance and high response times for connected hosts. Performance problems on a CLARiiON array configuration requiring performance analysis Response times for host applications have increased or are unacceptably high. Performance Service Requests will not be fully investigated until both the SPCollects and NAR (or NAZ) files are available and a detailed description of the symptoms has been provided. The SPCollects have the configuration and fault information for the CLARiiON and the NAR files have the performance data. The detail of the performance issue symptoms is also needed so that the engineer will know which period needs to be analyzed in detail and the LUNs that had the noticeable performance impact. Without this detail, only the general performance statistics will be analyzed to look at issues affecting all LUNs. Customer Support Services investigate performance cases to identify issues that are affecting customer sites. Performance tuning (to achieve peak performance for a host or application) and reconfiguration is a task for a Storage Administrator or a Solutions Architect so this will not be handled in the Service Request. Performance issues can have a number of causes including hardware faults, forced flushing (caused by write cache being 100% full), head contention (several busy LUNs sharing the same drives), or the storage array being at its maximum utilization. For each event, Technical Support needs to know the following, in the description of the performance problem: The time (with the time zone GMT/UTC difference) and date. The server names or CLARiiON Storage Groups that own the LUNs. The LUNs that the affected application was using at that time. Is it reading or writing performance that is suffering? Obtain the basic diagnostic files that must cover the period when the problems occur: SPcollects for both Storage Processors, which were run after the problem occurred (so that their logs would include the time of the incident). See Navisphere (or Unisphere) Help article "Generating and transferring SP log files" or EMC Knowledgebase solution emc60493 for instructions on how to obtain SPcollects. Navisphere (or Unisphere) Analyzer .NAR or .NAZ files, which cover the whole period when the problem was occurring. If the problem occurs throughout the day, include NAR / NAZ files which cover a total period of at least 24 hours:
Root Cause:
http://knowledgebase.emc.com/emcice/resultDisplay2.do?result=-1&clusterName=DefaultCluster&page=&docType=1006&groupid... 7/27/2011
How to gather the necessary information for a CLARiiON performance analysis
NAR and NAZ files contain the same data, but only NAR files can be opened in Navisphere / Unisphere Analyzer. If the CLARiiON has an Analyzer license ('-NavisphereAnalyzer' in the software list) then NAR files will be produced; otherwise data logging produces NAZ files. NAZ files are for use by EMC support only, to assist with break/fix performance diagnostics. Before starting the data logging, the archive interval should be set appropriately: A typical NAR (or NAZ) file will contain 155 polling intervals of data. Therefore, if the archive polling interval was 120 seconds, there would be 5 hours and 10 minutes of data in each file. Once each file is full, it will be closed and a new one started. These can later be merged together, in order to produce graphs covering a longer period. To change the archive interval in Navisphere, go to the Data Logging window, in the Navisphere Analyzer menu. To change the archive interval in Unisphere, highlight the Monitoring Tab and select Analyzer. Click on the 'Performance Data Logging' link. To change the archive interval using the CLI, use the following command: naviseccli ~ analyzer -set -narinterval 120 (where ~ would be where the arguments -h <IP address>, -user <username> etc. may need to included) The archive interval should be set to 120 or 60 seconds, in order the produce a detailed record of performance. However, large configurations may take longer than 60 or even 120 seconds to poll, in which case the interval would need to be set to a higher interval, such as 240 seconds. The results recorded will be averages of the performance over archive intervals. Therefore if the archive polling interval was set at 600 seconds, performance problems which only lasted a few minutes, would be hard to diagnose For Flare release 22 and earlier, you will need to be in Engineering Mode to change the Archive Interval (see emc255033) To start data logging: In Navisphere Manager, select Tools -> Analyzer -> Data Logging. In Unisphere, highlight the Monitoring Tab and select Analyzer. Click on the 'Performance Data Logging' link. Make sure the correct CLARiiON is selected (in the 'Target' box), then click on start. If the archive interval needs changing, click on the stop button, change the value, then click on the start button (this is for Flare release 24 and higher, for Flare release 22 and earlier, you start analyzer by enabling "Statistics Logging") Alternatively use the following CLI command: naviseccli ~ analyzer -start (where ~ would include the SP IP address and the Navisphere login) A NAR (or NAZ) file will not be available to download while it is still open for data logging. Therefore the list of NAR files will only include the files which have been completed (which normally happens when there are 155 polling intervals contained in them), not the latest file. To obtain the latest data, the data logging can be stopped and then started again, which will close the current NAR file and then start a new one. Alternatively the following CLI command will start a new NAR file and make the latest NAR file available to download (although you may need to wait for a minute or so for it to be closed): naviseccli ~ analyzer -archive -new (where ~ would include the SP IP address and the Navisphere login) To retrieve the completed archives (after they have had time to gather sufficient information and the performance issues were occurring during this period): In Unisphere (Release 30 and above): Click on the CLARiiON which is to be monitored from the Unisphere Dashboard. Highlight the 'Monitoring' box and a drop-down menu will appear. From this drop-down menu, select Analyzer. Select SP A. Select the relevant archives and click Retrieve. Repeat for SP B. In Navisphere Manager (Release 24 to 29): Select Tools -> Analyzer -> Archive -> Retrieve Make sure the correct CLARiiON is selected. Select SP A. Select the relevant archives and click Retrieve. Repeat for SP B. For Flare release 24 and higher, the archive file which is currently being logged to will not be shown. To close the current file and start and new one, stop and then start the data logging (allow each file to run for at least ten times the logging period). Alternatively use the following CLI command to retrieve all the archives from one SP: naviseccli -h <SP IP> analyzer -archive -all If you have remote access to the SP, via EMC Remote or Remotely Anywhere, then the NAR/NAZ files can be transferred directly from C:\EMC\Archives folder. (Flare release 24 and higher) See article emc255033 for further information on using Navisphere / Unisphere to collect NAR/NAZ files. For Release 22 and below, see article emc227465. Do not use the 'naviseccli ~ analyzer -archiveretrieve' in Release 24 or higher. Instead use
Page 2 of 4
Fix:
http://knowledgebase.emc.com/emcice/resultDisplay2.do?result=-1&clusterName=DefaultCluster&page=&docType=1006&groupid... 7/27/2011
How to gather the necessary information for a CLARiiON performance analysis
'naviseccli ~ analyzer -archive <-all or -file filenames> - see emc240526 The NAR/NAZ file title contains the date and time (in GMT) of the last data point logged. It is important to stress that the Navisphere files have to be for the period when the problems were actually occurring. Upload files to Powerlink in the same form that they were produced on the SP. Do not merge NAR files and do not decrypt NAZ files. Any changes to the files can introduce errors, so it is important that the original files are supplied. A guide to running Navisphere Analyzer can be found by going to 'Help' in Navisphere. Select 'Help topics', followed by 'Analyzing storage-system performance using Analyzer' A guide to using Naviseccli for retrieving NAR files can be found on Powerlink, titled: 'EMC® Navisphere® Analyzer Command Line Interface (CLI)'. Host Grab (or EMC Report) output files from the affected hosts. Select this link to Powerlink and then follow this path to the emcgrab/EMCREPORTS host utility downloads: Home > Support > Product and Diagnostic Tools > Grab Utilities. Select Next, until you find your operating system utility. Download the utility and the README file. Follow the README file to run the utility for the host operating system chosen. Alternatively, these grabs can be downloaded from ftp://ftp.emc.com/pub/emcgrab/ Switch logs may be needed, especially if the performance problem only exists on certain paths. Please refer to the following EMC knowledgebase articles based on the model switch being used: McDATA: emc92923 or emc86536 Brocade: emc42373 Cisco: emc97851 or emc102214 Release 24 and later stores the old archives but encrypts them (.NAZ) if the latest enabler is not present. Therefore, Release 24 does not require an enabler to gather performance information. Arrays running any Release of FLARE from 14 to 22 will need a Navisphere Analyzer Enabler loaded. The enabler will appear as the following in Navisphere (right-clicking the CLARiiON array -> Properties -> Software): -NavisphereAnalyzer For an intermittent problem in FLARE pre-Release 24, it is possible to script the archive retrieval to store the archives. (See solution emc148982.) If no recent NAR or NAZ files have been produced and the CLARiiON is running Release 24 or higher core software, then the data logging has not been started. To start Data Logging:
Page 3 of 4
Notes:
2. 3. 4.
Go to the Analyzer menu bar in Navisphere and select Data Logging. Check that Periodic Archiving is enabled, and if not enable it. (This will create files every few hours, rather than putting all the data into a single file.) Click Start to begin the data logging. By Default this will run for 7 days before automatically stopping again. (This is the "Stop Automatically After" parameter.)
Notes:
Alternatively, the performance logging can be prepared with the following naviseccli commands for both SP:
2. 3. 4. 5.
naviseccli -h <SP A IP> setstats -on naviseccli -h <SP B IP> setstats -on naviseccli -h <SP A IP> analyzer -set -default naviseccli -h <SP A IP> analyzer -set -periodicarchiving 1
The logging can then be started on both SP with the following command, and the logging will keep running for 7 days (unless it is manually stopped again):
naviseccli -h <SP A IP> analyzer -start CLARiiON Support has a break/fix charter and we approach each performance case from that perspective. If we find that the hardware and software is operating as designed (health check) and we can find nothing broken, we will still take the next step and attempt to identify the cause. So while we can check over the logs to look for evidence of a fault or mis-configuration, we do not produce detailed performance reports in these cases. If the cause is determined to be configuration related (such as spindle overload or an under provisioned raid group), we will attempt to identify what the possible causes are, but making configuration recommendations or performance tuning would be outside of our charter. In this case, we will contact your local Account Service Representative (ASR) and copy your EMC District Service Manager to let them know that this particular issue may require additional assistance. If you require help with resolving the configuration issue(s), then Professional Services can be engaged.
Notes:
http://knowledgebase.emc.com/emcice/resultDisplay2.do?result=-1&clusterName=DefaultCluster&page=&docType=1006&groupid... 7/27/2011
How to gather the necessary information for a CLARiiON performance analysis
Page 4 of 4
http://knowledgebase.emc.com/emcice/resultDisplay2.do?result=-1&clusterName=DefaultCluster&page=&docType=1006&groupid... 7/27/2011
Page 1 of 4
EMC Knowledgebase
"How to gather the necessary information for a CLARiiON performance analysis"
ID: Usage: Date Created: Last Modified: STATUS: Audience: emc161922 1416 05/24/2007 05/12/2011 Approved Customer
Knowledgebase Solution Question: Question: Question: Question: Question: Question: Question: Environment: Environment: Environment: Environment: Environment: Environment: Problem: Problem: Change: Gathering Navisphere Analyzer .NAR files and configuration information for a CLARiiON performance case Information and data files that must be obtained for a Performance Analysis on all types of CLARiiON How to gather the necessary information for a CLARiiON performance analysis What logs do I collect for CLARiiON Performance issues? What logs to collect for CLARiiON performance troubleshooting? What data to collect for CLARiiON performance troubleshooting? What do I need to submit a performance problem? Product: CLARiiON Product: VNX Family EMC SW: Navisphere Analyzer EMC SW: Unisphere Analyzer Product: Celerra NS Network Server Product: EMC Disk Library (EDL) Poor read and write performance and high response times for connected hosts. Performance problems on a CLARiiON array configuration requiring performance analysis Response times for host applications have increased or are unacceptably high. Performance Service Requests will not be fully investigated until both the SPCollects and NAR (or NAZ) files are available and a detailed description of the symptoms has been provided. The SPCollects have the configuration and fault information for the CLARiiON and the NAR files have the performance data. The detail of the performance issue symptoms is also needed so that the engineer will know which period needs to be analyzed in detail and the LUNs that had the noticeable performance impact. Without this detail, only the general performance statistics will be analyzed to look at issues affecting all LUNs. Customer Support Services investigate performance cases to identify issues that are affecting customer sites. Performance tuning (to achieve peak performance for a host or application) and reconfiguration is a task for a Storage Administrator or a Solutions Architect so this will not be handled in the Service Request. Performance issues can have a number of causes including hardware faults, forced flushing (caused by write cache being 100% full), head contention (several busy LUNs sharing the same drives), or the storage array being at its maximum utilization. For each event, Technical Support needs to know the following, in the description of the performance problem: The time (with the time zone GMT/UTC difference) and date. The server names or CLARiiON Storage Groups that own the LUNs. The LUNs that the affected application was using at that time. Is it reading or writing performance that is suffering? Obtain the basic diagnostic files that must cover the period when the problems occur: SPcollects for both Storage Processors, which were run after the problem occurred (so that their logs would include the time of the incident). See Navisphere (or Unisphere) Help article "Generating and transferring SP log files" or EMC Knowledgebase solution emc60493 for instructions on how to obtain SPcollects. Navisphere (or Unisphere) Analyzer .NAR or .NAZ files, which cover the whole period when the problem was occurring. If the problem occurs throughout the day, include NAR / NAZ files which cover a total period of at least 24 hours:
Root Cause:
http://knowledgebase.emc.com/emcice/resultDisplay2.do?result=-1&clusterName=DefaultCluster&page=&docType=1006&groupid... 7/27/2011
How to gather the necessary information for a CLARiiON performance analysis
NAR and NAZ files contain the same data, but only NAR files can be opened in Navisphere / Unisphere Analyzer. If the CLARiiON has an Analyzer license ('-NavisphereAnalyzer' in the software list) then NAR files will be produced; otherwise data logging produces NAZ files. NAZ files are for use by EMC support only, to assist with break/fix performance diagnostics. Before starting the data logging, the archive interval should be set appropriately: A typical NAR (or NAZ) file will contain 155 polling intervals of data. Therefore, if the archive polling interval was 120 seconds, there would be 5 hours and 10 minutes of data in each file. Once each file is full, it will be closed and a new one started. These can later be merged together, in order to produce graphs covering a longer period. To change the archive interval in Navisphere, go to the Data Logging window, in the Navisphere Analyzer menu. To change the archive interval in Unisphere, highlight the Monitoring Tab and select Analyzer. Click on the 'Performance Data Logging' link. To change the archive interval using the CLI, use the following command: naviseccli ~ analyzer -set -narinterval 120 (where ~ would be where the arguments -h <IP address>, -user <username> etc. may need to included) The archive interval should be set to 120 or 60 seconds, in order the produce a detailed record of performance. However, large configurations may take longer than 60 or even 120 seconds to poll, in which case the interval would need to be set to a higher interval, such as 240 seconds. The results recorded will be averages of the performance over archive intervals. Therefore if the archive polling interval was set at 600 seconds, performance problems which only lasted a few minutes, would be hard to diagnose For Flare release 22 and earlier, you will need to be in Engineering Mode to change the Archive Interval (see emc255033) To start data logging: In Navisphere Manager, select Tools -> Analyzer -> Data Logging. In Unisphere, highlight the Monitoring Tab and select Analyzer. Click on the 'Performance Data Logging' link. Make sure the correct CLARiiON is selected (in the 'Target' box), then click on start. If the archive interval needs changing, click on the stop button, change the value, then click on the start button (this is for Flare release 24 and higher, for Flare release 22 and earlier, you start analyzer by enabling "Statistics Logging") Alternatively use the following CLI command: naviseccli ~ analyzer -start (where ~ would include the SP IP address and the Navisphere login) A NAR (or NAZ) file will not be available to download while it is still open for data logging. Therefore the list of NAR files will only include the files which have been completed (which normally happens when there are 155 polling intervals contained in them), not the latest file. To obtain the latest data, the data logging can be stopped and then started again, which will close the current NAR file and then start a new one. Alternatively the following CLI command will start a new NAR file and make the latest NAR file available to download (although you may need to wait for a minute or so for it to be closed): naviseccli ~ analyzer -archive -new (where ~ would include the SP IP address and the Navisphere login) To retrieve the completed archives (after they have had time to gather sufficient information and the performance issues were occurring during this period): In Unisphere (Release 30 and above): Click on the CLARiiON which is to be monitored from the Unisphere Dashboard. Highlight the 'Monitoring' box and a drop-down menu will appear. From this drop-down menu, select Analyzer. Select SP A. Select the relevant archives and click Retrieve. Repeat for SP B. In Navisphere Manager (Release 24 to 29): Select Tools -> Analyzer -> Archive -> Retrieve Make sure the correct CLARiiON is selected. Select SP A. Select the relevant archives and click Retrieve. Repeat for SP B. For Flare release 24 and higher, the archive file which is currently being logged to will not be shown. To close the current file and start and new one, stop and then start the data logging (allow each file to run for at least ten times the logging period). Alternatively use the following CLI command to retrieve all the archives from one SP: naviseccli -h <SP IP> analyzer -archive -all If you have remote access to the SP, via EMC Remote or Remotely Anywhere, then the NAR/NAZ files can be transferred directly from C:\EMC\Archives folder. (Flare release 24 and higher) See article emc255033 for further information on using Navisphere / Unisphere to collect NAR/NAZ files. For Release 22 and below, see article emc227465. Do not use the 'naviseccli ~ analyzer -archiveretrieve' in Release 24 or higher. Instead use
Page 2 of 4
Fix:
http://knowledgebase.emc.com/emcice/resultDisplay2.do?result=-1&clusterName=DefaultCluster&page=&docType=1006&groupid... 7/27/2011
How to gather the necessary information for a CLARiiON performance analysis
'naviseccli ~ analyzer -archive <-all or -file filenames> - see emc240526 The NAR/NAZ file title contains the date and time (in GMT) of the last data point logged. It is important to stress that the Navisphere files have to be for the period when the problems were actually occurring. Upload files to Powerlink in the same form that they were produced on the SP. Do not merge NAR files and do not decrypt NAZ files. Any changes to the files can introduce errors, so it is important that the original files are supplied. A guide to running Navisphere Analyzer can be found by going to 'Help' in Navisphere. Select 'Help topics', followed by 'Analyzing storage-system performance using Analyzer' A guide to using Naviseccli for retrieving NAR files can be found on Powerlink, titled: 'EMC® Navisphere® Analyzer Command Line Interface (CLI)'. Host Grab (or EMC Report) output files from the affected hosts. Select this link to Powerlink and then follow this path to the emcgrab/EMCREPORTS host utility downloads: Home > Support > Product and Diagnostic Tools > Grab Utilities. Select Next, until you find your operating system utility. Download the utility and the README file. Follow the README file to run the utility for the host operating system chosen. Alternatively, these grabs can be downloaded from ftp://ftp.emc.com/pub/emcgrab/ Switch logs may be needed, especially if the performance problem only exists on certain paths. Please refer to the following EMC knowledgebase articles based on the model switch being used: McDATA: emc92923 or emc86536 Brocade: emc42373 Cisco: emc97851 or emc102214 Release 24 and later stores the old archives but encrypts them (.NAZ) if the latest enabler is not present. Therefore, Release 24 does not require an enabler to gather performance information. Arrays running any Release of FLARE from 14 to 22 will need a Navisphere Analyzer Enabler loaded. The enabler will appear as the following in Navisphere (right-clicking the CLARiiON array -> Properties -> Software): -NavisphereAnalyzer For an intermittent problem in FLARE pre-Release 24, it is possible to script the archive retrieval to store the archives. (See solution emc148982.) If no recent NAR or NAZ files have been produced and the CLARiiON is running Release 24 or higher core software, then the data logging has not been started. To start Data Logging:
Page 3 of 4
Notes:
2. 3. 4.
Go to the Analyzer menu bar in Navisphere and select Data Logging. Check that Periodic Archiving is enabled, and if not enable it. (This will create files every few hours, rather than putting all the data into a single file.) Click Start to begin the data logging. By Default this will run for 7 days before automatically stopping again. (This is the "Stop Automatically After" parameter.)
Notes:
Alternatively, the performance logging can be prepared with the following naviseccli commands for both SP:
2. 3. 4. 5.
naviseccli -h <SP A IP> setstats -on naviseccli -h <SP B IP> setstats -on naviseccli -h <SP A IP> analyzer -set -default naviseccli -h <SP A IP> analyzer -set -periodicarchiving 1
The logging can then be started on both SP with the following command, and the logging will keep running for 7 days (unless it is manually stopped again):
naviseccli -h <SP A IP> analyzer -start CLARiiON Support has a break/fix charter and we approach each performance case from that perspective. If we find that the hardware and software is operating as designed (health check) and we can find nothing broken, we will still take the next step and attempt to identify the cause. So while we can check over the logs to look for evidence of a fault or mis-configuration, we do not produce detailed performance reports in these cases. If the cause is determined to be configuration related (such as spindle overload or an under provisioned raid group), we will attempt to identify what the possible causes are, but making configuration recommendations or performance tuning would be outside of our charter. In this case, we will contact your local Account Service Representative (ASR) and copy your EMC District Service Manager to let them know that this particular issue may require additional assistance. If you require help with resolving the configuration issue(s), then Professional Services can be engaged.
Notes:
http://knowledgebase.emc.com/emcice/resultDisplay2.do?result=-1&clusterName=DefaultCluster&page=&docType=1006&groupid... 7/27/2011
How to gather the necessary information for a CLARiiON performance analysis
Page 4 of 4
http://knowledgebase.emc.com/emcice/resultDisplay2.do?result=-1&clusterName=DefaultCluster&page=&docType=1006&groupid... 7/27/2011
