Archive

Author Archive

Warning “Valid SP-Cache is for Storage Processor SP…” event code 0x7241

February 15th, 2017 No comments

On a VNX 5300 array, after powering down, unseating, and re-seating a SP I encountered the event code 0x7241 with warning message “Valid SP-Cache is for Storage Processor SP…”.

Disabling write and read caches on each SP did not resolve.  Restarting management services on the SPs did not resolve.

To resolve, I restarted one SP within the Unisphere GUI, then after it was back up, I restarted the other SP.

 

 

 

 

Categories: EMC VNX Tags:

VMWare SRM – When trying to protect a VM – There are not enough licenses installed to perform the operation

April 6th, 2016 No comments

So there were plenty of licenses, what else can be wrong?

 

Within the log files @ C:\ProgramData\VMware\VMware vCenter Site Recovery Manager\Logs I saw the following:

2016-04-06T15:38:31.479-05:00 [09472 warning ‘Licensing’] Unable to decode license ”: INVALID_SERIAL
2016-04-06T15:38:31.480-05:00 [07916 info ‘Licensing’] Initializing with license key:
2016-04-06T15:38:31.480-05:00 [07916 verbose ‘PropertyProvider’] RecordOp ASSIGN: asset, DrLicenseManager
2016-04-06T15:38:31.480-05:00 [07916 warning ‘Licensing’] The license key ” expired on 1970-01-01T00:00:00Z
2016-04-06T15:38:31.481-05:00 [09240 warning ‘Licensing’] This SRM instance is no longer in compliance. 41 4(s) are not licensed for protection.

In the web client under home > licensing > solutions I found something that didn’t exactly refer to SRM but I assigned the SRM key to it anyways.  After assigning this key the problem was resolved.

Categories: Uncategorized Tags:

Return the X-Frame-Options HTTP header in IIS 7 for Exchange OWA

December 18th, 2015 No comments

To prevent click-jacking, add the HTTP response header “X-Frame-Options” into IIS for websites and or Exchange OWA:

– Open IIS Manager and click on the server name in the left column.  Drill down if you only want to apply to one website.
– In Features View, double-click HTTP Response Headers.
– On the HTTP Response Headers page, in the Actions pane, click Add.
– In the Add Custom HTTP Response Header dialog box, add a header called “X-FRAME-OPTIONS”, and assign it’s value to “SAMEORIGIN”.
– Click OK

 

You can validate correct function by visiting one of these websites:

https://securityheaders.io
http://web-sniffer.net/

Categories: Uncategorized Tags:

Server Connection: Not Connected to SRM server

December 17th, 2015 3 comments

I was unable to find this problem documented anywhere, though there was a reference to it on another blog here:   http://www.virtualmachinery.co.uk/2015/03/upgrading-site-recovery-manager-55-to.html

 

The problem presents itself this way, looking at SRM in the web client in version 5.5 of VMWare, 5.8.1 of SRM:

Capture

As you can see, client connection shows as connected, however server connection shows as “Not Connected to SRM server”.  It wasn’t obvious to me, but what this means is the sites are not connecting to each other, even though they are paired and everything else looks green.

Additionally you will notice that the option to replicate changes to the secondary site before failover will be grayed out.

I spent several days troubleshooting this before I found an indicator in the logs that pointed to certificate errors.  I believe that if I was able to un-pair and then re-pair the sites, this would have been resolved.  However in order to un pair sites, you must first delete the recovery plans and protection groups.  When attempting to delete, the status would say deleting and never complete.

Ultimately to resolve I uninstalled SRM at both sites, deleting all data from database.  I then reinstalled and reconfigured SRM, protection groups, and recovery plans.

 

Categories: Uncategorized Tags:

Recommended extensions to block @ spam filter

December 10th, 2015 No comments

*.ade
*.adp
*.arj
*.asx
*.bas
*.bat
*.cab
*.chm
*.cmd
*.com
*.cpl
*.crt
*.exe
*.hlp
*.hta
*.inf
*.ins
*.jar
*.js
*.jse
*.jsp
*.lib
*.lnk
*.mdb
*.mde
*.msi
*.msp
*.nsc
*.pcd
*.pif
*.pptm
*.ps1
*.reg
*.rwa
*.scr
*.sct
*.shs
*.vb
*.vbe
*.vbs
*.wmd
*.wsc
*.wsf
*.wsh

Additionally you may consider scanning these closer, quarantining, or blocking:
*.rar (block any that are encrypted/can not be scanned)
*.zip (block any that are encrypted/can not be scanned)
*.pdf (block any that are encrypted/can not be scanned)
*.xlsm (macro enabled xls)
*.docm (macro enabled docs)
*.doc (block any that are macro enabled if possible)

Categories: Uncategorized Tags:

Phishing test providers I recommend

August 6th, 2015 No comments
  • phishingbox
  • threatsim
  • wombat security
  • knowbe4

If training isn’t important to you, go with phishingbox. They are the cheapest.

If training is important to you, go with ThreatSim or Wombat Security.

I find knowbe4’s training materials to be meh, but that may just be me.

My personal recommendation is ThreatSim. Their training is lagging behind, but their support is beyond excellent. I suspect they will become a major player in a year from now.

Edit: ThreatSim has been acquired by Wombat Security – this will likely increase the cost of ThreatSim in 2016

Categories: Uncategorized Tags:

MS14-025/KB2928120: An Update for Group Policy Preferences

May 15th, 2014 No comments

Looking at this article: http://blogs.technet.com/b/srd/archive/2014/05/13/ms14-025-an-update-for-group-policy-preferences.aspx

I grabbed the check script from here (bottom of the page) http://support.microsoft.com/kb/2962486

and ran it on my domain controller.  The script immediately gave me the error “cannot bind to argument to parameter ‘path’ because it is null”.

Apparently this is an uncaught exception when no XML files exist in the path subfolders.  It appears that ONLY group policy preferences are stored in XML, and this XML file will only show up if group policy preferences are implemented, meaning if you don’t have XML files in %windir%\SYSVOL\domain then you are not affected by this patch.  Group policies themselves appear to be stored as INF and other types.

 

Categories: Uncategorized Tags:

Bringing a single domain controller up in an isolated network

May 14th, 2014 No comments

 

I wanted to create a quick test lab so I spun up a copy of a virtualized domain controller into an isolated network. The domain controller came up in a failed state with DNS and Active Directory non-functional.

Apparently in a multi domain controller network it is a requirement that the domain controller be able to sync with other domain controllers/role masters in order to function.

Because this was the only domain controller in the network, and I wanted to get the test network up quickly, I performed the following workarounds:

 

(Thanks to user zabo2012 on the veeam forums at http://forums.veeam.com/vmware-vsphere-f24/restoring-2012-domain-controller-vm-t18629.html for the awesome instructions)

 

boot the machine up in dsrm ( bcdedit /set safeboot dsrepair )

log in with ds repair mode password .\Administrator

run the bcdedit command to set and remove dsrepair mode ( bcdedit /deletevalue safeboot )

net stop ntfrs

open regedit and

Open Regedit
Browse to the following extension: HKLM\SYSTEM\CurrentControlSet\Services\NTDS\Parameters
Add the following dword (32 bit) value: Repl Perform Initial Synchronizations
And leave this set to 0.
http://www.veeam.com/kb_articles.html/kb1280
then

open regedit and expand: hklm\SYSTEM\CurrentControlSet\Services\NtFrs\Parameters\Backup/Restore\Process at Startup
Set the burflags to d2 (sometimes you will have to use d4, but only do this in isolated network or it will overwrite other DC’s during replication)
http://www.veeam.com/kb_articles.html/kb1278

reboot

Edit:

I noticed that although I was able to get other servers to authenticate off the DC after doing the above, I wasn’t able to access AD Users and Computers on the DC itself.

Seizing the roles from the other DCs (that are not available in the isolated test lab) fixed this.  To seize the other domain controller FSMO roles:

ntdsutil
roles
connections
connect to server <dns name of local dc server>
quit

seize schema master
seize naming master
seize rid master
seize PDC
seize infrastructure master

quit
quit

After seizing roles I now see the expected information in AD Users and Computers

Edit 2:

I continued to have problems with an Exchange server that was in the same test lab as the isolated domain controller so I made a few more changes:

I performed a metadata cleanup, removing all the domain controllers that were not in the isolated lab environment, using the GUI > http://technet.microsoft.com/en-us/library/cc816907%28v=ws.10%29.aspx#bkmk_graphical

I then set the burflag to d4 (below) and restarted the domain controller.  After that exchange was working correctly.

open regedit and expand: hklm\SYSTEM\CurrentControlSet\Services\NtFrs\Parameters\Backup/Restore\Process at Startup
Set the burflags to d4
http://www.veeam.com/kb_articles.html/kb1278

 

 

Categories: Uncategorized Tags:

Continuous offline archiving of EMC VNX array performance data

May 8th, 2014 No comments

 

The EMC VNX arrays do not offer a good (inexpensive) way to archive performance data continuously to a management server for future retrieval.  You CAN turn on performance data logging and have it periodically archive to the array itself, but I prefer not to have multiple GB of archived performance data on the same array I may be troubleshooting in the future, not to mention that is one more item to review on my maintenance checklist.

 

Turning on performance data logging

First thing, to enable performance monitoring and generate NAR files check (and uncheck) the following options in Unisphere > System > Monitoring & Alerts > Statistics > Performance Data Logging:

 

vnx_perf_data_archiving_screenshot_1

 

Next click start to start performance logging.  Re-verify you have unchecked the “stop automatically after” option.  The array will periodically archive performance data to .nar files on the array itself.  In my environment the array archives to a nar file about once every 12 hours for each storage processor.  You may force the array to archive to a .nar file by stopping then starting the data logging.

Note: In order to review NAR files after they are generated you must have the Unisphere Analyzer enabler installed on the array, otherwise you will have to engage EMC support to review the performance logs for you.

 

Retrieving performance logs from the array and archiving to a server

Install naviseccli on a server, then edit the below vbscript code, entering your own values for the IP addresses of the SPs, user, password, and file path.  Create a scheduled task that executes cscript.exe against the vbscript code on the server on a daily basis.  The script places a call to each SP, stores all NAR files on that SP to the directory of your choosing, then deletes all the NAR files from that SP.

 


'grab perf logs from array then delete logs off array

Set objShell = WScript.CreateObject("WScript.Shell")
Set objExecObject = objShell.Exec("cmd /c naviseccli -Address <ip of SP_A> -User <san username> -Password <san password> -Scope 0 analyzer -archive -all -o -path <folder path (ex: C:\EMC\data_archive)>
")
WScript.Sleep 60000
Set objExecObject = objShell.Exec("cmd /c naviseccli -Address <ip of SP_B> -User <san username> -Password <san password> -Scope 0 analyzer -archive -all -o -path <folder path (ex: C:\EMC\data_archive)>
WScript.Sleep 60000
Set objExecObject = objShell.Exec("cmd /c naviseccli -Address <ip of SP_A> -User <san username> -Password <san password> -Scope 0 analyzer -archive -delete -all -o")
WScript.Sleep 60000
Set objExecObject = objShell.Exec("cmd /c naviseccli -Address <ip of SP_B> -User <san username> -Password <san password> -Scope 0 analyzer -archive -delete -all -o")

 

Now you have continuously archived data from your array that you can now open in Unisphere Analyzer to review array performance.

 

 

 

Categories: EMC VNX Tags:

Proactive copy to hot spare on EMC VNX array

March 5th, 2014 1 comment

 

 

SAN errors, oh no!

 

The process started when the array emailed me a couple of soft media errors, so I glanced at the SPA and SPB event logs in Navisphere and saw this:

2014.3.5_initial_errors

Notice that the majority of errors showed up as informational and not warning or critical, meaning the array will not indicate to anyone that this drive is about to fail, yeesh.

Note also that all the errors occur on the same disk, Bus 0 Enclosure 1 Disk 7, a NLSAS 2TB drive in my environment.

Event codes included 0x6a0, 0x820, and 0x801 with descriptions of disk soft media error, soft scsi bus error, and soft media error.  My suggestion is to filter only on description, and do searches for “error” to find all the messages.

Reviewing the disks within the navisphere GUI showed that no disks were faulted, the dashboard showed no errors, and the hot spare was not in use, meaning the array did not believe the drive should be failed yet.


Time to be proactive…

 

I opened a case with EMC support, and sent them screenshots of the SP event logs, and SPCollects from both SPA and SPB, and noted that the error occurred on the same drive over 100 times in one day.  The support representative immediately requested I do a proactive copy to the hot spare disk, and ordered a replacement disk sent to me.

A proactive copy is preferred because instead of requiring the array to rebuild the RAID array to a new disk (and endure the performance degradation inherent in this procedure), it copies the data from one disk to another, then tells the RAID array to use the hot spare disk, then disables the failing disk, skipping the rebuild process altogether and hence no RAID rebuild performance degradation.

I first tried to do the proactive copy from the navisphere gui without success, below.

2014.3.5_proactivecopy_greyedout

Note the option to copy is greyed out.  Apparently VNXs new mixed RAID storage pools prevents this option from being used, so I moved onto the CLI.

Passing the command
naviseccli -Address <SP IP address> -User <user> -Password <password> -Scope 0 copytohotspare 0_1_7 -initiate
(where 0_1_7 is the failing disk) worked correctly, starting the proactive copy from the failing disk to the hotspare of the same type.

 

 

Checking progress of the proactive copy

 

Now to check progress of the copy…

First I tried looking at the disks in the GUI

2014.3.5_noprogress_gui_1

The disk state is listed as “Copying to Hot Spare(100%)”.  Hmm, 100% doesn’t seem right, I just started this procedure.  (Looking at the RAID LUN within the GUI showed the state as “transitioning” without any progress indicators)

Then I tried through the CLI

2014.3.5_noprogress_cli

Well that also doesn’t look right, either.  I continued on looking at SPCollect logs, SP event logs, and looking all over the Internet, including the EMC community forums, without finding any answers.  (running getlun on the RAID LUN from the CLI didn’t show any progress indicators also)

 

Update: I figured out a way to gauge progress, though a bit crude.  See the update at the bottom.

 

Eventually after 19 hours (NLSAS 2TB) the process completed, throwing event logs 0x6b0, 0x604, 0x67d, 0x6a8, 0x67c, 0x7a7, 0x6ab0 along with many others indicating it had marked the failing drive as failed as expected.

2014.3.5_proactivecopy_completed

Complete list of event logs thrown as part of the proactive copy completion:
0x6b0, 0x712d4601, 0x906, 0x7a7, 0x608, 0x6a1, 0x602, 0x7a5, 0x6a8, 0x712789a0, 0x67b, 0x67c, 0x603, 0x602, 0x712d0508, 0x604, 0x712d0507, 0x2580, 0x906, 0x7a6, 0x799, 0x712d4602, 0x712d4601, 0x67d,  0x7400, 0x740a, 0x2580and probably some others I missed.

 

At this point I gave the EMC CE a call and scheduled replacement of the drive.  Once the replacement drive is in place, the array should copy the information on the hot spare back to the replacement drive, then mark the hot spare drive as available again.


Update
: It appears a progress indicator of sorts is included in the lustat command in the SPCollect logs.  By running SPCollects over and over again I can gauge the progress of proactive copies and equalizations when the drive is replaced.

progress_indicator_found

This information is contained within the SPCollect zip file, within the *_sus zip file, within the SPx_cfg_info.txt file.  By looking at the EQZ percentage, I can gauge roughly when it will finish, and more importantly that the equalization or proactive copy is progressing.


Update
: I came across a new EMC article today that may have a command that will work to gauge progress better.  I have not attempted it myself yet.

From:
https://community.emc.com/docs/DOC-7962

 

emc copy to hotspare progress

 

 

Categories: EMC VNX Tags: