Hey all - I'm geting Disk Performance Degraded warnings on our ISIS 5500. It keeps referring to DISK 1. I'd like to replace the ISB but none of the LED indicator lights are red so I'm not sure which one to replace.

A couple questions:

1) Is DISK 1 referring to the ISB in the very top left of the 16 ISB's?

2) What is the procedure for replacing? Simply pull out the defective ISB and insert the spare? Do I need to shut down the ISIS before replacing or leave everything running?

See System Logs:

Hi Kyle.

The disk 1 is related to 1 of the 3 raid sets inside the isis. Not to an ISB with a microcontroller with its own IP as found in isis 7000.

What you need to do is to open the LSI (now called Avago) megaraid storage manager utility on the isis. It should be installed by default but you can also dowload it from Avid here:

http://avid.force.com/pkb/articles/en_US/faq/MegaRaid-software-for-the-following-products 

In there you will find all 16 drives and you need to find if and how many drives show an elevated media error count.

Healthy drives show 0 and from experience seeing a drive with less than 10 is not worrying. When you see the degration you mention, you will find 1 or more drives with a much higher media error count, like >100 or even >1000. Although not failed yet you should replace these drives with new drives from Avid.

Thanks for the reply! I've installed the MSM software and have identified Slot 8 as the failing driving with a media error count of 193.

I know ISIS 5500 is mostly an automated process - would you recommend waiting for the ISB to fail, and let the ISIS redistrubute to the spare, then replace? or swap it out before then? And if swapping immediately - can this be done while the ISIS is running?

I have a new spare ready, I'm just trying to get this process firmly understood before pulling out the ISB.

I would only force failover if it is actually causing you issues. Often "network degraded" can be safely ignored.

If users are having performance or stability issues (bin not saving, crashing, etc.) then you can pull the dying drive and force drive recovery to the hot spare.

Make sure you confirm it is the correct drive. Check the serial number before you pull it.

Thanks Dom,

We weren't experiencing performance issues until just today. I'm trying to transcode AMA linked XAVC FS7 Media, a common procedure, and I keep getting the following error message:

Exception: DIDReader::XraiseUnexpectedStreamStatus async file system error.

This also happens when trying to transcode AMA linked XDCAMEX media.

Do you think this is a sign to force failover?

SOLVED: I've replaced the degraded ISB. What follows are the steps I took after the recommendations in this thread:

-downloaded MegaRaid Storage manager onto the ISIS. Logged into the IP to display the slot configuration.

-located slot 8 which listed a media error count of over 500. All others had a count of zero.

-right clicked on slot 8 - select locate - One of the 16 ISB's began blinking a single red indicator light. Slot 8 ISB confirmed. Took photo of slot 8 ISB SN displayed by MSM.

-Disconnected all workstations from ISIS

-Pulled ISB in slot 8 forcing failover. Confirmed SN on physical drive with photo taken. Confirmed, pulled the correct ISB.

-MSM displays the hotspare ISB turned into an active ISB

-Insert Cold Spare ISB into empty slot.

-ISIS management console confirms drive is rebuilding, slowly but surely.

Here are some performance issues that led me to force failover:

*async file system error
*failed to get sample size from ama plugin
*unable to transcode ama linked media
*unable to complete an MXF OP1a sequence export (no error message displayed, just failed to export a file)

Since replacing the ISB these errors have stopped and we're back to normal :)

Well done!

One of my biggest bugbears of several with Isis, 5500 in our case, is it’s singular inability to realise when a disk is dying and boot it out of the raidset. Instead we have to rely on annoyed editors complaining about an inability to transcode or write files to a workspace and then get out my Sherlock Holmes hat and magnifying glass to find out which of 144 drives has failed this time. Since we have a drive failure roughly every 4 weeks I get to do this a lot.

Frankly it’s utterly amateurish regardless of whether the fault lies with LSI or Avid, and is one of the reasons our next storage system is unlikely to have Avid on the bezels.

doug shannon:

get out my Sherlock Holmes hat and magnifying glass to find out which of 144 drives has failed this time.

100% agree that LSI+Avid are letting the system down with bad design. I've been searching if the LSI card/driver could use a media error count amount to initiate the hot spare replacement but never found anything. Sadly Avid also did not find it necessary to create the ability to monitor individual drives, initiate a replacement or automate replacement in the ISIS web page (through the MegaCLI or StorCLI). As ISIS 5000 was already end of development and will go end of support in a month, I don't think our criticism will serve any purpose.

doug shannon:

one of the reasons our next storage system is unlikely to have Avid on the bezels.

The good news is that Avid corrected this in Nexis. Nexis always provided monitoring of all individual media drives from day 1 and since 2018.4 also the internal SSD drives for the SDA. I consider Nexis a great step forward combining features from Unity, ISIS5x00 and 7x00. And at it's current price I believe you should keep Avid storage on your options list.

That said, as nothing is perfect, I still believe Nexis can be improved especially when it comes to nic teaming and how client and network issues can affect overal system stability. In my opinion Nexis client needs more enhanced (setup) fault analysis and error reporting as very similar Sherlock Holmes methods are needed when for no obvious reason systems can't write to workspaces or not write with the correct speed al of a sudden. 

I have not seen a product in the Video SAN/NAS space that proactively "fails" the drive in order to initiate an early rebuild. I have tried Terrablock, EditShare, XSAN/StorNext, Promax, etc...

With ISIS Toolbox SMART feedback we usually can figure out which drive is about to fail.

With Nexis, Avid does have a drive state called "dying". But we have not experienced it other than through a bug so I can't tell you how well it will work.

IMHO, Avid ISIS is by far the best video shared storage solution we have ever experienced.

It is the easiest to deploy. Easiest to administer. Definitely easiest to maintain. 

There are storages that are cheaper and faster. But ISIS overall is still the best.