Latest post Wed, May 18 2011 9:25 PM by Doc Avid. 10 replies.
Page 1 of 1 (11 items)
Sort Posts: Previous Next
  • Fri, May 6 2011 1:48 AM

    • danversj
    • Not Ranked
    • Joined on Fri, May 6 2011
    • Posts 4
    • Points 70

    Accessing/editing off network attached storage

    I'm not a regular user of Avid products, but we create video files that end-up being edited on Avids.

    On a job we can create upwards of 80 hours of SD or HD video which is then transported on removable drives or RAID arrays for later post-production.

    We use EVS products and can write our files in OPAtom format.  The ideal workflow would be to connect these drives into an Interplay-connected Media Network so that multiple MCA stations can access them instantly.  We want to avoid a wait while the media transfers or imports into a fixed shared storage array.

    Basically we want to write to a portable RAID array (with onboard PC - the EVS "XFly"), then connect that RAID array to an Interplay system (via Gigabit Ethernet) and edit directly off it, and not have to wait for it to import.

    Is this possible?

  • Sun, May 8 2011 8:27 PM In reply to

    • TrevorA
    • Top 100 Contributor
    • Joined on Thu, Oct 13 2005
    • London
    • Posts 1,142
    • Points 12,925

    Re: Accessing/editing off network attached storage

    danversj:

    Basically we want to write to a portable RAID array (with onboard PC - the EVS "XFly"), then connect that RAID array to an Interplay system (via Gigabit Ethernet) and edit directly off it, and not have to wait for it to import.

    Is this possible?

    AIUI - No I don't think you can share a NAS that isn't ISIS but I suspect you aren't the only one asking. 

    I have had a similar issue with trying to access previously consolidated media - always seem to need to copy it back to ISIS to access it.

    You won't need to import - but I think you will need to copy the media to the ISIS. 

    You could mount the drive locally and have the local indexer deal with it - as long as you only need to edit with the material in one edit suite. 

    Or you could try working with it as a virtual drive? Never tried this...

  • Mon, May 9 2011 2:34 AM In reply to

    Re: Accessing/editing off network attached storage

    You can mount network volumes to a MC. You should first put the command alldrives in the Console. Then you can reach that volume, and of course it should have the Avid MediaFile folder structure to get recognized.
    Hope it helps.

    * Avid Media Composer 2018.2 * Interplay | Production 3.8.1 * Media Central 2.10

  • Mon, May 9 2011 2:43 AM In reply to

    • danversj
    • Not Ranked
    • Joined on Fri, May 6 2011
    • Posts 4
    • Points 70

    Re: Accessing/editing off network attached storage

    Thanks TrevorA, great answer.

    I was thinking of perhaps just attaching the drive to one MCA PC, or to a central server then having all the other MCA PCs mount the share as a drive letter.  Of course, this would allow multiple access to the same footage - is this inviting trouble?  It seems too easy to be a realistic solution.

    The virtual drive letter would help prevent multiple access of the same footage at the same time (is this a problem? - they would be just reading the footage, not writing to the drive.).  But we're currently trying to move away from a filename/folder based method of organising our footage, and trying to move towards sorting our footage with EVS metadata.  This would involve throwing all the footage into the root directory of the drive or drives and not worrying too much about file names.  We'd like for the editors to search from within Interplay or the Media Tool to find the footage they want.

    We currently ship the footage on 4 separate SATA drives in caddies (as OP1a MXFs), but the edit suite can only have 2 drives plugged in at the same time.  So it's someone's job on the Monday morning after shooting to ingest this footage into the suites where they'll be needed.  It's pretty slow and inefficient.

    The new EVS product called "XFly" is a portable RAID array with onboard PC (probably Windows XP).  I wonder if it would be possible to install some sort of Avid software on it to enable it act as part of an Avid Interplay / Medianetwork or ISIS?  I'm sure the client would be happy to pay a license fee for this.

  • Mon, May 9 2011 7:01 AM In reply to

    • TrevorA
    • Top 100 Contributor
    • Joined on Thu, Oct 13 2005
    • London
    • Posts 1,142
    • Points 12,925

    Re: Accessing/editing off network attached storage

    danversj:
    we're currently trying to move away from a filename/folder based method of organising our footage, and trying to move towards sorting our footage with EVS metadata.

    Personally, as an editor, I'd prefer a sensible file naming convention that carries across to the clip name - much the way you might work in FCP. As powerful as the interplay search engine is and as much metadata as the EVS/IPDirector can give you, to have the main nuggets in the file&clip name makes sifting bins and searching timelines quick and easy.

    You could do the file naming once the drives are back from the field and before editing - as long as you aren't keeping versions in the field that you are going to need to sync back to.

    Going forward I can see the FCPx keyword system (which is somewhat akin to virtual bins) being very useful. 

    The 'alldrives' solution may well work with the Xfly though. I suspect that XFly is actually a bit late to market as you can plug any old NAS into an XT and have it copy the files there I think? 

    As far as your current setup goes could you not plug the raw SATA drives into a JBOD chassis? You could try the alldrives network share with this (or even the single SATA drives) and see if you can access them across the network. You'll probably run into issues if pulling lots of material across the network, especially if not RAIDED.   

  • Thu, May 12 2011 12:43 AM In reply to

    • danversj
    • Not Ranked
    • Joined on Fri, May 6 2011
    • Posts 4
    • Points 70

    Re: Accessing/editing off network attached storage

    TrevorA:

    danversj:
    we're currently trying to move away from a filename/folder based method of organising our footage, and trying to move towards sorting our footage with EVS metadata.

    Personally, as an editor, I'd prefer a sensible file naming convention that carries across to the clip name - much the way you might work in FCP. As powerful as the interplay search engine is and as much metadata as the EVS/IPDirector can give you, to have the main nuggets in the file&clip name makes sifting bins and searching timelines quick and easy.

    You could do the file naming once the drives are back from the field and before editing - as long as you aren't keeping versions in the field that you are going to need to sync back to.

    Going forward I can see the FCPx keyword system (which is somewhat akin to virtual bins) being very useful. 

    The EVS system allows us to set up just about any file name sytem.  We can keep doing that - remembering that file names allow you to sort only on the first parameter - be it Clip Name, Timecode IN, etc.  That part of the operation is automatic and I don't have a problem with it.  For us, the labor-intensive part is sorting all the files into separate folders.  We do motor racing, so the events are multi-day, so a folder for each day.  Then under each day there are folders for the different race categories.  The EVS system doesn't seem to easily allow us to automaticaly categorise clips and program recordings by event, category and race - without having to manually keyword each clip - we want to use keywords to descripe what happens in each clip.  There are ways I'm sure, we just need a bit more research.

    TrevorA:

    The 'alldrives' solution may well work with the Xfly though. I suspect that XFly is actually a bit late to market as you can plug any old NAS into an XT and have it copy the files there I think? 

    As far as your current setup goes could you not plug the raw SATA drives into a JBOD chassis? You could try the alldrives network share with this (or even the single SATA drives) and see if you can access them across the network. You'll probably run into issues if pulling lots of material across the network, especially if not RAIDED. 

    We currently plug 2 of the SATA drives at a time into a JBOD chassis.  I don't have direct control over the edit suite setup - currenty they are happy using the 2 drive bays to import the OP1a MXFs into the MC stations where they're needed.  I was recently asked my opinion to upgrade the system and I've recommended the XFly and OPAtom files.  I also recommended a Unity system of some sort, but now I'm not so sure - in the edit suite "edit while capture" is not necessarily so important.  The footage can be delivered in OPAtom format already, on drives that are fast enough to edit off.  We shouldn't have to wait for any sort of ingest/import/copy process before editing.

    With the multiple Media Composers running in standalone mode and accessing the shared drive (using "alldrives"), is there a problem with each MC station trying to write database files to the Avid Mediafiles folders - and overwriting each other's index files?

    The JBOD SATA drives would probably be fast enough - they each have a sustained (write) transfer rate of about 34MB/s which probably should cope with simultaneous access from 2 MC stations.  I think the RAID array will be better because all the footage can be put in one location.  We can then use the IP Director to create logs and playlists that have AAF and/or XML EDLs that link to any of the clips and program recordings.

    It's a bit ironic that in doing research to try to get more reasons to buy a Unity system, I've found reasons *not* to buy one.  Not being able to plug in external storage and immediately edit off it (via Unity/Interplay) is a problem.

  • Thu, May 12 2011 6:19 PM In reply to

    • Chris Aust
    • Top 500 Contributor
    • Joined on Thu, Oct 13 2005
    • Munich
    • Posts 436
    • Points 4,765
    • Avid Certified Instructors - Video
      Avid Certified User
      Moderator: Avid Certified Instructors - Video

    Re: Accessing/editing off network attached storage

    Hi,

    sorry, I did not read the previous threads and just flew over it, so maybe this was answered before.

    How does Interplay track MediaFiles? The MediaFiles are on a shared storage (ISIS or MediaNet 5) on the virtual drives called workspaces. To make Interplay and all connected clients and servers see the MediaFiles, the workspaces containing those files must be added to the MediaIndexer by configuring the MediaIndexer Service. All MediaFiles that need to be shared have to be on those workspaces.

    "Alldrives" on a client will not make the files shared!

    "AMA" on a client will not make the files shared!

    Copying the files to a client will not make the files shared!

    Copying the files to a workspace will not make the files shared! - Why? This is a operation only on a operating system level and not through the database! You will need to consolidate to make the Indexer talk to the database when consolidation is in progress and to tell the database where you want to have the files when checking in.

    So the only way is to consolidate the files through a editing client! Allright, here you have your bottleneck!

    Why can I not just mount a drive on a local client and use it as a share for other editing clients? Well, ever heard of bandwidth?!?

    Well, you can add the local drive of the editing client to your MediaIndexer Server, but this will mess up your infrastructure! MediaIndexer is keeping track of the files and a huge amount of communication is going on between the workspaces and the MediaIndexer. That is a reason why a MediaIndexer is connected with both lines to the ISIS in Zone1. You don't really want all of that traffic on your local machine! Additionally the Indexer is configured using the UNC and not drive letters. A drive M: would be a drive N: on another machine!

    Every ISIS chassis provides a bandwdith of 400MB per second. If you have 6 chassis then it would be 2,4GB per second of bandwidth. Editing clients are 1Gb ethernet connected to the chassis or a switch. If you have Highrez clients then you use 10Gb connected clients via xfp-ports. You do not want to use that connection for accessing mediafiles in realtime for other editing clients!

    When buying a shared storage system with a CMS you have to consider a lot of things. First of all you will have to think about your ingest and playout. You will need to think about the amount of programm you need to produce per day, thus leading to the amount of editing clients, storage and bandwidth. Thus you will also need to think about the resolution you want to work with. And security of you data!!! Configuring a ISIS or Unity environment with interplay is not an easy task. A lot of thinking needs to be done. You do not just buy ISIS and go. You will start with project management and end with database design and dry run. I know companies where this takes several months! On the other hand for postproduction this can be accomplished much faster.

    So what could be the solution with EVS? There is a EVS integration using TransferManagers and InterplayWebServices to directly transfer media to the ISIS or Unity storage and checking it in to the database. That is a common procedure and many companies work like that. Ask your EVS or Avid reseller, they should give you the information you need. In the Munich EVS office I saw a project folder with many examples of Interplay and EVS integration.

    The question is if you really need Interplay! With the amount of media you create and tracking the data the answer is probably yes, but only a project manager can tell you.

    So there is no way to efficiently just attach field drives to interplay and edit directly from that in a shared environment. The reason is just a technical one.

    hope this answers the initial question

    -chris

     

    MC/Symphony 8.7.2 | VMware Workstation 14 virtual Interplay on Ubuntu Linux 16.04 4.15.03| Dell XPS 15 |||| obsolete MacBookPro 17", 2.5GHz Quad Core... [view my complete system specs]

    Chris Aust

    Munich, Germany

    Avid Workflow Consultant | ACI | Trainer

    Splash!Zone

     

  • Fri, May 13 2011 2:11 AM In reply to

    • danversj
    • Not Ranked
    • Joined on Fri, May 6 2011
    • Posts 4
    • Points 70

    Re: Accessing/editing off network attached storage

    Hi Chris, thanks for your interest - your quotes below are edited for length...

    Chris Aust:

    How does Interplay track MediaFiles? The MediaFiles are on a shared storage (ISIS or MediaNet 5) on the virtual drives called workspaces. To make Interplay and all connected clients and servers see the MediaFiles, the workspaces containing those files must be added to the MediaIndexer by configuring the MediaIndexer Service. All MediaFiles that need to be shared have to be on those workspaces.

    Copying the files to a client will not make the files shared!

    Copying the files to a workspace will not make the files shared! - Why? This is a operation only on a operating system level and not through the database! You will need to consolidate to make the Indexer talk to the database when consolidation is in progress and to tell the database where you want to have the files when checking in.

    So the only way is to consolidate the files through a editing client! Allright, here you have your bottleneck!

    Why can I not just mount a drive on a local client and use it as a share for other editing clients? Well, ever heard of bandwidth?!?

    Well, you can add the local drive of the editing client to your MediaIndexer Server, but this will mess up your infrastructure! MediaIndexer is keeping track of the files and a huge amount of communication is going on between the workspaces and the MediaIndexer. That is a reason why a MediaIndexer is connected with both lines to the ISIS in Zone1. You don't really want all of that traffic on your local machine! Additionally the Indexer is configured using the UNC and not drive letters. A drive M: would be a drive N: on another machine!

    So what could be the solution with EVS? There is a EVS integration using TransferManagers and InterplayWebServices to directly transfer media to the ISIS or Unity storage and checking it in to the database. That is a common procedure and many companies work like that. Ask your EVS or Avid reseller, they should give you the information you need. In the Munich EVS office I saw a project folder with many examples of Interplay and EVS integration.

    The question is if you really need Interplay! With the amount of media you create and tracking the data the answer is probably yes, but only a project manager can tell you.

    So there is no way to efficiently just attach field drives to interplay and edit directly from that in a shared environment. The reason is just a technical one.

    Thanks for giving me a bit more insight into the problem.  I've been reading elsewhere in these forums of people successfully doing the "alldrives" trick.  I know it's unsupported, and I can now more understand why.  Still, for our purposes, with only 2 or 3 MC stations, we might give it a go.

    Obviously, an Interplay system overall is a superior solution.  From the EVS perspective, transferring clips and live streams to the Avids in via Transfer Manager or WebServices works a treat.  I love it when we have one on site at our OBs - which is not that often.

    Our problem occurs when we have either a standalone Avid on site, or no Avid at all, and we have to transport 60-80 hours of footage from the OB to the edit suite on hard drives.  Time is money, obviously, and the editors would like to be able to use the footage immediately.  At the moment, to work with Interplay, there is a wait before all this footage can be used - it needs to be transferred, imported or ingested before it can be edited with.  Logistically, there are ways of managing this wait, but wouldn't it be better if we didn't have to wait?

    What I'm after, and it seems to not exist (yet), is a portable Avid storage solution that can interface directly and immediately with a non-portable Interplay system.  By portable I mean "person-portable".  Basically, a network-attached 8TB RAID array that can fit in a carry-on-luggage-sized case on an aircraft.  At the OB we fill it full of footage.  Perhaps it has an onboard OS that indexes the footage in the way that Avids like.  The array is then transported to the edit suite.  The onboard OS connects to the edit suite ISIS system, the index files that were written at the OB are read by the ISIS and the footage on the RAID is shared.  Ideally, while the editors edit with the footage off the portable RAID array, the ISIS transfers the footage from the portable RAID to the main ISIS storage, and updates pointers to the footage in the background.  Once all the footage has been transferred, the portable RAID can be disconnected.

    I'm guessing the difficult bit here is merging two separate stores of footage, with two separate indicies.  As an outsider, it seems that the hardware could cope, but the software is not yet designed to deal with this situation. The EVS XFly has 2 gigabit ports, an onboard OS (probably Windows XP) and the early specs say is can handle 1000Mbit/s sustained writes - read speeds are probably faster.  Certainly it can't match an ISIS chassis for bandwidth, but it's not slow, either.  I don't think it invalidates the concept of a portable storage module that can plug into an ISIS.

    The connection style isn't really important - be it Gigabit Ethernet or eSATA because to make this work, you could just pull the drives out of the portable chassis and plug them directly into a module on the ISIS.  What we want is for Interplay to (nearly) instantly recognise the addition of a large store of video files.  It should be able to index these pretty quickly and add them to the database.  The files are already OPAtom MXFs as IMX50 (MPEG50) or DHxHD 120.  It shouldn't have to feed the footage linearly through an "import/ingest" software module.  It shouldn't need to touch the video files, just recognise that they exist.

    We could probably work out a workflow where we don't need Interplay.  But then we are locked into that workflow.  Interplay would give us a lot more flexibility to cope with any unusual situations in the future, and allow our workflow to smoothly evolve.

    I'm going into detail in these posts because I think it's a question that will be asked a lot.  More and more workflows are becoming tapeless.  Being freed from the limitations of tape (or other ENG disk / flashdrive-based formats) means that entire Outside Broadcasts (with many program feeds and camera angles) are being transferred, on hard drives, from the field to the edit suite for immediate post-production.  This post-production process is under pressure to put out finished products very quickly. In our case there are multiple clients and multiple products to produce from the one set of recorded footage.  We need multiple editors to work on the recorded footage from the OB, and we need to do it fast.  If there's any company that can come up with a solution for this situation, I'm sure it's Avid. :)

  • Fri, May 13 2011 7:55 AM In reply to

    • TrevorA
    • Top 100 Contributor
    • Joined on Thu, Oct 13 2005
    • London
    • Posts 1,142
    • Points 12,925

    Re: Accessing/editing off network attached storage

    danversj:
    The EVS system doesn't seem to easily allow us to automaticaly categorise clips and program recordings by event, category and race - without having to manually keyword each clip - we want to use keywords to descripe what happens in each clip.

    Automatic keywords are possibly your friend here (event = "x" keyword added to all clips until you change it to event = "y"). Alternatively, or in addition, IPLogger is also v. useful and may help. Or you could have the EVS operators change the camera name on a per race basis, but I'm guessing that is impractical. 

    All of this stuff is still really in its infancy* and is mostly used on a 'just in time' basis (i.e. we only figured out how to get to air 'just in time' we didn't have enough time to plan out a workflow...). Especially with everyone being freelance and therefore out of the door, and out of a job, as soon as the truck is de-rigged.

    * primary school level, maybe

     

  • Tue, May 17 2011 12:28 PM In reply to

    Re: Accessing/editing off network attached storage

    Chris Aust:
    "Alldrives" on a client will not make the files shared!


    Sorry guys, I didn't saw the part where you said that you need to share that media...  that was why I proposed the "alldrives trick"

    Sorry. 

    * Avid Media Composer 2018.2 * Interplay | Production 3.8.1 * Media Central 2.10

  • Wed, May 18 2011 9:25 PM In reply to

    • Doc Avid
    • Top 50 Contributor
    • Joined on Mon, Jul 17 2006
    • Los Angeles / Hollywood
    • Posts 2,321
    • Points 25,890

    Re: Accessing/editing off network attached storage

    Is someone doing this?  Most likely-   

     

    The issue you do have is that your files  

    A. Have no metadata associated with them -  no masterclips

    B. Are not checked into interplay.

     

    so why is it likely that this IS being done?   There are workflows out there that have been created for animation, where frames are renedered, gathered, then wrapped into an avid media file.   

     

    So look through the SDK's and see if there is something that can be used to add that extra Avid metadata without visiting Media Composer.

     

    And then it's a simple matter of checking the materials into Interplay AFTER getting the media on Indexed Avid shared storage.  

    You need yourself a code monkey, and some cash.

     

    JDS

    So many systems, so little space. So... everything Avid and a Hyundai Veloster Turbo. [view my complete system specs]

    Jeff Sengpiehl ACSR Mac/PC/Unity/ISIS Los Angles CA AIM Docavid - Yahoo Docavid Join the Avid-L- It is ALL there: http://groups.yahoo.com/group/Avid-L2 http://tinyurl.com/zl6ez http://tinyurl.com/27qw4o for your Iphone

    Filed under:
Page 1 of 1 (11 items)

© Copyright 2011 Avid Technology, Inc.  Terms of Use |  Privacy Policy |  Site Map |  Find a Reseller