Toward a Video-for-Operations Ecosystem

Imagine an industrial facility with dozens of video cameras that continually record and store video of plant operations … for weeks, if not months. If the cameras are strategically placed, they will capture a visual record of nearly everything that happens in the facility. When an issue deserves study and analysis, you will already have a rich mine of detailed, provably accurate information. Further, everyone uses specialized cameras and mobile devices to record interesting events spontaneously. These video clips are intelligently integrated with the permanent recordings to add depth and texture to the visual (and audio) record.I describe a vision of integrated video handling in industrial operations. My goal is to identify and interconnect many independent technology building blocks into one coherent “system” … using techniques and products that are commercially available today. It is, I believe, roughly the best system we can build right now. However, I hope to see far better designs evolve over time and those will be subjects for future posts.

Video for Operations Ecosystem

Details of the Functional Elements in the Ecosystem:

Security video cameras on fixed installations. Cameras have automated tagging features built in
Video Management System (VMS) to record and store video feeds for extended periods.
Software filtering and export tools to extract shorter video clips based on tags and issues of interest
Mobile video camcorders with ability to transfer videos
Computer screen “screencasting” software
Cameras and apps (possibly with tagging) on mobile devices
Video files from Machine Vision Systems
Performance analysis software to do fine tagging and manipulation from ingested raw video
Software to organize, analyze, share, and display information from the videos and from the available tags.
Cloud service to upload video from all sources
Software to tag video from various sources prior to archiving them to a video library that can handle associated tagged video clips from all sources.
Software tools to share tagged video and accept new tags for existing video.
Desktop and mobile apps that retrieve and display video clips that are filtered and sorted according to their tags.

Unfortunately, most of the useful information will be buried in a mass of raw video and, until recently, there was no easy way to extract it. This has been the main practical objection to using video for process improvement. At DeeperPoint, we are exploring designs for operations video “ecosystems” to mine this video content more effectively. What we have found is that no one currently sells a complete ecosystem, but there are many independent tools (hardware, desktop, cloud and mobile) that might integrate into a workable solution. We believe we’ve identified at least one practical configuration that has the tools and integration to be useful in day-to-day operations. The diagram above summarizes the building blocks of this solution and later sections detail the relevant tools for each step.

The diagram shows three special markers for elements that recur throughout the ecosystem. The first is the “tagging” tools that place tags on interesting bits of video as they are captured, ingested or processed. Effective tagging allows downstream software search tools and “filters” to use the tags to quickly isolate and retrieve video segments for a specific interest. The third marker flags elements where there have been dramatic technology or cost advances in the past 2 to 3 years. That is almost the entire ecosystem! It is why we believe it is time to give this ecosystem serious consideration. In our opinion, most of these new features are game changers.

Video Security Systems (1-3)

We have written several previous blog articles about the recent developments in security video camera systems. Those articles talk about the price drops, the resolution improvements, the speed increases, the potential for greater process visibility, and why previous objections may not longer be valid.

Cameras, Network & Software Systems for Video Monitoring

The most critical factor for including this technology in the ecosystem is the the systems have advanced so far in the past 3 to 5 years. Today, a fairly standard video surveillance system includes the following as standard features:

High quality HD (or better) video cameras networked by Ethernet and possibly wireless.
Industrial cameras with embedded intelligence to watch designated zones, read license plates, recognize and track people, look for items left behind, etc. etc.
Industrial cameras have microphones and digital inputs to recognized audio and alarms.
Sophisticated video compression algorithms that dramatically reduce bandwidth and video storage volumes.
Video Management System servers accept high quality video feeds from dozens, even hundreds, of cameras and can track and search the camera alerts and meta-tags.

You can buy inexpensive systems with a central recorder, 8 or 16 cameras and pay under $2500. Those have limitations that greatly reduce their ability to contribute to a larger ecosystem. A better idea is to source a moderately priced system from a major industrial vendor like Avigilon, Axis, Bosch, Dahua, Hikvision, Milestone, Panasonic or Sony. These systems are geared to industrial and commercial installations. In most cases, companies will already have a security system contractor that is a distributor for some of these manufacturers. Since the market is so competitive, it may be easier to stick with the existing contractor.

Cameras and Video Sources (4-6)

We can’t hope to keep track of all the products that might have a role in the ecosystem. It seems like every personal electronic device (e.g., phones, tablets, laptops) can record HD video and upload it to a PC or cloud server. In addition, there are specialized cameras (e.g., microscopes, drones and thermography) that add more visual perspectives.

Diverse Industrial Video Sources

So far, most of these products seem to be single-purpose applications. They do amazing thing(s), but they aren’t integrated into a larger ecosystem. The video files transfer easily enough … MP4 is nearly standard … but the meta-data does not. For example, a thermographic imager produces a standard video file with unique tags and descriptors.

Dartfish Family of Communicating Software

An exception is the Dartfish family of mobile video performance analysis apps. Dartfish Express does spontaneous video capture, tagging and markup. Dartfish EasyTag and Dartfish Note can independently log metatags that are later merged back with videos from other sources. As noted at the end of this post, this problem of integrating the meta-tagging represents a major ecosystem design challenge.

Industrial Machine Vision Software (7)

It is increasingly common to find that equipment manufacturers have built machine vision into their new equipment. These vision systems are engineered elements that mesh tightly with the machine’s other devices and operation. For example, they may display views that are hidden or unsafe for operators to see directly. They may inspect the product in real time. They may monitor visible problems and events and signal them to the operator. The common denominator is that this type of software is virtually always custom-coded for the specific engineering situation. It is not transferable to studying other equipment or other types of activity.

There are many companies that build custom-coded machine vision systems for specific applications, not tied to a specific machine or vendor. Again, they must be custom-engineered for a narrow purpose and the results are not portable to other problems. Since they are custom-engineered, they also tend to be fairly expensive. The custom software is generally built using sophisticated video processing software libraries. These speed development and reduce cost, but require expert programmers to assemble and calibrate.

Vision system sorting vegetables

These software types generally don’t serve our interest in permanent, permanent video systems to study general operations. However, a general-purpose system might be helpful for companies that are contemplating investments in targeted machine vision. Security video cameras have become so good that they can see and record a lot of interesting detail. That might be helpful during the planning and procurement phases of machine vision acquisition.

Video Editing Software (8)

This category is populated with sophisticated video editing tools like Adobe Premiere and Adobe After Effects. These suites do everything a movie or video director needs to create a feature length video or movie. They handle different time formats effortlessly, make precise edits and transformations, handle audio with aplomb, create and add complex graphic overlays and special effects, and they can translate video between most major video formats. Importantly, this category is a big market with companies that make large investments in R&D. These products won’t disappear and you can expect them to improve every year.

Non-Linear Video Editor

We find that video editing software is best used to prepare videos for later analysis. Beyond that, their abilities are limited because they aren’t designed to measure things and their tagging methods are not shareable … which is the main agenda when you study and understand real-world events and operations. However, if you want to combine videos in complex ways to see obscure features, or play them with synchronized audio tracks, these are the tools to use.

Video Performance Measurement and Analysis Software (9)

This is the category with the most relevance to mining data from video for operational use. Unfortunately, there is no comprehensive product or solution. The members of this category tend to be isolated software tools for narrow applications. For example, one sub-category uses video to study ergonomics and safety (e.g, Siemens Xsens and TEA Captiv). They focus on human body movements and impacts and it’s hard to see how they would apply to machine operation or process workflow. Another example is software that studies customer behavior in retail settings (e.g., Brickstream and Xovis). You can track individual customer movements to improve product placement and checkout workflow, but they can’t look at individual actions or activity.

Analyzing activity to extract process events and measured variables

Ironically, the most promising sub-category is designed to study elite athletics. This is the software that coaches use to analyze videos of Usain Bolt and Michael Phelps to help them go faster. It’s the software that NFL teams use to break down the films from Sunday for team study on Monday. It is the software that was used by hundreds of medalists at recent Olympics and winning teams at the World Cup. This video analysis software is used by coaches around the world in sports as diverse as golf, judo, pole vault, tennis, soccer, equestrian and NASCAR. There are so many aspects to study in sports that these software products must have versatility to survive.

At DeeperPoint, we’ve been working with Dartfish (www.dartfish.com), the acknowledged world leader in sports video analysis. Dartfish has many features and capabilities that are purpose-built to extract qualitative and quantitative data from performance videos. More importantly, it has built an imposing list of features because it has been listening to the needs of coaches and athletes in a wide range of sports for nearly two decades. The examples below from sports and industry illustrate some the analyses that can be done with Dartfish.

Click on the images to see enlarged versions:

Image title

Your subtitle here

Searchable Cloud Video Libraries (10-12)

After you have captured, tagged, selected, edited and analyzed the video clips, you will want to archive them where they are easy to access, share and reuse with relevant stakeholders. That likely means uploading them to a cloud video repository with the security and software to handle them properly.

Alternatives for Cloud Video Archiving, Tagging, and Searching

There are a multitude of options (from YouTube to paid hosting systems like Dartfish.tv), so it is a good idea to define a list of requirements and evaluation criteria to help you choose. Here are some likely considerations:

Can you tag the videos with searchable descriptions?
Can you tag and search for segments within the video clips to create libraries of similar content from different clips?
What does the search engine look at and how does it return the results?
How closely can you control access to specific clips by individuals and groups?
How easily can you upload content?
How strong is its security model?
How well does it work with mobile devices?
etc. etc. etc.

This is possibly the most dynamic and competitive area in the video for operations ecosystem. Many software companies are building cloud video libraries that might be useful, but mostly you just have to try them out and see.

Mobile Video Viewing and Analysis Apps (13)

Working with and complementing the growing list of cloud video servers are dozens of mobile apps (IOS and Android) that promise to mark up video or extract data from video clips. Most are single-purpose apps with a focus on personal entertainment, social media or athletics. Driven by the convenience of working from a phone or a tablet, this category is worth watching. Any employee can carry these free or low cost apps on their phone … close at hand to record and annotate anything that they see.

Like most mobile apps, the programs in this category are tied to cloud servers that may do a lot more in the background. This can increase the apps’ appeal, but it raises security and privacy issues that need to be considered. Finally, the apps are evolving so quickly that it takes considerable effort to monitor the changes and new offerings. Again, we’re most familiar with the mobile apps published by Dartfish for sports applications. They seem promising supplements to a video operations ecosystem that uses permanent facility-wide video recording. Dartfish mobile apps allow users to grab spontaneous video clips from their phones or tablets and upload them to a cloud repository and share them in a controlled manner over the Internet. There is a simple “Dartfish Note” app that lets users tag real-life events with a few button clicks to their phone. These tagged events can be uploaded and synchronized against video from other sources… e.g., video segments from the permanent operations video monitoring system.

Mobile Video Apps

Bottom line, the growing list of mobile apps needs to be combined other software tools to be practical in an industrial setting. The mobile systems aren’t robust enough to fulfill the continuous improvement needs of a mechanical or process engineer. They are, however, be a great way to grab spontaneous, supplementary videos and integrate them later into the larger video context.

Ecosystem Integration Issues and Recommendation

With versatile tools, the recorded streams from a permanent operations video system have the potential to become a huge data mine. If this can be combined with properly tagged video from specialized cameras and sources, we are very close to having the ability to use video to recreate most of the mission-critical activities that have occurred in a given industrial facility. The challenge is to connect all of these pieces into an integrated ecosystem. For ecosystem construction, there are two competing approaches:

Find the best-in-class solutions for every part of the ecosystem and figure out how to wire them together.
Find the most complete overall video tagging system and wire in as many of the specialized components as will fit.

The problem with the best-in-class approach is to get the various components to talk to one another. Passing video files isn’t the problem. Virtually all software and systems will import and export MP4 videos. The still-unsolved problem is how to share the metadata (tags and descriptions). Building a really functional ecosystem workflow demands a way to pass metadata from one component to another and through all the downstream components. We see no sign of standards for sharing event tags and descriptions that can be industrially applied to the clips as a whole, let alone to segments inside the clips. If one exists, most technology vendors don’t honor it.

The second approach involves picking one system vendor to serve as the system backbone. Given the situation described above, the best option at the moment would be to anchor the ecosystem around the Dartfish family of products. Dartfish has already addressed many of the ecosystem integration challenges … in sports. Although it hasn’t been visible in business or industrial settings, its fundamentals are sound and our experiments give us confidence that it can cope. In particular, I would push all tagged or valuable video assets into Dartfish.tv.

The Dartfish system envelope covers a wide range of ecosystem functions. If it is not the best at any given function, it is at least reasonable. Most importantly, it maintains the same tagging and tag storage technology across all of its products. If you can define the tags in Dartfish early in the process, Dartfish will preserve them through subsequent activities. It also has some of the strongest tools to export, import and synchronize tag data with video from a variety of sources.

This is why DeeperPoint has spent so much time in these blogs exploring the Dartfish family’s capabilities. It’s not because we are financially wedded to the company or the product. It’s because, at the moment, Dartfish seems to be the only system product with the breadth and consistency to assemble a reasonably complete operations video ecosystem. We don’t see the situation changing for several years, since it will take at least that long for another vendor to assemble an integrated system with sufficient coverage and capability.

Update 6/21/2017 – Microsoft Office 365 Stream

Microsoft just added (literally yesterday) a new video hosting capability to my Office 365 installation. Office 365 Stream is a much improved tool for uploading, hosting, and sharing videos among Office 365 teams, SharePoint sites and Yammer Groups. My initial reactions are as follows:

Stream is an impressively serious offering that will make video more accessible to many business and industrial users. With Stream, Microsoft is strongly validating the growing importance of video to business operations.
Use of Stream is mostly limited to team members with paid subscriptions to Office 365. In a few cases (e.g., if you link it through Yammer), you can share videos with external ‘guests’ that take the trouble to authenticate their email addresses with Microsoft. That is free and one-time, but it is an extra step that will deter casual users. You can’t just add a link to an email and share it widely (e.g., in an open bid solicitation or email blast).
I don’t see support for detailed video tagging. You can fake it by adding search terms to the searchable description for the clip. However, that won’t narrow viewing down to a segment within a video clip. Plus, these ad-hoc tags are (like Twitter hashtags) notoriously hard to manage and keep consistent.
Stream has a cool feature that adds machine-generated closed captions to videos that are narrated in English or Spanish. However, I don’t see that the caption text factors into the search process. Even if it did, that wouldn’t be enough to assure effective searchability. Experts can say a lot of important things while only referring to the subject matter as “it”.
Stream is included in the base Office 365 business and enterprise subscriptions. If your staff or teams are subscribers, they have Stream at no extra cost.

The following updated ecosystem map shows roughly where Stream seems to fit at the moment. Stream’s impact on my ecosystem design is mostly as a potential future alternative to the Dartfish. As the backbone for a functional video ecosystem, Stream isn’t there yet.

For the moment, I would consider using Stream only if a) the team was already using Office 365, b) the team is mostly closed and works within itself, c) I was budget-challenged and, d) if I just wanted to dabble with the idea of using video. However, Microsoft seems to be making a strong statement about the value of video to business operations, so I expect Stream to improve rapidly in coming months and years. Then … who knows?

Add custom scale overlays to measure specific criteria
Create multi-frame overlays to break down complex actions
Use tags to count, measure and describe items and events in video.
Track locations and movements on a 2D plan or map.
Build a library with examples of common events and problems.