Manufacturing Plant Network is Down: What to Do

Manufacturing organizations are becoming increasingly dependent on computerized systems to produce their products. In addition to the more traditional applications that include resource planning, operations management, production line monitoring, and quality control, manufacturers are also leveraging computer-driven robots and plant floor automations to reduce costs and improve quality.   

Mission critical manufacturing system downtime can significantly impact the profitability of the organization. The financial impact of plant downtime is not the only issue manufacturers face when they experience critical processing failures. Loss of customer goodwill, bad press, idle employees, and legal ramifications must also be considered.

As a plant manager, you’re responsible for all daily operations. That responsibility includes ensuring the availability of all systems that are in the critical path of producing your product. One of the benefits of using a managed service provider is leveraging their expertise to accelerate problem identification and reduce resolution times.

Working with an MSP is different than interacting with your internal personnel. The following recommendations will allow your organization to fully leverage the benefits of your MSP relationship and ensure that system issues are resolved as quickly as possible.

Build the Foundation for High Quality MSP Support

Problem resolution time is a key metric that all organizations use to evaluate their managed service providers. During the client on-boarding phase, the MSP will ask your organization to categorize the systems they are supporting based on their importance to plant operations. The vendor will provide different contact mechanisms and communication protocols based on the severity of the issue and the criticality of the system experiencing the problem.

The MSP will ask your organization to provide detailed problem notification and escalation contact information. This information should include the primary contacts for each system the MSP supports and the escalation procedures to use when that person is unavailable. The inability of the MSP to quickly contact client personnel that are responsible for the system lengthens problem resolution times. Keep the contact information current throughout the relationship life cycle.

In addition, the MSP will request contact information for system hardware and software vendors. If the MSP identifies a vendor product to be the root cause of the issue, their technicians will use this information to contact the manufacturer to provide additional assistance.

Identify the Scope and Criticality of the Problem

Your initial goal is to identify the basic, preliminary information that you will need to reduce the issue’s impact on plant operations, categorize the severity of the problem and allow the MSP to perform their initial analysis. The following questions will help you to gather the information you need:

• What systems are affected?
• Can the plant still operate effectively until the problem is resolved?
• Is the functionality that the system provides totally or partially unavailable?
• How many personnel are affected by the problem?
• Is the problem intermittent or continuous?
• Has this problem occurred before?
• Can you mitigate the problem’s impact until the issue is resolved?
• Are there any alternative processes that personnel can perform to allow manufacturing operations to continue?

Contact the MSP Responsible for the System

During the onboarding phase of the relationship, the MSP and client will agree upon a set of problem notification procedures based on problem severity and system criticality. The following steps will ensure that the MSP addresses the problem according to the predefined procedures:

• For problems that are not affecting critical applications or plant operations, adhere to the notification procedures defined for non-critical problems during the on-boarding phase. Use the appropriate communication mechanism to provide a detailed description of the problem and its impact on plant operations.
• For critical issues, the MSP will provide a 24×7 helpline. Call the helpline, describe the problem’s severity, and ask the MSP to start a conference call. Participants should include the plant personnel that best understand the system experiencing the problem and the MSP’s technicians responsible for supporting the platform.

Conference Call Best Practices

Conference calls allow plant personnel and MSP technicians to coordinate their actions. In addition, they leverage the group’s combined expertise and analysis activities to more quickly resolve the problem. But conference calls can quickly become non-productive if they are led incorrectly. Here are a couple of best practices to help you lead productive conference calls:

• Assign a moderator that is responsible for keeping the discussions on track and reducing side conversations.
• Assign a participant to act as a note taker to record discussions and assigned action items.
• Communicate all of the information pertaining to the problem to the managed service provider.
• Ask the MSP to assign technicians to begin troubleshooting the problem and provide regular status updates. The most common status frequencies are 15-minute and 30-minute intervals. You want the MSP’s staff to focus on problem analysis and resolution, not continuous communications. A common practice used by most MSPs is to ask their technician to activate the speaker on their cell phone and place it in close proximity to their work area.
• Although most hardware and software manufacturers are reluctant to join a customer conference call, they will communicate with a support technician that is responsible for resolving the problem. Savvy MSP technicians will activate a three-way phone conversation to allow the conference call participants to hear their conversations.

Perform a Postmortem Examination

The MSP will adhere to your organization’s postmortem action items and documentation requirements or provide their own procedures. The most important questions to ask during this process are:

• What was the true root cause of the failure?
• Were there any contributing factors? An example of a contributing factor would be a human error caused by a lack of training or poor support procedure documentation.
• What steps can be taken to prevent the problem from occurring again?
• How do we improve the overall problem resolution process?

Wrap-up

Entering into a relationship with a managed service provider allows organizations to leverage deep-dive skills and expertise in technologies that they prefer not to support internally. One of the most critical components of that relationship is how the MSP and client jointly handle problems affecting daily operations. Designing a well thought out set of problem resolution procedures and adhering to the best practices outlined in this document will allow MSP clients to ensure that issues are quickly addressed.