technology service providers

10 Feb 2015

Hot Topics Tom Hinkel 1 Comment

FFIEC Issues Update to Business Continuity Guidance

The FFIEC just issued new BCP Guidance in the form of a 16 page addendum to the existing 2008 IT Handbook on Business Continuity Planning. It is titled “Appendix J: Strengthening the Resilience of Outsourced Technology Services”, and it has significant implications for both financial institutions and service providers, and across the entire business relationship between the two.

The following excerpt summarizes the intent of the update pretty succinctly:

A financial institution should be able to demonstrate the ability to recover critical IT systems and resume normal business operations regardless of whether the process is supported in-house or at a TSP (technology service provider) for all types of adverse events (e.g., natural disaster, infrastructure failure, technology failure, availability of staff, or cyber attack).

The appendix is focused on third-party Technology Service Providers (TSP’s), and organized in four sections (with associated sub-sections):

Third-party management
- Due Diligence
- Contracts
- Ongoing Monitoring
Third-party capacity
- Significant TSP Continuity Scenarios
- TSP Alternatives
Testing with third-party TSP’s
- Testing Scenarios
- Testing Complexity
Cyber resilience

Assuming that you already have a relatively compliant* business continuity plan, I see several areas that may need immediate attention:

Vendor management. Expect expanded vendor pre-contract due diligence and on-going oversight, including a detailed understanding of how the vendor manages their subcontractors. The guidance also introduces the concept of “concentration risk”, which is the increased use of, and over-reliance on, one or more key service providers.
Contracts. Expect increased contract requirements, including provisions related to subcontracting (see above), the right-to-audit, data ownership and handling, and how the servicer plans to respond to new guidance and regulations.
Testing. Expect an expanded testing section, including participation in critical vendor testing.
Cyber security. Cyber events should be factored into all aspects of your BCP, with emphasis on responding effectively to a cyber attack. Expect your incident response planning and testing to get increased scrutiny as well.

There is one more element of the guidance that may prove to be the most challenging of all for outsourced institutions. In the past, manual procedures were always the primary alternative to automation, but because of the increased dependence on outsourcing, it may no longer be feasible for an institution to operate manually for any length of time. In those situations the guidance suggests that you have an alternative service provider identified to assume operations, or that you consider the possibility of moving the operations in-house. Since the guidance admits that the latter option is likely not a valid one, that really only leaves the alternate provider as a possible solution. Of course any institution that has converted their core system to a new provider knows that process is fraught with challenges even when the conversion is anticipated and carefully planned. Undertaking the process after a sudden disruptive event is almost unthinkable, but the guidance expects you to going forward.

* A compliant BCP is built around a business impact analysis which identifies all critical business processes and their interdependencies, establishes clearly defined recovery time and recovery point objectives (RTO’s & RPO’s) for each process, specifies recovery procedures sufficient to restore process functionality within RTO’s, and then validates all procedures via testing.

05 Nov 2013

Hot Topics Tom Hinkel 0 comment

The OCC Sets a New Standard for Vendor Management…

…but will it become the new standard for institutions with other regulators? UPDATE – The answer is yes, at least for the Federal Reserve. Readers of this blog know that I’ve been predicting an increase in vendor management program scrutiny since early 2010. And although the FFIEC has been very active in this area, issuing multiple updates to outsourcing guidance in the past 2 years, it appears that the OCC is the first primary federal regulator (PFR) to formalize it into a prescriptive methodology.

So if you are a national bank or S&L regulated by the OCC, what you’ll want to know is “what changed”? They’ve been looking at your vendor management program for years as part of your safety & soundness exams, exactly what changes will they expect you to make going forward? The last time the OCC updated their vendor management guidance was back in 2001, so chances are you haven’t made many substantial changes in a while. That will change.

However if you are regulated by the FDIC or the Federal Reserve or the NCUA, so what? Nothing has changed, right? Well no…not yet anyway. Except for a change adding a “Vendor Management and Service Provider Oversight” section in the IT Officer’s Questionnaire back in 2007, the FDIC hasn’t issued any new or updated guidance since 2001. Similarly, the NCUA last issued guidance in 2007 but it was really a re-statement of existing guidance that was first issued in 2001. So considering the proliferation of outsourcing in the last 10 years, I believe all of the other regulators are overdue for updates. Furthermore, I believe the OCC did a very good job with this guidance, and all financial institutions regardless of regulator would be wise to take a close look.

So what’s changed? I compared the original 2001 bulletin (OCC 2001-47) side-by-side with the new one (OCC 2013-29), and although most of the content was very similar, there were some significant differences. Initially they both start out the same way; stating that banks are increasing both the number and the complexity of outsourced relationships. But the updated guidance goes on to state that…

“The OCC is concerned that the quality of risk management over third-party relationships may not be keeping pace with the level of risk and complexity of these relationships.”

They specifically cited failure to assess the direct and indirect costs, failure to perform adequate due diligence and monitoring, and multiple contract issues, as troublesome trends.

Conceptually, the new guidance focuses around a 5-phase “life-cycle” process of risk management. The life-cycle consists of:

Planning,
Due diligence and third-party selection,
Contract negotiation,
Ongoing monitoring, and
Termination

First of all, a “cycle” concept strongly suggests that a once-a-year approach to program updates is not sufficient. Secondly, I think the planning, or pre-vendor, phase is potentially the most significant in terms of the changes that regulators will expect going forward. For one thing, beginning the vendor management process BEFORE beginning the relationship (i.e. before the vendor becomes a vendor) seems like a contradiction in terms (although it is not entirely new to readers of this blog), so many institutions may have skipped this phase entirely. But it is at this planning stage that elements like strategic justification and complexity and impact on existing customers are assessed. Those are only a few of the considerations in the planning phase, the guidance lists 13 in all.

The due diligence and contract phases are clearly defined and also expanded, but fairly consistent with existing guidance*. And although termination is now defined as a separate phase, the expectations really haven’t changed much there either.

On-going monitoring (the traditional oversight phase) has been greatly expanded however. The original guidance had 3 oversight activities; the third party’s financial condition, its controls, and the quality of its service and support. The new guidance still has those 3…and adds 11 more. Everything from insurance coverage, to regulatory compliance, to business continuity and managing customer complaints.

But perhaps the biggest expansion of expectations in the new guidance is the banks’ responsibility to understand how the vendor manages their subcontractors. Banks are expected to…

“Evaluate the third party’s ability to assess, monitor, and mitigate risks from its use of subcontractors and to ensure that the same level of quality and controls exists no matter where the subcontractors’ operations reside.” (Bold added)

Shorter version: “Know your vendor…and your vendor’s vendor”. And this expectation impacts all phases of the risk management life-cycle. Subcontractor concerns start in the planning stage, continue through due diligence and contract considerations, add control expectations to on-going monitoring, and even impact termination considerations.

In summary, everything expands. Your pre-vendor & pre-contract due diligence expands, oversight requirements (and the associated controls) increase, and of course everything must be documented…which also expands! The original guidance listed 5 items typically contained in proper documentation, the updated guidance lists 8 items. But it’s the very first item on the list that caught my attention because it would appear to actually re-define a vendor. Originally the vendor listing was expected to consist of simply “a list of significant vendors or other third parties”, which, depending on the definition of “significant”, was a fairly short list for most institutions. Now it must consist of “a current inventory of all third-party relationships”, which leaves nothing to interpretation and expands your vendor list considerably.**

So if you are regulated by the OCC you can expect these new requirements to be incorporated into the examination process fairly soon. If not, use this as a wake-up call. I think you can expect the other federal regulators to follow suit with their own revised guidance. The OCC has just set the gold standard. Use this opportunity to get ahead of your regulator by revisiting and enhancing your vendor management program now.

* Safe Systems customers can get updated due diligence and contract checklists from their account manager.

** All vendors on the list must be risk assessed, and although the risk categories didn’t change (operational, compliance, reputation, strategic and credit) some of the risk elements did. Matt Gunn pointed out one of the more interesting changes in his recent TechComply post. I’ll cover that and others in a future post.

20 Aug 2013

Ask the Guru, From the Field Tom Hinkel 2 Comments

Ask the Guru: Vendor vs. Service Provider

Hey Guru
I recently had an FDIC examiner tell me that we needed to make a better distinction between a vendor and a service provider. His point seemed to be that by lumping them together in our vendor management program we were “over-analyzing” them. He suggested that we should be focused instead only on those few key providers that pose the greatest risk of identity theft. Our approach has always been to assess each and every vendor. Is this a new approach?

I don’t think so, although I think I know where the examiner is coming from on the vendor vs. service provider distinction. First of all, let’s understand what is meant by a “service provider”. The traditional definition of a service provider was one who provided services subject to the Bank Service Company Act (BSCA), which dates back to 1962. As defined in Section 3 of the Act, these services include:

“…check and deposit sorting and posting, computation and posting of interest and other credits and charges, preparation and mailing of checks, statements, notices, and similar items, or any other clerical, bookkeeping, accounting, statistical, or similar functions performed for a depository institution.”

But lately the definition has expanded way beyond the BSCA, and today almost anything you can outsource can conceivably be provided by a “service provider”. In fact according to the FDIC, the products and services provided can vary widely:

“…core processing; information and transaction processing and settlement activities that support banking functions such as lending, deposit-taking, funds transfer, fiduciary, or trading activities; Internet-related services; security monitoring; systems development and maintenance; aggregation services; digital certification services, and call centers.”

Furthermore, in a 2010 interview with BankInfoSecurity, Don Saxinger (Team Lead – IT and Operations Risk at FDIC) said this regarding what constitutes a service provider:

“We are not always so sure ourselves, to be quite honest…but, in general, I would look at it from a banking function perspective. If this is a function of the bank, where somebody is performing some service for you that is a banking function or a decision-making function, including your operations and your technology and you have outsourced it, then yes, that would be a technology service that is (BSCA) reportable.”

Finally, the Federal Reserve defines a service provider as:

“… any party, whether affiliated or not, that is permitted access to a financial institution’s customer information through the provision of services directly to the institution. For example, a processor that directly obtains, processes, stores, or transmits customer information on an institution’s behalf is its service provider. Similarly, an attorney, accountant, or consultant who performs services for a financial institution and has access to customer information is a service provider for the institution.”

And in their Guidance on Managing Outsourcing Risk…

“Service providers is broadly defined to include all entities that have entered into a contractural relationship with a financial insitiution to provide business functions or activities”

So access to customer information seems to be the common thread, not necessarily the services provided. Clearly the regulators have an expanded view of a “service provider”, and so should you. Keep doing what you’re doing. Run all providers through the same risk-ranking formula, and go from there!

One last thought…don’t get confused by different terms. According the the FDIC as far back as 2001, other terms synonymous with “service providers” include vendors, subcontractors, external service provider (ESPs) and outsourcers.

04 Jun 2013

Hot Topics Tom Hinkel 1 Comment

Incident Response in an Outsourced World

UPDATE – On June 6th the FFIEC formed the Cybersecurity and Critical Infrastructure Working Group, designed to enhance communications between and among the FFIEC members agencies as well as other key financial industry committees and councils. The goal of this group will undoubtedly be to increase the defense and resiliency of financial institutions to cyber attacks, but the question is “what effect will this have on new regulatory requirements and best practices”? Will annual testing of your incident response plan be a requirement, just as testing your BCP is now? I think you can count on it…

I’ve asked the following question at several recent speaking engagements: “Can you remember the last time you heard about a financial institution being hacked, and having its information stolen?” No responses. I then ask a second question: “Can anyone remember the last time a service provider was hacked, and financial institution data stolen?”. Heartland…TJX…FIS…almost every hand goes up.

As financial institutions have gotten pretty good at hardening and protecting data, cyber criminals are focusing more and more on the service providers as the weak link in the information security chain. And wherever there are incidents making the news, the regulators are sure to follow with new regulations and increased reinforcement of existing ones.

The regulators make no distinction between your responsibilities for data within your direct control, and data outside your direct control;

“Management is responsible for ensuring the protection of institution and customer data, even when that data is transmitted, processed, stored, or disposed of by a service provider.” (Emphasis added)

In other words, you have 100% of the responsibility, and zero control. All you have is oversight, which is at best predictive and reactive, and NOT preventive. So you use the vendor’s past history and third-party audit reports to try to predict their ability to prevent security incidents, but in the end you must have a robust incident response plan to effectively react to the inevitable vendor incident.

The FFIEC last issued guidance on incident response plans in 2005 (actually just an interpretation of GLBA 501b provisions), stating that…

“…every financial institution should develop and implement a response program designed to address incidents of unauthorized access to sensitive customer information maintained by the financial institution or its service provider.” (Emphasis added)

The guidance specified certain minimum components for an incident response plan, including:

Assessing the nature and scope of an incident and identifying what customer information systems and types of customer information have been accessed or misused;
Notifying its primary federal regulator as soon as possible when the institution becomes aware of an incident involving unauthorized access to or use of sensitive customer information;
If required, filing a timely SAR, and in situations involving federal criminal violations requiring immediate attention, such as when a reportable violation is ongoing, promptly notifying appropriate law enforcement authorities;
Taking appropriate steps to contain and control the incident to prevent further unauthorized access to or use of customer information; and
Notifying customers when warranted in a manner designed to ensure that a customer can reasonably be expected to receive it.

The guidance goes on to state that even if the incident originated with a service provider the institution is still responsible for notifying their customers and regulator. Although they may contract that back to the service provider, I have personally not seen notification outsourcing to be commonplace, and in fact I would not recommend it. An incident could carry reputation risk, but mishandled regulator or customer notification could carry significant regulatory and financial risks. In other words, while the former could be embarrassing and costly, the latter could shut you down.

So to summarize the challenges:

Financial institutions are outsourcing more and more critical products and services.
Service providers must be held to the same data security standards as the institution, but…
…the regulators are only slowly catching up, resulting in a mismatch between the FI’s security, and the service provider’s.
Cyber criminals are exploiting that mismatch to increasingly, and successfully, target institutions via their service providers.

What can be done to address these challenges? Vendor selection due diligence and on-going oversight are still very important, but because of the lack of control, an effective incident response plan is the best, and perhaps only, defense. Yes, preventive controls are always best, but lacking those, being able to quickly react to a service provider incident is essential to minimizing the damage. When was the last time you reviewed your incident response plan? Does it contain all of the minimum elements listed above? Better yet, when was the last time you tested it?

Just as with disaster recovery, the only truly effective plan is one that is periodically updated and tested. But unlike DR plans, most institutions don’t even update their incident responses plans, let alone test them. And while there are no specific indications that regulators have increased scrutiny of incident response plans just yet, I would not be at all surprised if they do so in the near future. Get ahead of this issue now by updating your plan and testing it. Use a scenario from recent events, there are certainly plenty of real-world examples to choose from. Gather the members of your incident response team together and walk through your response, the entire test shouldn’t take more than an hour or so. Answer the following questions:

What went wrong to cause the incident? Why (times 5…root cause)? If this is a vendor incident, full immediate disclosure of all of the facts to get to the root cause may be difficult, but request them anyway…in writing.
Was our customer or other confidential data exposed? If so, can it be classified as “sensitive customer information“?
Is this a reportable incident to our regulators? If so, do we notify them or does the vendor? (Check your contract)
Is this a reportable incident to our customers? How do we decide if “misuse of the information has occurred or it is reasonably possible that misuse will occur“?
Is this a reportable incident to law enforcement?
What if the incident involved a denial of service attack, but no customer information was involved? A response may not be required, but should you?
What can we do to prevent this from happening again (see #1), and if we can’t prevent it, are there steps we should take to reduce the possibility? Can the residual risk be transferred?

Make sure to document the test, and then test again the next time an incident makes the news. It may not prevent the next incident from involving you, but it could definitely minimize the impact!

NOTE: For more on this topic, Safe Systems will be hosting the webinar “How to Conduct an Incident Response Test” on 6/27. The presentation will be open to both customers and non-customers and is free of charge, but registration is required. Sign up here.

01 Nov 2012

Hot Topics Tom Hinkel 0 comment

FFIEC Updates Technology Service Provider Guidance

Just posted, the new Booklet rescinds and replaces the previous one issued in March 2003, and is the first Booklet replacement since Retail Payment Systems in 2010. In general this is not so much a complete re-write as a reinforcement of the importance the agency places on strong vendor management, which is a concept that we’ve seen from other recent FFIEC releases and updates.

So with all the similarity between this new publication and the one almost 10 years ago, I think it’s instructive to focus on the differences between the two to see how the FFIEC’s thinking has evolved. It also allows the institutions affected to know exactly what they need to change or adjust to remain in compliance.

First of all, both Booklets state the following:

“A financial institution’s use of a TSP (technology service provider) to provide needed products and services does not diminish the responsibility of the institution’s board of directors and management to ensure that the activities are conducted in a safe and sound manner and in compliance with applicable laws and regulations…”

and perhaps for extra emphasis the new Booklet adds the following verbiage:

“…just as if the institution were to perform the activities in-house.”

Nothing new here, all institutions are acutely aware that they bear full responsibility for the confidentiality, integrity and availability of their customer’s data regardless of where it may reside. This re-statement is perhaps insignificant by itself, but interesting when taken in combination with the next sentence:

Old guidance – “Financial institutions should have a comprehensive outsourcing risk management process to govern their TSP relationships.”

New guidance – “Agencies expect financial institutions to have a comprehensive, enterprise risk management process in place that addresses vendor management for their relationships with TSPs.”

What is significant here is the addition of the word “enterprise” to the risk management process, indicating that you must acknowledge that vendors carry multidimensional risks. These risks include not just operational risk (risk of failure), but strategic risk, regulatory risk and reputation risks as well.

However to me the most significant change in the guidance is in the sentence beginning with “…(the risk management) process should include…”, because this is what the regulators will expect from you. Compare these:

Old guidance – “Such processes should include risk assessment, selection of service providers, contract review, and monitoring of service providers.”

New guidance – “The risk management process should include risk assessments and robust due diligence for the selection of TSPs, contract development, and ongoing monitoring of all TSPs’ performance.”

It’s clear that regulators will expect much more from your vendor risk management process going forward. Not simply selecting a service provider, but robust due diligence in the selection process. Not just contract review, but contract development. And not just basic monitoring, but ongoing monitoring of all TSP’s performance.

The new guidance goes on to state that federal regulators expect technology service providers to be familiar with, and adhere to, not just this Booklet, but all 11 Booklets in the IT Examination Handbook series. One more reinforcement that there are not 2 standards of measurement…one for financial institutions and one for vendors…but only one. And that one same standard will be enforced by the same federal regulators that currently examine you.

The guidance goes on to describe how they will classify service providers (by size and criticality of the services they provide), and how that classification will determine who will examine them, and how often they can expect to be examined. As far as who can expect to be examined, any service provider that provides any of the following services:

Core application processors
Electronic funds transfer switches
Internet banking providers
Item processors
Managed security servicers
Data storage servicers
Business continuity providers

So pretty much anyone that provides an application, system, or process that is vital to the successful continuance of a critical business activity, or anyone that interfaces with a critical business system, can expect to be examined.

Aside from being more comprehensive, the actual examination process hasn’t changed much. Examiners will still scrutinize the AMDS (Audit, Management, Development and Acquisition, and Support and Delivery) components, and will still assign a 1 through 5 numerical score to each component with 1 representing the highest or best, and 5, the lowest rating or worst. Examiners will then use the component scores to determine the overall composite rating. Again, nothing new there.

So in summary, not a drastic change as much as a reiteration with amplification and clarification. Simply put, more of the same…more regulatory expectations for your vendor management program, which means more scrutiny by the examiners (for you and for your vendors), all of which means more effort on everyone’s part!

09 Apr 2012

Hot Topics Tom Hinkel 1 Comment

FFIEC Handbook Update – Outsourcing

The FFIEC has just added a section to the Outsourcing Technology Services IT Examination Handbook, and it should be required reading for financial institutions as well as any managed service providers. The new section is Appendix D: Managed Security Service Providers, and it is the first significant change to the Handbook since it was released in 2004. It addresses the fact that because of the increasing sophistication of the threat environment, and the lack of internal expertise, a growing number of financial institutions are (either partially or completely) outsourcing their security management functions to unaffiliated third-party vendors.

Because of the critical and sensitive nature of these security services, and the loss of control when these services are outsourced, the guidance stresses that institution must address additional risks beyond their normal vendor management responsibilities. Specifically, more emphasis must be placed on the contract and on oversight of the vendor’s processes, infrastructure, and control environment.

The most interesting addition to the guidance for me is the “Emerging Risks” section, which is the first time the FFIEC has addressed cloud computing. Although it is addressed from the perspective of the service provider, it defines cloud computing this way:

“…client users receive information technology services on demand from third-party service providers via the Internet “cloud.” In cloud environments, a client or customer will relocate their resources such as data, applications, and services to computing facilities outside the corporate firewall, which the end user then accesses via the Internet.”

Any data transmitted, stored or processed outside the security confines of the corporate firewall is considered higher risk data, and must have additional controls. This would seem to infer that data in the cloud should be classified differently in your data-flow diagram, and have a correspondingly higher protection profile.* It will be interesting to see if this will be the FFIEC’s approach when and if they address cloud computing in the future.

The guidance also has a useful MSSP Engagement Criteria matrix that institutions can use to evaluate their own service providers, as well as a set of MSSP Examination Procedures, which service providers (like mine) can use to prepare for future examinations. In summary, financial institutions would be wise to familiarize themselves with the new guidance, after all to quote from the last line;

“As with all outsourcing arrangements FI management can outsource the daily responsibilities and expertise; however, they cannot outsource accountability.”

* A protection profile is a description of the protections that should be afforded to data in each classification.

12 Next →