31 May 2011

Hot Topics Tom Hinkel 0 comment

Time to re-think the role of the network administrator?

Traditionally, the network administrator needed to operate at “ground-level”. Network maintenance was highly specialized and problematic, requiring a constant hands-on approach. And in the very early days (when the Guru started… “he who speaks of floppy disks”…) there were few formal training classes, most of what you learned was by trial and error…lots of error!

Today’s network administrator still has plenty of trial and error learning, but there is much less of it then there used to be. Consider this:

How important is the Internet to your problem resolution process? Can you even imagine doing your job without the Internet?
Colleges, universities and technical schools have had formal degree programs for training network administrators for years, assuring that even the most inexperienced admin has a broad base of knowledge to draw from starting on day one.
Although server and desktop operating systems and applications are more complex today, they are also much (somewhat?) more stable than they used to be, with much more mature, feature-rich, administrative interfaces.
Largely because of the first three items, there are a lot more resources available for support today. It’s often more cost effective to reach out to an expert in a particular area then it is to spend hours on trial and error.
Most importantly though, many of the routine administrative tasks can now be automated and/or outsourced (patch management, AV updates, etc.), removing not only the drudgery of the task, but also removing the uncertainty of the human element as well. And as I’ve written about here, both auditors and examiners prefer automated controls for that very reason.

So the focus of the network administrator’s job has really evolved from a hands-on, high-touch, ground-level role, into more of a higher-level, managerial role. They still have primary responsibility in their traditional role of (as the FFIEC states), “…implementing the policies, standards, and procedures in their day-to-day operational role”, but they now often assist in the development of those policies as well. Most admins also sit on the IT steering committee, and in that capacity they also have the shared responsibility to coordinate the IT strategic plan, and by extension, the overall strategic plan. But it’s difficult to have an enterprise-wide view if you’re stuck in the trenches unlocking a user account or struggling with AV or patch updates.

Given the right tools, today’s network administrator is able to add value in many ways. Furthermore I don’t know of a single one that wouldn’t jump at the chance to assume a higher profile in their organization (with the associated increase in net value). If you are a network administrator, here are 3 ways you could get the conversation started:

“You know, regulators are increasingly focusing on reporting to verify that we are following our procedures. Give me the tools I need to gather, analyze and report, and I’ll be able validate that we’re doing what we say we’re doing, the way we say we’re doing it. This should reduce our exposure to future regulatory findings.”
“Recent experience has shown that auditors and examiners really prefer automated tools for routine repetitive tasks. An added advantage is that this will free me up to manage the process from a slightly different perspective, allowing me to not only apply controls, but assess their effectiveness as well.”
“Effectively managing strategic risk means providing management with timely, actionable information that will allow them to rapidly respond and react to changes in the information landscape. Include me in the strategic planning process and I’ll be able to better understand the mission, and deliver the right information in the right format at the right time.”

And if you manage a network administrator and you haven’t had this discussion yet, don’t wait for them to approach you…ask them what you can do to elevate them above the drudgery, and get them more involved in managing the process instead of drowning in it. I’ll bet you’ll find a source of value you never knew you had!

17 May 2011

From the Field Tom Hinkel 0 comment

The IT Strategic Plan – Why, Who, & How

One of the most common examination findings recently (particularly with the FDIC) has been the lack of an IT Strategic Plan. I’m not sure why the focus lately (perhaps the shift from the CAMELS “A” to the “M”?), but the concept is certainly not new. The regulatory mandate for it is found in the 2004 FFIEC Management Handbook:

“The Board of Directors and management should* implement an IT planning process that:

Aligns IT with the corporate wide strategic plan;
Aligns IT strategically and operationally with business units;
Maintains an IT infrastructure to support current and planned business operations;
Integrates IT spending into the budgeting process and weighs direct and indirect benefits against the total cost of ownership of the technology; and
Ensures the identification and assessment of risk before changes or new investment in technology.”

The first requirement of an effective IT planning process is alignment with the overall strategic plan, yet whenever I ask a group of financial professionals how many have seen their own strategic plan, very few hands go up. I get more hands in a group of senior management than in a group of network administrators, which seems to make sense except for one thing; the administrators are the ones actually maintaining the IT infrastructure (#3 above). So the very folks tasked with making sure the infrastructure is aligned with the overall strategic plan, probably haven’t even seen it!

This is the real disconnect from my perspective. Although you can develop an IT Strategic Plan from a template fairly quickly using standardized verbiage, integrating it into the overall plan, and then executing it, is much trickier. It should be a live document, linking the overall Strategic Plan with IT projects and issues through the IT Steering Committee. In fact, the FFIEC even suggests that the IT Steering Committee is the ideal forum for this, stating that the committee:

“…may also oversee the development and maintenance of the IT strategic plan.”

And furthermore,

“The information technology steering committee’s cross-functional membership makes it well suited for balancing or aligning the organization’s IT investment with its strategic and operational objectives.”

So the Management Handbook strongly suggests that IT steering is the best forum, and that everyone from the Board of Directors, to IT line management, to business unit management should participate. But this brings us back to dilemma I mentioned above; that IT line management (and most business unit management, for that matter) are rarely familiar enough with the overall strategic plan to effectively affect alignment. This brings us to the “how”:

Step 1 – Senior management must communicate the mission,
Step 2 – Ensure that the IT Committee is tasked with implementing that mission by making sure all IT initiatives support and enhance the missions’ goals and objectives.
Step 3 – Most importantly, make sure the committee has the tools and expertise necessary to effectively monitor, gather, analyze and report the data that will document the entire process. Because in the end…

“…institutions that are better at keeping IT aligned with changing business goals and objectives are positioned to compete more effectively.”

*In FFIEC-speak, “should” really translates to “must”.

11 May 2011

From the Field Tom Hinkel 0 comment

Using Technology to Drive Compliance

In the past year to year and a half, nearly all of the IT examination findings I’ve seen have in the broad category of “documentation”, or more specifically, lack thereof. In other words, policies and procedures were satisfactory, but documentation was either non-existent, or insufficient, to demonstrate that actual practices followed policy and procedure.

To visualize this, consider that the compliance process consists of three overlapping areas of endeavor:

Written polices begin the process, which must always have regulatory guidance as their target. Policies should track guidance precisely; if guidance states that you should or must do something, your policies should state that you do, or you will.

If policies are “what” you do, written procedures are the “how”. And just as policies align with guidance, procedures should flow logically from, and align with, your policies. For example, your information security policy states (among other things) that you will protect the privacy and security of customer information. Your procedures contain the detailed steps (or controls) that you will take to prevent, detect and correct unauthorized access to, or use of, customer information. Controls like securing the perimeter of your network, updating server and workstation patches, installing and updating Anti-virus, etc.

So you have the “what” and the “how”, but as I mentioned previously, the vast majority of audit and examination findings in the past couple of years were due to deficiencies in the third area; actual (documented) practices. And this is where technology can be of tremendous assistance.

[pullquote]Auditors and examiners much prefer automated systems over manual. Automated systems don’t forget, or get too busy, or take vacations or sick days. They aren’t subject to human error or inconsistencies. [/pullquote]

Auditors and examiners much prefer automated systems over manual. Automated systems don’t forget, or get too busy, or take vacations or sick days. They aren’t subject to human error or inconsistencies. In fact, some processes like firewall logging, normalization, and analysis are virtually impossible to implement manually because of the sheer volume of data generated by these devices.* And while other areas like patch management and Anti-virus updates are possible to implement manually, auditors much prefer automated processes because they ensure polices are applied in a consistent and timely manner.

But perhaps the biggest boost to your compliance efforts from technology is in the area of reporting, and specifically, automated reporting. In today’s compliance environment, if you can’t prove you’re following your procedures, the expectation from the examiners is that you aren’t. This is the one area that has evolved more than any other in the past couple years. And automated reporting provides that documentation without human intervention, easing the burden on the network administrator. Auditors (internal and external) and examiners also like automated reporting because they have a higher confidence in the integrity of the data. And the IT Steering Committee likes it because it is much easier to review and approve reports prepared and presented in a standardized format.

So in summary, technology enables automation, and automation enhances compliance. And along the way everyone from the Board of Directors, to management committees, to the network administrator, benefits from it.

* The FDIC IT Officer’s Pre-Examination Questionnaire validates the difficulty of manual processes to manage logs when it asks:

“Do you have a formal intrusion detection program, other than basic logging (emphasis mine), for monitoring host and/or network activity”

20 Apr 2011

Hot Topics Tom Hinkel 0 comment

A Recurring Theme in FDIC Consent Orders

If you look at any of the recent FDIC Consent Orders, you will quickly see a common theme. I randomly pulled a few off the top of the list, and the verbiage was very similar, and in many cases identical:

…the Board shall enhance its participation in the affairs of the Bank
…the Bank’s board of directors shall increase its participation in the affairs of the Bank
…the Board shall participate fully in the oversight of the Bank’s compliance management system
…the Board shall participate fully in the oversight of the Bank’s Compliance Management System
…the Board shall increase its participation in the affairs of the Bank
…the Bank shall have and retain qualified management
…Bank’s board of directors shall increase its participation in the affairs of the Bank
…the Bank shall have and retain qualified management.
…the Board shall increase its participation in the affairs of the Bank
…the Bank’s board of directors (“Board”) shall increase its participation in the affairs of the Bank

In almost every case, regardless of the main thrust of the Consent Order, this was usually the first requirement. In other words, although the Order may have been imposed because of financial weakness, or lending policy non-conformance, or some other reason, the examiners want to establish up front that the Board and Senior Management are at fault for failing to prevent, detect, and/or correct the problem ahead of time. Furthermore, regardless of their past participation, in every case they are expected to increase their oversight in the future.

Of course, not only must this occur, but it must also be documented. If recent examination experience has taught us anything, it is that if you don’t have it documented, it didn’t happen. The challenge is this; typically the Board defines the broad goals and objectives of the institution in the strategic plan, and delegates the day-to-day responsibility of implementing those goals to committees. In a perfect world, the mandates flow down from the Board to the committees, and status reporting flows back up from the committees to the Board. (Graphic illustration) In reality, there are multiple points of failure in this top-down, bottom-up model:

Does the Board have a well-defined, 3-5 year Strategic Plan?
Has this plan been communicated to all stakeholders?
Have committees been formed, staffed, and tasked with implementing the details of the plan?
Are there well-defined objectives and benchmarks in place to measure alignment between strategic goals and actual performance?
Does the Board have access to adequate, timely information (reporting), and the necessary expertise, to determine if their strategic goals and objectives are being achieved?

A “No” answer at any point in this process causes the whole process to break down. And even a “Yes, but we didn’t document it…”, is not enough to satisfy the examiners. So how best to document each step? Taken in order from above:

Make sure the institution has a valid, up-to-date, Strategic Plan, and…
…the plan has been communicated to all stakeholders. This isn’t as onerous as it sounds…the plan shouldn’t change that often.
The mission statement for all committees should reinforce their alignment and coordination with the Strategic Plan, and any risk assessments conducted by the committees must measure strategic risk.
Evaluate each new product, service and vendor against its ability to further the objectives of the Strategic Plan, and…
…make sure this information is summarized and presented to the Board at a frequency commensurate with the pace of change within the institution.

As I’ve mentioned before, the Tech Steering Committee is the ideal committee to report all things IT to the Board. If you utilize a standard agenda, which includes discussion of on-going or proposed IT initiatives (and their alignment with Strategy), document the meetings, and report progress to the Board periodically, you will satisfy the IT oversight requirement. Once the top-down and bottom-up process is in place for IT, simply duplicate it across the enterprise!

08 Apr 2011

From the Field Tom Hinkel 0 comment

“Concentration of duties”

It is not unusual for a community financial institution with limited personnel to have the Information Security Officer (ISO) act as a backup network administrator. In fact, this is a relatively common practice in an environment where key personnel will typically wear several hats. And there are practical reasons for this; the ISO is typically tech-savvy, and can act as an expedient resource when needed. Often when admin (or privileged) access is required, it is for a business critical purpose.

However, we have received several post-exam reports recently that examiners are taking a closer look at this practice. The finding is called “concentration of duties” (or sometimes “separation of duties”), and it addresses the very legitimate concern that the ISO must act in an oversight capacity to the network administrator, and that oversight dynamic is lost if the ISO has administrative capabilities. In fact in their Information Technology Officer’s Questionnaire, the FDIC requires you to “…briefly describe any known conflicts or concentrations of duties” . This oversight dynamic is exactly what they are referring to.*

If your institution engages in this multiple-hat practice, there are several things you can do to address this with the regulators. The first is to transfer the administrative oversight responsibilities from the ISO to a committee, typically the audit or tech steering committee. This requires more frequent meetings (preferably monthly, but no less than quarterly), and a strict adherence to an agenda that always includes discussion (and documentation) of rights and permission changes whether or not there were any. You may also want to consider event log monitoring software that can collect and aggregate all administrative user activity, and preferably store it on a logically separate system.

It’s also a good idea to have the committee review and re-approve all privileged accounts at each meeting. Another best practice is to make sure the ISO has a user account for administrative activities separate from their everyday user account. This assures that all activity is properly captured and reported. Finally, never share log in credentials…particularly admin accounts.

Also, review the section on privileged user access from the FFIEC IT Examination Handbook, Information Security Booklet, Page 23:

Authorization for privileged access should be tightly controlled. Privileged access refers to the ability to override system or application controls. Good practices for controlling privileged access include

Identifying each privilege associated with each system component,
Implementing a process to allocate privileges and allocating those privileges either on a need-to-use or an event-by-event basis,
Documenting the granting and administrative limits on privileges,
Finding alternate ways of achieving the business objectives,
Assigning privileges to a unique user ID apart from the one used for normal business use,
Logging and auditing the use of privileged access,
Reviewing privileged access rights at appropriate intervals and regularly reviewing privilege access allocations, and
Prohibiting shared privileged access by multiple users.

Incorporate these best practices into your access rights administration process. In the end, what’s expected is that you understand the risk of “concentration of duties”, and balance that against your business needs, given your size and complexity and the nature and scope of your operations. If you understand the residual risk, and believe your business needs are best met by sharing admin duties with your ISO, make sure your examiner knows how you got to that decision, and how you plan to manage it going forward.

*Note – Although you may be tempted to answer “No” to this question in order to avoid drawing attention to it, you are much better off responding “Yes”, and then describing your risk assessment process and resulting controls. It may not prevent the finding, but you will have a proactive response to it, which almost always implies more effective risk management.

04 Apr 2011

From the Field Tom Hinkel 8 Comments

The Control Self-Assessment (CSA)

If there was a process that was mentioned 43 times in 7 of the 12 FFIEC IT Examination Handbooks, (including 12 times in the Information Security Handbook alone!), would you consider implementing it? How about if it virtually assured better audits and examinations? OK, you’re interested, but the last thing you need is to implement another complicated process, right? What if the framework is probably already in place at your institution, and all you need to do is fine-tune it a bit?

I’m referring to the Control Self-Assessment (CSA), and let’s first make the regulatory case for it. The FFIEC Operations Handbook says:

Periodic control self-assessments allow management to gauge performance, as well as the criticality of systems and emerging risks.

And…

Senior management should require periodic self-assessments to provide an ongoing assessment of policy adequacy and compliance and ensure prompt corrective action of significant deficiencies.

If you’re familiar with “FFIEC-speak”, then you know that “should” really translates to “must”. But the Information Security Handbook makes the most compelling argument for utilizing the CSA in your risk management program:

Control self-assessments validate the adequacy and effectiveness of the control environment. They also facilitate early identification of emerging or changing risks.

So there is plenty of regulatory support for the CSA process, what about the audit and exam benefits? All of the major auditing standards bodies (IIA, AICPA, ISACA) address the importance of internal control reviews. Indeed most auditors say that institutions with an internal CSA process in place generally demonstrate a much more evolved risk management process, resulting in fewer, and less severe, audit findings. This stands to reason, as they tend to identify, and correct, control weaknesses prior to audit, as opposed to waiting for the auditor to identify them. And since one of the first things the examiner wants to see when they come in is your most recent audit, this often results in fewer examination findings as well.

One more reason to implement a CSA process from the examination perspective is something I touched on here…for those institutions trying to maximize their CAMELS IT composite ratings, one of the biggest differentiators between a “1” and a “2” is that in institutions rated a “1” “…management identifies weaknesses promptly (i.e. internally) and takes appropriate corrective action to resolve audit and regulatory concerns”. Conversely, in those institutions rated a “2” “…greater reliance is placed on audit and regulatory intervention to identify and resolve concerns”. A CAMELS “3” rating speaks directly to the CSA, stating that “…self-assessment practices are weak…“.

OK, so there are certainly lots of very good reasons to implement a CSA process in your institution. How can this be done with minimal disruption and the least amount of resource overhead? Chances are you already have a Tech Steering Committee, right? If the committee consists of members representative of all functional units within the organization, it has the support of senior management, and is empowered to report on all risk management controls, all that’s needed is a standardized agenda to follow. The agenda should address the following concerns:

Identification of risks and exposures
Assessment of the controls in place to reduce risks to acceptable levels
Analysis of the gap between how well the controls are working, and how well management expects them to work

As you can see, this is not substantially different from what you are probably already doing in your current Tech Steering Committee meetings. In fact, this list is really only a sub-set of your larger agenda…the only possible difference is that any and all findings in the gap analysis must be assigned to a responsible party for remediation.

In summary; the FFIEC strongly encourages it, the auditors and examiners love it, and for most institutions it’s not too difficult to implement and administer. But if you only need one good reason to consider the CSA process, it should be this:

Improved audit and examination ratings!

12 Next →