Guide – Cloak & Cyber

Choosing a consultant can seem like an overwhelming task. Throw in the "cybersecurity" concept and then it can get downright mystical. This article aims to help the reader select a good consultant out of a sea of potential confusion. Some research and selection criteria wrapped in a process will help anyone choose a cybersecurity consultant with strong character and expertise that meets the needs of the moment. We often go into situations hoping to make a clear decision but end up deciding on something less rational. In the following sections we look at why clients fire consultants, end up hiring consultants (despite otherwise good intentions), and how a simple process eliminates a lot of hand wringing. Jump Right to the Selection Process ResearchInterviewAnalysis and Follow-on InterviewBudget Considerations and Selection Choosing a Cybersecurity Consultant According to the Oxford Review, there are 8 reasons an organization looks for a consultant (or consulting firm): Specialist knowhow, experience, methods etc.Particular individual professionals or gurus.That they have access to specialist resources or other attributes.Diagnosis, problem formulation, or strategy formulation servicesExperience of a particular methodology.Pre-existing solution to a specific problem.Knowledge, skills and experience of implementing a particular solution.Ability to analyze, evaluate or capitalize on the results of a solution This boils down to three types of needs: The organization doesn't know something and needs the knowledge.The organization can't do something and needs the skillsThe organization knows a problem exists and needs deeper/clearer identification and solutions. These needs (reasons for going the consultant route) translate into two essential selection criteria when choosing cybersecurity consultant: Cybersecurity knowledge and/or technical skillsCybersecurity problem identification and solution prescription But, as they say, this is where the story gets interesting, because a criterion is not the same as a process. In Implementing Value Pricing, the author presents a survey' result listing three reasons for terminating a relationship with a consultant. Consultant doesn't treat client rightConsultant ignores the clientConsultant fails to cooperate That same survey lists the top reasons a consultant was selected: Interpersonal skillsAggressivenessInterest in the customerAbility to explain procedures Willingness to give advicePerceived honesty Notice criteria such as experience, expertise, technical skill, and problem-solving ability are not on either list? Many go into the selection process thinking they are looking for knowledge and deep technical skill sets but end up selecting for, and firing over, perceived interpersonal and communication skills. Part of the reason this happens (in theory at least) is the organization wants to select for skill but as soon as the conversations start, they are at a loss for an effective knowledge and skill selection process. Then the searching organization defaults to perceived interpersonal traits the may or may not be relevant. After all the "research" and conversations, the winning consultant is the one with the best pitch and price point. Then after the selection is concluded, there are a myriad of psychological phenomenon (ex. reinforcement bias) to make sure the choice was rational. That is until the consultant's shenanigans have worn thin, and the separation occurs (termination or no consideration for follow-on business). Then the client was wondering what went wrong. "After all, when I asked, they stated that they were in fact, good." Bottom line, most organizations looking for outside help in the form of a consultant already understand they need technical and interpersonal skills, it's a useable process that is lacking. So instead, they end up selecting on feelings and say it was based on reason. But don't beat yourself up, everyone does this with almost every decision. That is until a decision-making process is followed. Or in this case, a cyber consultant selection process is followed. Cybersecurity Consultant Selection Process After all is considered, an organization seeks one thing: success. Success defined here as an effective solution to the problem. Everything else is either secondary or a measure for success. When an organization goes into a consulting selection process, it's important to pick up on those signals for success. The four-step process proposed here is a means of taking measurement of those signals and making decision based on those measurement (rather than "feel"): ResearchInterviewAnalysis and Follow-on InterviewBudget Considerations and Selection Budget consideration as last for a good reason. Most (if not all firms) have options to fit almost any budget. However, before constricting a solution with a potentially inadequate budget it's a good idea to have conversations, get bids related to varying statements of work, and get a feel for likelihood of sucess. During this process make sure to take decent notes. The primary selection criteria are listed below: Focused Expertise and SkillExperienceReputationProcessMeasures of SuccessEffective communication of complex ideas (the interpersonal skill that get measured) Research Complete the research in 2 phases: generate an initial cyber consultant list and then reduce that list through reputation research. The research objective is to get an initial measure of focused expertise, and reputation. 1. Initial List: Gather together a list of cybersecurity consultants (and firms) with expertise in the space your problem exits. Consider this like brainstorming - no option too expensive (if prices are advertised), no firm is too big (or small), etc. - all that matters is the consultant offers the services and expertise to solve the problem. Need to test current level of security: penetration testing, vulnerability scanning, red teaming, etc.Just get breached: Indecent respond and forensics (eDisco)Want to build/install security controls: security architects and engineersHave no idea: cyber strategy development, general cyber security adviceOngoing security monitoring and response: MSSPs, and/or eXtended Response Services.. and dozens upon dozens more After the initial consultant/firm list hits twenty or so, reorder them according to focus. That is, consultants and firms offering a wide range of services get lower priority then consultants with focused services and expertise on the specific problem and/or in the organizations industry. This reorder won't guarantee the right candidate out of the gate, but it will increase your chances of finding the right fit with the right price sooner. [This is not a specialist vs generalist debate. This is about one firm courageous enough to stake a claim around a limited scope of expertise to the exclusion of other potential work, and another firm unwilling to focus long enough to become an expert at anything because they have to "know everything" or can't bear turning down work. Of course, there are firm with a vast array of expertise, but these firms are large, and expensive.] 2. Reputation Research: With the ordered list in hand, it's time to do some research. Read this short article on reputational research from the Legal Beagle blog. Other than the 5 main steps, the content won't get duplicated. Better Business BureauYelpWhois databaseLawsuit Search ("v [name]")EDGAR Database Here are a few extra ways to enrich the research through the addition of the following: Glassdoor Company Reviews: A job site offering employees the ability to complain (legit or otherwise) about companies. Look for both the good and the bad. Linkdn Search: Networking site offer the ability to do additional research on the firm and its employees.References: These aren't as important, given that a consultant is only going to provide references they believe will put them in a good light. However, if you gather references and happen to come across a disgruntled client (maybe even a current client looking to bail), that is something that should impact the final reputation score. When doing research, do research on the firm and the owners. Some of the larger firms got their start in a completely different consulting industry (business strategy, management, accounting, etc.) and later added a cybersecurity practice. The larger firm may have a good reputation, but the partner's reputation may be trash. With smaller firms, you run the risk of a hoodwinker try to sell snake oil expertise. Definitely do a "v [name]" search on the owners and partners. If something comes up, take a few minutes to read it and make sure it's relevant and the right person. Interview With a reduced list it's time to setup calls. This is the interview step but don't treat it like a job interview, where you are hiring an employee. This is more of a conversation where the goal is to gather signals of success and get a sense of their communication skills. The interview objective is to improve the measure of focused expertise and skill, gain an understanding of their process, and learn how they measure success on projects. Focused Expertise & Skill: Make sure the consultant grasps the issues at hand. Ask broad and penetrating questions to ferret out indicators of success or lack of knowledge. Example questions: How have you solved this particular problem before?What other problems have you solved?How did you solve those?What fundamental principles in security are relevant to my situation? Ask for examples and looks for specific details, pain points, moments of adulation, etc. Someone who has worked through a problem (and not just stood next to the person who worked the problem) will have battle scars. The answers will reflect those scars. Red flags are answers that are mealymouthed ("Confidentiality prevents me from sharing specifics"), loaded with esoteric industry terms ("Our FIM and DLP will prevent data exfil), or brush offs ("Trust me, I'm an industry expert"). Process: Ask about their process for dealing handling a client and for executing on a solution to the problem at hand. This isn't a request for their secret sauce. If you need a risk assessment, ask them what methodology they use (quantitative, qualitative). If it's a specific industry process like NIST, ask them why, when they diverge, and for what reasons. You're not looking for a workbook or walkthrough, get a sense they do in fact have a process they follow. If they get a follow-on interview, ask again, and make sure their answers match. Red flags are no process, they stumble into a process, or the process description is too different in the second conversation. Measures of Success: Find out how they measure project success. What key performance indicators (KPI) do they track across all clients and project types so they can ensure effective results and improve over time? The number of cybersecurity problems are legion (and sometime this side of demonic), but they can all get simplified into a few categories: Strategy and Risk: There are no plans or understanding of handling any cyber uncertainties.Controls: Lack of effective policies and technical countermeasures. Insight: No awareness of strategy or control effectiveness. Incident Response: Company has been breached and needs help. A good consultant will help identify high-level problems during the call. After the identification, when the consultant offers their services (I.E., potential solutions), ask them how they measure the success of that solution. Red flags are no KPIs or poor KPIs. No KPIs means they haven't thought much about what success means or improvement, bad KPIs incentive poor behavior. Effective Commnication of Complex Ideas: Do you understand anything they are saying? If you don't, it is not you're problem. If the consultant is unable to bring things from the complex and esoteric down to the understandable and digestible, then there is a major risk you end up with a result you can't understand or can't take action on. This isn't a measure of how well everyone gets along. Good consultants (and firm sales teams) know that the feeling a client walks away with from a meeting is more influential than any demonstration of technical prowess (see survey results from the Choosing a Consulting section above). Red flags are poor ability to summarize concepts, cyber industry process, problem solutions, etc. Pricing Model: This is an honorable mention and isn't a selection criterion, per se. Though if feelings are strong one way or another, put it on the list. It's mentioned here because part of any consulting selection process is understanding what everything will cost, and the pricing model used has a hidden effect on the final bill. The question here is what kind of pricing model the consultant will use to solve your problem: fixed fee or flexible fees. The pricing model indicates where the firm places the risk. There are different kinds of fixed-fee pricing - subscription, project based, value based, etc, - and it means the price is known at the beginning, and there are no surprises. Upfront sticker shock driving you away from better consultants is a potential problem. The firm takes more risk with fixed pricing because scoping is complex, therefore they build in price buffers on every project to make sure they can deliver on projects they underpriced (read: under scoped). Flexible pricing is more or less an "hours worked" model. You are billed for every hour the consultant, or every consultant at the firm who work on your project, works. This is open ended, and you have no idea of the final price. Unlike fixed pricing, you take all the pricing risk. There is no "penalty" for an under scoped project. Red flags here are an unwillingness to discuss pricing model. Keep in mind that a model is not that same a process. Many consultants have proprietary processes for producing a bid price. This process they believe, keeps them competitive. Analysis and Follow-on Interview This is where the information gathered from research and interviews is reviewed and organized (provided real-time organization wasn't taking place) and the list of any remaining candidates get scored. The objective is to narrow the selection down to three candidates. The three with the highest scores. These candidates are the ones most likely to solve the problem in an effective and measurable way. Each score is a range from 0 - 5 (poor to excellent) and entered into a matrix. Create a column for each candidate. Selection CriteriaStrider ConsultingOrc ConsultingFocused Expertise & Skill54Experience53Reputation40Processes51Measures of Success (KPIs)31Effective Communication30Total Score249Consultant Scoring Example The maximum score is 30. A 30 is an estimate of the best chance of success given the information at hand. It doesn't mean a 30 is without risk, just that it is difficult to detect. Going down from 30 increases the risk of failure. Failure here is a range from an ineffective solution to no solution. After the matrix is filled, setup follow-on calls with the top three scoring candidates to fill in any blanks, answer any nagging questions, and talk scope of work and price. [This is not an article on negotiation but one point on price talks is important. If after the consultant gives a bid price and you begin price negotiations, you have locked yourself out from other candidates as soon as either of you come to a price agreement. Do to otherwise is to negotiate in bad faith. If they do lower their price and it's acceptable, respond "I have to think about that, how long is the price good for?" The advice here is to wait on negotiations until you have a "winner" or are having a hard time selecting a "winner"] Budget Consideration and Selection The high criteria scores provide a range for what the project should cost. (Another reason why budgeting should wait till this step.) Creating a budget first will provide to high of a temptation to fudge the scores or select the highest scoring firm within budget. And if the budget was too low (I.E., outside industry parameters to solve your problem) the risk of failure is high. Things not done right are things done over, or at the very least left to rot and create an ungodly stink of resentment. Selection CriteriaStrider ConsultingHelm's Deep ConsultingMoria ConsultingCriteria Total Score241919Bid Price$15,000$18,000$10,000Matrix Excerpt You may find a high scoring company with a bid well below the others. This is worth investigating. Call them back and ask why they are so much cheaper. The best scenario is that #1 on the scoreboard is also affordable but in the likely case this is not the case, you have 2 options: select one of the 3 with the best trade-off (score for price) or repeat the previous step to incorporate another three or so firms into the decision. Don't repeat this ad infinitum. If after going through six candidates with reasonable score to price ranges, a decision still hasn't been made and price is the issue, then there is a possible conclusion to consider: you don't take the problem as seriously as you thought. Conclusion The proposed process is built on subjective rankings determined from gathered information. This is still better than going into a call with just an idea (however strong) and having charm roll you like cool breeze that turns into a blizzard. The research, selection criteria, red flags, and scoring all act as buffers against sales tactics and emotional leaps.