What is evaluated by FIPS, what is evaluated by Common Criteria, what is evaluated by NIAP, and what does a Security Assessment provide? What is a secure device, and what is a secure product?
Secure IT is a decades-old pipe dream. The products must be secure, but most of them are not. Security requires not only secure cryptography and perfect security features, but also a secure environment and management.
For the uninitiated, a FIPS certification is a seal of approval for the security of the product and functionality; for the initiated, it is not. The uninitiated are misled by vendors, and the initiated are not the least bit surprised why security incidents continually occur in the USA at government, administration and the commercial sector despite certified products. This is because FIPS only evaluates compliance with NIST standards with respect to cryptography. However, security is not limited to cryptography and FIPS does not even evaluate the functionality and security of all uses of cryptography in a system. FIPS is not a security certification and only certifies cryptographic security to a limited extent.
Cryptographic Certifications, Functionality Certifications and Security Assessments
A certification attests the compliance with certain, clearly defined requirements. These can be cryptographic specifications, as for example with FIPS, and they can also be functionality specifications, as for example with Common Criteria. If you want to have cryptography, functionality and product security checked, you need a security assessment. Every product and every implementation is individual and only a security assessment can address the peculiarities and all relevant aspects of a product. This is why a security assessment cannot be standardized.
Standardized IT security certifications are limited in their informative value. A standardized certification needs standardized specifications in order to cover products from different manufacturers with different implementations. Standardized certifications show what has been evaluated according to which specifications and has satisfied the specified criteria. There are different standardized security certifications for different purposes. The significance of a certification depends on what is tested, how and in what depth of evaluation. Depending on the security requirements and purpose, the requirements are different. For the most common security certifications in the West, there are several gradations covering different security requirements. Only very few meet higher safety requirements. It should be noted that the common standardized security certifications are designed to protect sensitive data, but not classified data. If the quality of the product is insufficient, combined with insufficient depth and quality of the evaluation, even a standardized security certification does not make a product secure.
All of this is really about risk management. You can only assess risks if you know what they are.
FIPS
Especially US vendors and vendors who want to sell their products on the US market rely on FIPS (Federal Information Processing Standard). This is defined by NIST (National Institute of Standard and Technology) and has now reached revision 140-3. NIST is subordinate to the American Department of Commerce, whose goal is to promote the American economy. Any product that is not FIPS certified and uses the American standards is considered insecure by NIST. Equally insecure as if no encryption and authentication methods were in place and all data were available in plain text, even to the outside world. As a result, government, administration, military and their suppliers are only allowed to use FIPS-certified products in FIPS mode. Unfortunately, FIPS-certified products are not the least bit more secure than non-FIPS-certified products. There are a number of products that are not FIPS-certified, but which clearly outperform all FIPS-certified products in terms of security and product security. While NIST claims with FIPS that certified products have been extensively evaluated, at least in the area of cryptography, the processes created for evaluation are insufficiently monitored and controlled in terms of quality. For FIPS, only two things are checked: The cryptographic algorithms used in the "Cryptographic Algorithm Verification Program (CAVP)" and the cryptographic module in the "Cryptographic Module Validation Program (CMVP)".
Verification is performed by private service providers that have qualified under the Cryptographic and Security Testing (CST) Laboratory Accreditation Program (LAP) . Manufacturers who want FIPS certification for their products must contact one of these private service providers. For most manufacturers, the primary concern is not the security of their products and the security of their implementation, but to obtain FIPS certification. For the manufacturer, there are only two KPIs in this regard: (1) cost of FIPS certification, and (2) time to obtain FIPS certification. Only with certification does the full size of the North American market open up to the manufacturer. And marketing needs FIPS certification to be able to claim that the product is secure because it is FIPS certified. Many private service providers have adapted their offerings to meet this customer need. As a result, security suffers. Not even secure cryptography is guaranteed anymore. But its customer gets FIPS certification.
There are four different progressive levels of FIPS certification. Each higher level places greater security requirements on the cryptographic module. Level 1 stands for little security requirements, Level 2 stands for very easily met security requirements for cryptography in a device, and Level 3 stands for not so easily met security requirements for cryptography in a device. And all this is limited to the algorithms used and the cryptographic module that is used.
- Level 1 requires at least one approved and tested encryption algorithm besides a production-ready solution. So not much.
- Level 2 something more secure/less insecure and also requires role-based authentication, a tamper-evident physical device (a sticker as a seal will do) and an operating system with a Common Criteria EAL2 certification.
- Level 3 is the level that should be achieved for appliances. In addition to Level 1 and Level 2 requirements, the appliance must be tamper-proof, have separation of logical and physical interfaces through which critical security parameters enter or leave the system, demonstrate identity-based authentication, and allow private keys into or out of the appliance only in encrypted form. If tampering is detected, all data is automatically deleted and overwritten with zeros, a process known as "zeroization".
- Level 4 involves a more extensive formal check and is not very common in practice..
Regardless of the certified level, all vendors advertise that they have received FIPS certification, even if it is only Level 1 or Level 2. Without asking, vendors usually do not provide more specific information. For companies and government agencies, Level 3 should be the norm for devices and Level 2 the exception.
To assess the maximum achievable level of cryptographic security, the FIPS evaluation report is needed. In order to assess the effective security, a security assessment is needed that covers the entire product.
Are FIPS certified devices secure?
Most aren't. FIPS focuses on the cryptography, algorithms used, key exchange methods, random number generation and key storage. It does not look at the security of the overall solution, only the cryptographic aspect. If the device is insecure, the whole solution is insecure, and even a physically tamper-proof device can be insecure. But the overall security is not even evaluated. The lists of published CVEs for security solutions from the leading U.S. vendors of integrated and standalone security solutions look like this.
Are FIPS-certified solutions cryptographically secure?
Theoretically yes, in practice partially no. This can be representatively proven by the publicly known example of SilverPeak, a vendor placed by Gartner in the upper right quarter of the Magic Quadrant for SD-WAN and SASE and recommended as the first choice. Well-known and renowned security researcher Denis Kolegov, associate professor of IT security at Tomsk State University, analyzed SilverPeak's cryptography in a limited security assessment and rated it as unfit for purpose. This is despite FIPS certification at Level 2, for which the cryptography should actually have been checked before the certificate was issued.
The problems found include:
- Nonces (Number Only Used Once), are used and stored in multiple locations, even though a nonce must be unique and deleted after one use. In SilverPeak's case, the SD-WAN controller distributed the same nonce to each of the connected devices that encrypt network traffic. So the same nonce was present in three places.
- Using the nonces that were not immediately deleted, it was possible to calculate the symmetric keys used to encrypt the network connections and thus decrypt all the traffic.
- The devices are insecure and even administrator rights can be obtained quickly. This then allows access to the nonces.
- The authentication did not check the validity of a certificate, but only whether a certificate was present. This is like a personal check at the border, where it is only checked whether the person has an ID card, but not whether the ID card is issued to this person. As a consequence, it was possible to contact other SilverPeak devices without a valid certificate.
The CVEs can be found on the SilverPeak website and the HP/Aruba website. Details about the issues reported can also be found on the Github page of the SD-WAN New Hope project.
That market share is more important than product quality and security was shown by Hewlett Packard (HP) who took over SilverPeak for about 1 billion US dollars despite poor product quality and lack of security. Product quality and security apparently didn't matter to Gartner either when SIlverPeak was positioned as a leader in the upper right portion of the quadrant. Since SilverPeak used this Magic Quadrant massively for its own advertising, one can assume that there was quite a bit of money flowing in the direction of Gartner.
Common Criteria
With Common Criteria, the product to be evaluated (Target of Evaluation/TOE) is evaluated against a security target with a certain evaluation depth. The greater the evaluation depth, the more closely the product is evaluated. The more demanding the security target and the greater the evaluation, the greater the confidence in security.
Common Criteria recognizes seven evaluation assurance levels (EAL), of which EAL2 to EAL4 are mainly used in practice:
- EAL1 is suitable for products that meet specific customer requirements and require low security.
- EAL2 is a level that is often used for applications with safety functions. It is also an "entry level" for developers who aspire to a higher level but want to familiarize themselves with the evaluation and certification process beforehand.
- EAL3 is a level typically chosen for complex products (such as operating systems) when a higher level is deemed too costly.
- EAL4 is a level suitable for products and customers that require and need a high level of security.
- EAL5 is a level chosen when customers require additional security (smart card components).
In order to assess the maximum achievable security, the following documents are required: The Security Target, the evaluation level used and the evaluation report. In most cases, the Security Target covers only a sub-area of the product and accordingly only this sub-area is checked with the selected evaluation depth. Standardized protection profiles for an evaluation depth up to EAL4 are rare and mostly country-specific. A complete evaluation to EAL4+ is costly and time-consuming. In addition, only a few countries recognize such certifications up to an evaluation depth of EAL4. As a result, what has been evaluated and certified according to EAL4 in one country is only recognized as having been evaluated according to EAL2 in other countries. Standardized protection profiles up to EAL4 that are accepted by several countries would solve this dilemma. For individual subareas such as hardware security modules for E-IDs (EiDAS), there is now such a Common Criteria Protection Profile against which commercial products can be checked and tested with an evaluation depth of EAL4+. This is not yet the case for most other areas and other countries.
The NSA, together with mostly American vendors, has found a commercially optimal solution. Standardized Protection Profiles with an evaluation depth of EAL2, which cover only partial aspects and are easy to fulfill and are based on FIPS for cryptography.
NIAP (National Information Assurance Partnership)
NIAP is a partnership between the NSA and mostly American vendors. NIAP uses standardized Protection Profiles (PP) in the Common Criteria Evaluation and Validation Scheme (CCEVS), which are mostly based on FIPS. NIAP was initiated by the NSA and should enable American vendors to obtain an internationally recognized certification, Common Criteria, easily and cost-effectively. The security granted is moderate and limited to US standards. NIAP stands neither for secure products nor for secure cryptography. The list of certified products includes products with numerous security vulnerabilities. The NSA's partnership with U.S. manufacturers is largely oriented toward the interests of manufacturers and U.S. business, not security.
Other security certifications and approvals
IT security certifications show what has been evaluated and what has been approved. Many countries have their own cryptographic specifications and their own certifications. Often, only a subset of the product is evaluated and certified, rather than the entire product. Certifications do not guarantee security, but they do indicate what has been evaluated. Approvals, on the other hand, approve a product or solution for a specific use case. In most cases, an approval requires an up-to-date security assessment. Such approvals may also specify that the product may only be used in combination with another approved product. So even with approvals, it's important to know the details. Most products for classified data are subject to export and sometimes import restrictions.
What's missing for secure products with secure cryptography
Whether a product is secure, has more secure cryptography and secure and complete security features can only be determined by conducting a security assessment. This is time-consuming and requires deep subject and technical knowledge. The effort is great in all aspects. The idea of a National Cybersecurity Test Center, as is the case in Switzerland, is promising, but many questions remain. Such a test center would have to conduct a full security assessment for each product. This would require the cooperation of the respective manufacturer. It is unclear who would bear the costs and whether the results would be accessible to third parties. The sheer number of different products and their ongoing development pose a further challenge. This is because every further development can lead to new security vulnerabilities. If the results of these security assessments can be made available to third parties, specifically current or potential customers of a manufacturer's product, this would be a quantum leap for IT security and risk management. This should then also cost something. But since the costs would be shared by many in this way, they would be quite bearable.
The consequence of insecure products
The goal should be secure products with secure cryptography and secure functionality. Secure products are not an end in themselves, but prevent harm and increased operational overhead. They are the basis for secure cryptography because data is always in clear text before encryption and after decryption. They prevent security incidents from occurring because of them. Insecure products increase the risk of loss of data confidentiality, data integrity and data availability just as insecure cryptography does. They can also be abused as a gateway into networks. Plugging security holes by means of patches made available by the vendor leads to additional work and reduces the availability of the product. In the worst case, personal data and company secrets are leaked. In the case of leakage of personal data, there is a fine on top of that. These are actually all things that one would gladly do without. A good reason to rely on secure products with secure cryptography and secure security features. And these are not necessarily the ones that are advertised as "secure". It's not the cryptography that makes a product secure. That's why neither FIPS, nor EAL2, nor NIAP is enough. This is what the reality shows.
Why do customers buy insecure products?
For the most part, customers don't know any better because risk management fails and no due diligence is done. There is no economic incentive for vendors to offer secure products as long as customers accept insecure products and the manufacturer is not liable for anything.
Bruce Schneier, well-known security and cryptology expert has aptly summarized the current situation: “The market does not reward security or longevity. The market rewards features, time to market, and price. This is a market failure, and we need to recognize it and treat it as such.”
The market consists of vendors and customers. If customer behavior changes, vendors adapt to the changed customer behavior, because the customer decides what to buy.