Fedora Package Signing Server Compromise, CRLs, and Trust
2008/08/25/0916RTFA: http://rhn.redhat.com/errata/RHSA-2008-0855.html
Last week Red Hat detected an intrusion on certain of its computer systems and took immediate action. While the investigation into the intrusion is on-going, our initial focus was to review and test the distribution channel we use with our customers, Red Hat Network (RHN) and its associated security measures. Based on these efforts, we remain highly confident that our systems and processes prevented the intrusion from compromising RHN or the content distributed via RHN and accordingly believe that customers who keep their systems updated using Red Hat Network are not at risk. We are issuing this alert primarily for those who may obtain Red Hat binary packages via channels other than those of official Red Hat subscribers.
In connection with the incident, the intruder was able to sign a small number of OpenSSH packages relating only to Red Hat Enterprise Linux 4 (i386 and x86_64 architectures only) and Red Hat Enterprise Linux 5 (x86_64 architecture only).
Interesting situation here… Public key encryption is the primary basis for establishing trust in the online world. A private key is supposed to be a complete secret, such that anything signed by it can only have come from the person who holds that secret. As long as you trust the holder of the private key, then you can theoretically trust their works… or at least you can deduce whether or not their work has been tampered with en route.
The situation with Red Hat/Fedora, it would appear, is that some private keys have been compromised – specifically those used for signing packages. A “package” consists of pre-compiled software, ready to install. You don’t install the software if the signature can’t be verified. In this case, the signature will not be a good indicator of the validity of the package, because the signatures have been generated by the legitimate private key.
Specifically, in this case, the attack was aimed against the package-signing servers. This one is really major, because as an “infrastructure attack” it has massive scaling potential. We all knew this sort of thing was possible… but it’s finally time to confront the issue.
One of the hot topics on Bugtraq this month involved certificate revocation – namely, what to do when one of these signatures (or certificates) is compromised? …how does the world know that they should ignore the “bad” ones?
When an earlier Debian-specific OpenSSH vuln was discovered (predictable random number generation) the situation was easy, insofar as the vuln specifically reduced the number of bad keys that could be generated, thereby making the revocation list trivial to generate. This fix was disseminated as a series of Linux-distribution-specific packages, for even though the problem was Debian-specific, all clients must be aware of bad keys while connecting to a Debian-based machine. These updates were then gathered and installed by the package-management software that is standard with any modern Linux or BSD system.
The major question is about this very system of distribution. First, the revocation list may need to be updated more frequently than a 24-hour cron job would permit. That need doesn’t exist this year, but in ten years it may. Next, software needs to honor these revocation lists. TLS, the predominant browser-based encryption scheme, does include a specific mechanism for this purpose: the ability to import a Certificate Revocation List (CRL). However, in practice, this practice is essentially ignored. As a consequence, when any browser visits a SSL- or TLS-encrypted page on the web, the only check performed is to determine if the certificate (signature) is valid. Significantly, there is NO major mechanism for checking if a signature is valid but has been revoked.
This scenario is important in the Fedora case, where there are valid keys on the signing server, but those keys have been compromised and must be revoked… and that is a surprisingly difficult challenge.