Date: Tue, 31 Oct 1995 18:57:39 -0500 Last Minute Update The rest of this memo assumes that CERT released an advisory today and that SGI and DEC made patches available to customers. I know that both DEC and SGI are working on this vulnerability, and have seen one preliminary patch from DEC. Apparently, something happened, and the advisory wasn't released today. I was not contacted by CERT and only confirmed that nothing had gone out after people left the CERT office, so I'm not sure exactly what happened. I have decided to release this information anyway, because it has already been widely distributed earlier today within MIT, and I know of at least one vendor who is actively contacting customers. I suggest reading the rest of this memo, then coming back to the next paragraph. The quick fix will work for DEC OSF, but not for SGIs. The CERT advisory contained a different quick fix, but I do not have permission to release that, and don't feel it would be appropriate to release without explicit permission from CERT. Basically, the CERT approach was to make a statically linked login wrapper and have telnetd call that wrapper and trim the environment. I have examples of such a program we've used within MIT that I can probably release, although we haven't debugged it. Another option is to make telnetd set-uid root, but only executable by a new group created especially for this purpose. Then, create a user with the primary group that you just made, and have telnetd run as this user by inetd. This will mean that the real UID is not root, but the effective UID is root, so the _RLD_* will be ignored. I'm not sure about the other security implications of this; I stopped thinking about SGI contingencies when I heard a patch would be released today. I suspect that the problem will be hashed out on bugtraq@crimelab.com (listserv@netspace.org for subscriptions) fairly quickly. Also, I hope that SGI will release their patch in the next day or two. This memo contains a description of a vulnerability cause by an interaction between telnetd and shared library loaders on several versions of Unix. CERT should be issuing an advisory regarding this problem on October 31. While I have tried to keep an up to date list of available vendor patches in this memo, it is likely that the CERT advisory will contain more up-to-date vendor info. In particular, I don't have an exact URL to the FreeBSD, SGI or Digital Unix patch.This memo is intended to document the problem in MIT Kerberos and to provide additional detail that is likely not in the CERT advisory. Contents * Preface and History * Quick Fix * Environment Variables that Matter * Affected Telnetds * Telnetds that Work * Availability of Patches * Testing for Exposure * Verifying a Patch * Sample Patch * Acknowledgments Preface and History On Sunday, October 15, I discovered a bug in some versions of telnetd on some platforms that allows a user making a connection to cause login to load an alternate C library from an arbitrary location in the filesystem of the machine running telnetd. In the case of machines mounting distributed filespaces such as AFS or NFS, containing publicly writable anonymous FTP directories, or on which the user already has a non-root account, it is possible to gain root access. The problem is that telnetd will allow the client to pass LD_LIBRARY_PATH, LD_PRELOAD, and other run-time linker options into the process environment of the process that runs login. If the runtime linker honors these options for login, the attacker can cause a custom libc to be loaded. Such a libc could, for example, start a shell whenever some function like crypt() is called. If login calls this function before the setuid call, then the attacker will gain root access. Note that the user must be able to convince telnetd to run login in order for this attack to be successful. In particular, if an authentication system such as Kerberos is employed, and the telnetd requires authentication, then only users with valid accounts will be able to use this attack. Quick Fix Normally, programs that run with the set-user-ID or set-group-ID bit set do not use environment variables to pass information about where to find libraries. This is designed to prevent attacks where an intruder sets LD_LIBRARY_PATH and runs `su' or `login' from the command line. Since these are set-user-ID programs, they run as root; if they trust LD_LIBRARY_PATH, then they can load a user-supplied libc and be as insecure as telnetd. The test used by most runtime linkers to determine if LD_LIBRARY_PATH can be trusted checks to see if the effective user ID is equal to the real user ID. Since telnet is started as a root-owned process by inetd, its real user ID is root. So, even if login is set-user-ID, the test succeeds and LD_LIBRARY_PATH is trusted, creating the security problem. On many systems, if login is made set-group-ID, the test will fail because login's effective group will be different than the real group. This should only be used on a temporary basis. Unfortunately, this doesn't always work: in particular, it doesn't work for SGI Irix. (SGI has already released a patch, but other systems may exist where this also fails.) If you try this fix, you should go through the "Testing for Exposure" section. To make login set-group-ID follow these steps: 1) Create a new group (you might want to call it `__login', `__telnet', `tnbug' or something of the sort). In the rest of this document, I will assume that you have called the group `__login'. Make sure the group doesn't already exist, and make sure that no other programs are moved into this group. For information on how to create a group, consult your vendor's manuals and the man page for /etc/group. 2) Find your login program; it is often /usr/bin/login or /bin/login. I will assume here that it is /bin/login. Under AIX and some other operating systems, login may be a symlink to another program;change the group of the actual program, not of the symlink. 3) Look at the current permissions on login: bash$ ls -l /bin/login lrwxrwxrwx 1 root system 13 Mar 8 1994 /bin/login -> /usr/sbin/tsm bash$ ls -lL /bin/login -r-sr-xr-x 3 root security 59217 Aug 23 1994 /bin/login bash$ Note that because I am running on an AIX system, I gave the -L option to the second ls in order to trace through the symlink and find the real login. You should remember what group login is in so you can change things back after you get a vendor patch. 4) Change the group of login and set the set GID bit: bash$ su root's Password: bash# chgrp __login /bin/login bash# chmod g+s /bin/login bash# ls -lL /bin/login -r-sr-sr-x 3 root __login 59217 Aug 23 1994 /bin/login bash# Environment Variables that Matter This is not an exhaustive list of environment variables telnetd should filter, but it does contain several of the key variables on systems used at MIT and by people with whom I have consulted: LD_LIBRARY_PATH: At least Solaris, SunOS, NetBSD, Linux and Digital Unix use this as the path to look for shared libraries in. LD_PRELOAD: Solaris and possibly others load these object modules into the address space of the process before loading other shared libraries. LIBPATH: AIX uses this to locate its shared libraries. ELF_LD_LIBRARY_PATH: May be used by the Linux Elf loader; similar to LD_LIBRARY_PATH in function. According to the author of Linux ld.so (hjl@nynexst.com), this was only used by ld.so version 2.6. This version was only used for beta Elf development, and is apparently not used by any production Linux distributions. LD_AOUT_LIBRARY_PATH: Another Linuxism from ld.so-1.7.3. Same as LD_LIBRARY_PATH but only for a.out libraries. _RLD_ROOT: Digital Unix uses this to specify a prefix prepended to library paths stored inside executables. _RLD_LIST: A list of objects dynamically loaded into an executable by Digital Unix. _RLD_*: Used by Digital Unix and SGI. There are several apparently undocumented variables in /sbin/loader on Digital Unix and in the SGI runtime linker. LD_*: Several other variables have special meaning to certain operating systems. Stripping all these variables would probably be a good idea. IFS: It is possible that setting IFS could cause damage in environments where the user logs into an account that runs a shell script instead of granting full access. Affected Telnetds All telnetds derived from the Telnet package distributed by David Borman allow the environment options to be passed. Borman has released a patch for the problem as of October 19. The patch released on October 19, while secure, has a bug that prevents any telnet environment options from being handled. Another patch was released on October 23 that appears to work; see below for details. Besides his original release, here are a list of operating systems and security packages I'm aware of that include derivatives of this work: * NetBSD and FreeBSD are distributed with a vulnerable telnetd. (See below for patch info.) * The version of telnetd maintained in the Kerberos version 5 distribution by MIT. (patch available) * The Cygnus Network Security V4 95Q1 Free Network Release includes a vulnerable telnetd. (Previous releases did not contain telnetd.) A patch has been released. * OpenVision's OV*Secure contains a telnetd that is vulnerable; a patch is available. Other Vulnerable Telnetd Implementations This problem is not unique to code derived from the Borman telnet distribution. Other vulnerable implementations are known to include: * SGI Irix 5.3 (patch available) * Digital Unix. The telnetd distributed with stock Digital Unix appears to be vulnerable. DEC confirms they are investigating. * Linux. The telnetd distributed with Slackware Linux appears to be vulnerable, although I have not verified this. The maintainers of Debian GNU/Linux confirm their telnetd is vulnerable and released a patch; see below. A patch is also available for Redhat Linux. Telnetds that Work Below is a list of operating systems which come with telnetds that we know are not vulnerable. * SunOS 4.1.4. The Sunos 4.1.4 telnetd does not support passing of environment variables, so it is not vulnerable. * IBM AIX 4.1. This telnetd does not support environment options. * BSDI's BSD/OS. While the telnetd will pass any environment option, there doesn't appear to be an option to override the shared library path, so BSD/OS is probably not vulnerable. On October 19, Dave Borman confirmed that BSDI is not vulnerable to the attack, although the telnetd will accept any environment variable. * Telnetd on other systems that do not support shared libraries. This includes DEC Ultrix, and Cray Unicos. * According to LaMont Jones , "HP-UX is not vulnerable to this attack, due to our shared library implementation." Note that both AIX and SunOS can be vulnerable if the stock telnetd is replaced. Also, note that the stock Solaris telnetd has not been tested. Availability of Patches This is a list of patches I'm aware of at this time. As you develop a patch for your product/platform, please let me know; considering free operating systems affected by this problem, widely announcing the bug once patches are available is very important. Note that these patches are provided as-is, without any guarantee of correctness or security on the part of MIT, myself, or the patch creator. They are provided in a spirit of cooperation, not as a guaranteed fix. I cannot certify the degree of testing applied to these patches. Note that CERT and CIAC plan to announce bulletins about this problem on October 31, 1995. Where this memo conflicts with the information provided in the CERT advisory, assume CERT's information is more accurate: they have better vendor contacts, and have been actively confirming patch availability for the last few days. I am including this section in order to provide users with patch locations, where possible, in the same place where they first encounter details about this problem.I am maintaining it with information I receive, but not all vendors have replied to my earlier memo, so if your favorite vendor isn't listed here, check the CERT advisory before contacting them. * On October 19, David Borman released a new version of his telnet package, containing a fix to the problem. This original patch disabled passing environment options entirely, but was revised on October 23. The revised patch, and instructions for obtaining it are contained at the bottom of this message. Note that this patch does not deal with the ELF_LD_LIBRARY_PATH, although for most Linux users, this is not a problem. The version of telnet on net-dist.mit.edu contains this patch. * Greg Hudson checked a patch into the NetBSD-current source tree. This patch will be incorporated by any NetBSD-current users who update to the current telnetd. It will be in the NetBSD 1.1 release. NetBSD developers have not indicated whether they plan on releasing a patch for NetBSD 1.0 users. Note that the sample NetBSD patch distributed with an earlier version of the memo was incomplete; the version in the source tree as of October 18 is correct. * Sam Hartman patched the upcoming release of Kerberos 5. In addition, patches were generated against Kerberos 5 beta 5 and beta 4-3. The can be found at ftp://athena-dist.mit.edu/pub/kerberos/telnet-patch/. * Mark Eichin prepared patches for CNS. These patches will be available on the Cygnus web site http://www.cygnus.com/data/cns/telnetdpatch.html; support customers are being contacted directly. * OpenVision has a patch for the telnetd in OV*Secure 1.2 and will contact its customers directly. * Peter Tobias, released a patch for Debian GNU/Linux. This patch can be found in the networking utilities at ftp://ftp.debian.org/debian/debian-0.93/binary/net/netstd-1.21-1.deb. * Erik Troan confirms that Redhat Linux is vulnerable, indicating a patch can be found at ftp://ftp.redhat.com/pub/redhat-2.0/updates/NetKit-B-0.06-4.i386.rpm or ftp://ftp.pht.com/pub/linux/redhat/redhat-2.0/updates/NetKit-B-0.06-4.i386.rpm The fix is incorporated into the Redhat 2.1 release. * An SGI patch is available at ftp://sgigate.sgi.com/security/. * DEC confirmed they will have a preliminary patch available on October 31; they will be contacting customers and releasing patch info to CERT. * Andrey A. Chernov released a patch for FreeBSD, but did not include an URL where the patch could be obtained. * Bruce Lewis is preparing a patch for the MIT-distributed Athena telnet/telnet95. His patch is currently available within MIT. Within MIT, consult the netusers discuss meeting for more details. Testing for Exposure In order to test to see if your telnetd passes environment variables that effect shared libraries, it is important to understand what environment variables are used by your runtime linker. See the environment variables section for common examples. To be sure, you should run strings over your runtime loader. For example, the following shows the environment variables used by NetBSD: athena% strings - /usr/libexec/ld.so ld.so LD_LIBRARY_PATH LD_PRELOAD LD_NOSTD_PATH LD_SUPPRESS_WARNINGS LD_WARN_NON_PURE_CODE LD_NO_INTERN_SEARCH .init Cannot set breakpoint (%s) Cannot re-protect breakpoint (%s) LD_TRACE_LOADED_OBJECTS Naturally, this is only an excerpt of the output. Therefore, NetBSD probably honors the `LD_LIBRARY_PATH' variable. It appears to honor several other variables as well. (In fact, it honors most of the environment variables besides LD_PRELOAD, which hasn't been implemented yet.) If a system were non-standard, it might not be easy to know what was an environment variable and what was just a string in the binary. For example, the string `runpath' in the Solaris loader is not an environment variable, but a similar string `LIBPATH' in the AIX kernel is the AIX environment variable. You also have to find the dynamic loader, which isn't always easy. Look for a program called `rld', `ld.so', `loader', or some similar name. You should also check your vendor's documentation, but reading the documentation should not be a substitute for reading the binaries, for while binaries may deceive and obfuscate, they seldom lie. Now that you know what environment variables to check for, find out which telnetd your system runs. Note that the telnetd on my system is almost certainly not in the same place as yours: this session took place on a machine in the Athena environment, so it is running a custom MIT telnetd. However, the same techniques should work with `/etc/athena/telnetd' replaced with `/usr/sbin/in.telnetd' or whatever is appropriate for your system. athena% grep telnet /etc/inetd.conf telnet stream tcp nowait root /etc/athena/telnetd telnetd -a off athena% Now, check to see if it looks like it handles environment options at all (by grepping for `ENVIRON') and if it does anything special with linker environment variables. This test is *not* definitive: there are both false positives and negatives, but you can get a general idea of what to expect in later steps. athena% strings - /etc/athena/telnetd |grep ENVIRO ENVIRON VALUE and VAR are reversed! OLD-ENVIRON NEW-ENVIRON NEW-ENVIRON athena% strings - /etc/athena/telnetd |grep LD_ athena% Ok, it looks very much like I have a problem. My telnetd appears to support environment options--it even has an error message about it, but I see no mention of environment variables that should be restricted. Note that even if I saw no environment info in telnetd, I would continue with the test just to make sure. For the next step, telnet to the machine and see if it passes environment options such as LD_LIBRARY_PATH. Try to create a corrupt libc.so in /tmp by creating a zero length file of the same name; if the system is vulnerable, login will core dump when it tries to use the new libc. If this test fails, try a test using `ps -e' to see if the environment variable got set. In order to find out what library to create, I'll use the `ldd' command on the executable; you could also try looking through /lib, or under AIX, use `dump -H executable'. athena% ldd /etc/athena/telnetd /etc/athena/telnetd: -lcurses.2 => /usr/lib/libcurses.so.2.1 (0x10032000) -ltermcap.0 => /usr/lib/libtermcap.so.0.0 (0x1003d000) -lutil.3 => /usr/lib/libutil.so.3.1 (0x1003f000) -lc.12 => /usr/lib/libc.so.12.3 (0x10041000) athena% touch /tmp/libc.so.12.3 Now, we try and connect: athena% telnet ...including Athena's default telnet options: "-ax" telnet> env define LD_LIBRARY_PATH /tmp:/var/tmp telnet> env export LD_LIBRARY_PATH telnet> set options Will show option processing. telnet> open vulnerable-machine Trying 10.0.0.6... Connected to telnet-bug-exploit.MIT.EDU. Escape character is '^]'. SENT WILL LFLOW SENT WILL LINEMODE SENT WILL NEW-ENVIRON RCVD WILL SUPPRESS GO AHEAD RCVD DONT LINEMODE RCVD DO NEW-ENVIRON SENT WONT XDISPLOC RCVD DO OLD-ENVIRON SENT WONT OLD-ENVIRON SENT IAC SB TERMINAL-SPEED IS 9600,9600 RCVD IAC SB NEW-ENVIRON SEND SENT IAC SB NEW-ENVIRON IS USERVAR "LD_LIBRARY_PATH" VALUE "/tmp:/var/tmp" MIT SIPB NetBSD-Athena (xxx) (ttyp1) ld.so: login: libc.so.12.3: Undefined error: 0 Connection closed by foreign host. athena% This machine is obviously vulnerable. Now, an example of what happens if for some strange reason, login actually works, but the machine is still potentially vulnerable: (telnet session as above, but a login prompt) telnet> open vulnerable-machine Trying 10.0.0.6... Connected to telnet-bug-exploit.MIT.EDU. Escape character is '^]'. MIT SIPB NetBSD-Athena (xxx) (ttyp1) login: Now, we suspend the telnet and look at the login process that was created: athena% ps -ewwa |grep login 6997 p1 Is+ 0:00.05 LD_LIBRARY_PATH=/tmp:/var/tmp TERM=vt100 login -h somew athena% This indicates that the variable was passed, but login failed to act--possibly because you did something wrong when creating the library; your system is probably still vulnerable. If that variable was not present, but the -e flag works on your ps, and other processes displayed environment variables, your system is likely not vulnerable. Also, if neither an old-environ nor new-environ option was passed between the telnetd and telnet, you are almost certainly safe. However, passing this test should not be taken as a guarantee of complete security: you should still contact your telnet vendor unless you are sure you are safe. Verifying a Patch In the process of talking to vendors, distributing patches, and getting feedback, I've come up with a lot of `almost solutions' -- patches that are good enough to make you think they work, but that can be compromised. * A clever trick to get around exact match patches is to embed an equals sign in a variable name. For example, ask your client to send an option requesting that the variable LD_LIBRARY_PATH=/home/hartmans/exploits/sun4lib: be set to the value invalid:/lib:/usr/lib. Naturally, the call to setenv in telnetd adds another =, but that's soaked up by `invalid', and I still get to break into the system. I.E. Deal with variable names containing equals signs (=). * At least in the Borman BSD telnet, there are two calls to setenv: one for the last part of an environment option and one for the other parts. Make sure you cover both; this was the biggest problem with the sample patch I first distributed. * If it is possible to stuff a string into the environment twice with your telnetd, make sure you check all entries in the environment. For example, if you have a setenv() that doesn't check for duplicates, don't just use unsetenv() as this will remove the last item in the environment, leaving the others to be used by login. * Get all the important environment variables. Follow the instructions for testing vulnerability, and check all the potential environment variables found when you strings the loader. Considering the potential to miss variables, several people have suggested only allowing certain variables through. Borman is investigating this and several other options; unfortunately, anything less than a solution tailored to a particular vendor's operating system decreases the functionality provided by the environment option. Sample Patch Below, I include the official patch to telnet from David Borman as of October 23. Before the patch, I include a message I received on October 19; this includes useful information. As I received the message, it was not PGP-signed; its inclusion in this signed summary indicates that it has not been modified since I received it, and says nothing about the integrity of the communications link between myself and Mr. Borman. However, I have examined the patch, and it appears to be a valid fix for the bug. It also corresponds to the appropriate sections of the diff on the ftp server. Again, patches are provided as-is without a guarantee of correctness; you assume all risk for applying this patch. (As with all PGP-signed patches, you will need to remove leading dashes.) Date: Thu, 19 Oct 95 13:54:56 CDT From: dab@berserkly.cray.com (David A. Borman) Message-Id: <9510191854.AA03474@frenzy.cray.com> To: hartmans@MIT.EDU Subject: Re: telnet vulnerability giving root access Cc: cert@cert.org, tytso@MIT.EDU I have placed a version of the BSD Telnet distribution at: ftp://ftp.cray.com/src/telnet/telnet.95.10.23.NE.tar.Z with a fix for this problem. It changes telnetd to remove the LD_*, _RLD_*, IFS and LIBPATH environment variables before execing login. The version on ftp.cray.com does not contain the encryption code, that is on net-dist.mit.edu, and I have sent a new copy off to them to replace the current distribution. (The attached diffs also show a bugfix for a problem that was screwing up /etc/utmp on Solaris.) Also, BSDI is not affected, as they do not provide any way for the user to override the search path for shared libraries. UNICOS is unaffected, since we don't have shared libraries. Please feel free to pass on this message. -David Borman, dab@cray.com diff -cbr telnet.95.05.31/telnetd/sys_term.c telnet.95.10.23/telnetd/sys_term.c *** telnet.95.05.31/telnetd/sys_term.c Wed May 31 00:50:57 1995 - --- telnet.95.10.23/telnetd/sys_term.c Mon Oct 23 09:47:17 1995 *************** *** 32,38 **** */ #ifndef lint ! static char sccsid[] = "@(#)sys_term.c 8.4 (Berkeley) 5/30/95"; #endif /* not lint */ #include "telnetd.h" - --- 32,38 ---- */ #ifndef lint ! static char sccsid[] = "@(#)sys_term.c 8.4+1 (Berkeley) 5/30/95"; #endif /* not lint */ #include "telnetd.h" *************** *** 1570,1579 **** utmpx.ut_id[3] = SC_WILDC; utmpx.ut_type = LOGIN_PROCESS; (void) time(&utmpx.ut_tv.tv_sec); ! if (pututxline(&utmpx) == NULL) ! fatal(net, "pututxline failed"); #endif /* * -h : pass on name of host. * WARNING: -h is accepted by login if and only if - --- 1570,1581 ---- utmpx.ut_id[3] = SC_WILDC; utmpx.ut_type = LOGIN_PROCESS; (void) time(&utmpx.ut_tv.tv_sec); ! if (makeutx(&utmpx) == NULL) ! fatal(net, "makeutx failed"); #endif + scrub_env(); + /* * -h : pass on name of host. * WARNING: -h is accepted by login if and only if *************** *** 1809,1814 **** - --- 1811,1836 ---- return(argv); } #endif /* NEWINIT */ + + /* + * scrub_env() + * + * Remove a few things from the environment that + * don't need to be there. + */ + scrub_env() + { + register char **cpp, **cpp2; + + for (cpp2 = cpp = environ; *cpp; cpp++) { + if (strncmp(*cpp, "LD_", 3) && + strncmp(*cpp, "_RLD_", 5) && + strncmp(*cpp, "LIBPATH=", 8) && + strncmp(*cpp, "IFS=", 4)) + *cpp2++ = *cpp; + } + *cpp2 = 0; + } /* * cleanup() Acknowledgments In preparing this bug summary, I have received the help of several people. In particular, I would like to thank David Borman for quickly fixing the problem once notified, and Bruce Lewis for supplying a timely solution to the problem within MIT. In addition, John Hawkinson provided help developing exploit scripts and confirming that the bug existed on several systems. In addition, I would like to thank vendor security contacts for being responsive and working quickly to get patches ready as soon as possible. I would also like to thank those at MIT who reviewed drafts of this announcement and suggested improvements. - --Sam Hartman, MIT Kerberos Development team -----BEGIN PGP SIGNATURE----- Version: 2.6.2 iQEVAwUBMJa3e0JYVPVo3rXRAQGq7Qf9HJQoShE58Cwr2bd0fAlJDhGycyH8QD5Z 8eb+VacvAMPaHUBDJ0qziTkRVX3cQcWvCq2pYOU3oY4pu7LFoo1mDf/+L0IEHxW+ CnKYxseJaH1qW9CEovopACS+6AonbjmlW11p6xmIskONZNpE4IcQNctvLLR7qt7D GkbElAfFyJI5P41ssvySVr8JgWbepjFsdT+fvbT89rUDCb9ASaPIzeBW+lA40I5k MKRYq9droETIWO3vT7IzxplB4rlEdymtU3ibfr8wxsTOthMAQ0hQ2jgKoLCPIg1/ X2lX9hPHCW14t+t8/4MeO4+69FtLOt6+oISa3vRi5t6aFNpAtCdgCA== =rDwP -----END PGP SIGNATURE-----