The Ella Root Break-In

A tale of crime, corruption, and IRC

This is the story (admittedly technical at points) of how an outside user managed to get root access on one of our UNIX servers (ella.ariel.cs.yorku.ca) and what the intruder did. All of the exploits and back-doors described in this story have long since been removed from our systems. Ella is a dual-processor Sun Sparcstation 10 running Solaris 2.5.1.

At some point before June 19, 1998, an intruder managed to get access to cs981008 and either got access or had knowledge of cs981028 on the Ariel network. All external connections to the Ariel network must go through ella.ariel.cs.yorku.ca as all other machines reject external connections. The cs981008 account was almost assuredly sniffed from gmu.edu (George Mason University - Burke, Virginia):

(ella.ariel.cs.yorku.ca)
i Thu May 28 17:19:20 1998 cs981008     pts/64     arl-lalab-nt.cs.gmu.edu  
o Thu May 28 17:27:16 1998 cs981008     pts/64     arl-lalab-nt.cs.gmu.edu  
i Wed Jun 17 16:04:12 1998 cs981008     pts/8      arl-lalab-nt.cs.gmu.edu  
i Wed Jun 17 16:09:56 1998 cs981008     pts/57     arl-lalab-nt.cs.gmu.edu  
o Wed Jun 17 16:12:12 1998 cs981008     pts/57     arl-lalab-nt.cs.gmu.edu  
o Wed Jun 17 16:12:13 1998 cs981008     pts/8      arl-lalab-nt.cs.gmu.edu  
i Fri Jun 19 11:40:59 1998 cs981008     pts/25     hd33-036.hil.compuserve.com
o Fri Jun 19 11:45:13 1998 cs981008     pts/25     hd33-036.hil.compuserve.com
i Fri Jun 19 22:36:09 1998 cs981008     pts/3      dd21-048.dub.compuserve.com
o Fri Jun 19 22:41:19 1998 cs981008     pts/3      dd21-048.dub.compuserve.com
i Fri Jun 26 12:40:39 1998 cs981008     pts/28     arl-lalab-nt.cs.gmu.edu  
o Fri Jun 26 12:47:03 1998 cs981008     pts/28     arl-lalab-nt.cs.gmu.edu  
The logins from arl-lalab-nt.cs.gmu.edu appear to be the owner of the account and represent typical user traffic (pine,finger,talk,etc.) The logins from compuserve.com were from the intruder. Later we found evidence to suggest that the user had also compromised a system at gmu.edu, and so was probably able to run a sniffer there.

When the intruder logged in on June 19, 1998 via the cs981008 account, he took advantage of a buffer-overflow bug in /usr/lib/fs/ufs/ufsrestore which was a setuid-root program. On his first attempt (the login on pts/25) he tried to simply copy the program across the wire, and then "cat" it to a file on this side. This failed leaving a corrupted fragment of the program which he forgot to erase. On his second attempt, it appears that he ftp'ed a tar file from a remote site and that file contained the intrusion program which then succeeded.

He did not notice or attempt to erase the .history file left behind by tcsh so we have a record of his actions:

#+0898270913
ls -latF /usr/lib/fs/ufs/ufsrestore
#+0898271002
cat > a.c
#+0898310179
ls -tlF /usr/lib/fs/ufs/ufsrestore
#+0898310236
mkdir ...
#+0898310237
cd ...
#+0898310237
ftp
#+0898310361
chmod +x a
#+0898310362
./a
#+0898310372
gzip -d *.gz
#+0898310375
tar -vxf *.tar
#+0898310379
chmod a+rwx *
#+0898310381
chmod a+rwx ./
#+0898310382
./a
#+0898310467
cd ..
#+0898310468
rm -rf ...
#+0898310474
unalias rm
#+0898310476
rm -rf ...

After the intruder gained access, he made the following modifications, by downloading a tar file (perhaps already downloaded earlier) and unpacking it the root directory:

There were two immediate results of this intrusion. First, without an /etc/identd.fakeid file, the daemon spat out a message to syslog for every connection and periodically spat out error messages to syslog as well. These were noticed but no immediate concern was raised as they were being logged to the user facility and were suspected to be the result of a user trying to run a daemon as themselves. Second, the new /usr/bin/login sent very different messages to syslog than the original one. These were missed due to a syntax error in the syslog filtering script which was silently failing. Less technical aspects intruded at this point; some of us vanished for a week to the USENIX conference so investigating the first symptom.

Upon my return, I went back through my notes and tracked the symptom, discovering the daemon running as root. Since we did not run an ident daemon on ella, I knew that this had to be an intrusion; a quick check through our own logs convinced me that no one on the staff had put it there. At this point, I alerted everyone else on the staff to the

breach and set about gathering as much data as I possibly could. At this point I determined the source of the break-in (cs981008 account) and all of the modified files except for /usr/bin/login. After some discussion, we agreed not to try and lock him out of the system yet and to watch him and wait. We put traces on the Xinitpc program and the identd and rpc.rexd daemons.

Nothing happened for two weeks at which point he logged in using the hole in /usr/bin/login (which we had not discovered yet). Upon returning he did the following:

After analyzing the situation (and finding the /usr/bin/login hack) we devised a plan and locked him out of the system. However, before we were able to do so, the IRC bot sent out an alarm - apparently it was tracking the atime (and others) on its files and sent out a message when it saw somebody poking through its files. The intruder came back and wiped out some files including the sniffer log (from which we had already extracted sufficient data) but oddly left one of the sniffer binaries (named "tcsh") and the bot running.

Waiting for the intruder had mixed results - some accounts got compromised but at the same time we managed to get a list of about 30 sites which the intruder had either cracked or was affiliated with. Simple probes (e.g. finger) turned up somewhat open systems. Long streams of e-mail were sent out to affected parties and we have had at least one match in which the same guy with the same modus operandi had broken into somebody's name server.

Also interesting are the responses. Out of 28 messages sent to remote sites, about 13 never responded at all, even though mail seemed to get through okay. Out of the responses I received a large number, particularly from larger institutions were form letters promising future contact if more information was needed, that they were looking into it, and in some cases that they were aware of the situation or had had similar problems in recent history.

Update - July 21st

As of this day, there have been no signs that the intruder has returned. Details of the ufsrestore exploit can be found at http://rootshell.org/. There are also messages there which insinuate that this hole may not be fixed yet. We turned off the setuid-bit on ufsrestore almost immediately after we found the hole and suggest that others do the same.


Matt Robinson (matt@cs.yorku.ca)