Hard drives: if you are reading this article, it is very likely that you have one or more such devices. They are quite simple and are basically a set of 512-byte sectors numbered with increasing addresses, also called LBA (Logical Block Address). The computer to which the hard drive (HD) is connected can read and write data into these sectors. Usually a file system is used that abstracts all these sectors to files and folders.
It may seem to the non-specialist that HD equipment should be quite simple: just a device connected to a SATA port that can position its read/write heads and read or write data to the plates. However, their work is much more complicated: aren’t hard drives engaged in processing of failed blocks and attributes SMART, and don’t have a cache with which also somehow it is necessary to work?
All this means that there is something clever in a hard disk, and the cleverness of the device means that it can be cracked. I have always been interested in the possibility of hacking, so I decided to find out how hard drives work on a non-mechanical level. Such studies have already been conducted with different types of equipment, from PCI expansion cards and built-in controllers to laptops and even Apple keyboards. Usually the studies were conducted in order to prove that the possibility of hacking these devices could lead to software compromise, so I wanted to use the same approach: creating this hack, I wanted to create a hard drive capable of bypassing software protection.
To find out if it is possible to hack hard drives, I first needed to know them better. Fortunately, like most of you, I have a whole bunch of old and/or broken hard drives to study:
Of course, we all know how mechanical parts of hard drives should work, but I was not interested in them. What mattered to me was the small printed circuit board on the back of the HD, where the SATA and power connectors are located. That’s what these printed circuit boards look like:
You can see that the board has about four chips. This is what I managed to learn about them:
This is DRAM memory. It is the easiest to use, the specifications are easy to find. These chips can be between 8MB and 64MB in size, and this size is related to the cache size that the hard disk should have.
This is a spindle drive motor controller. This is a non-standard part, so specifications are hard to find, but some controllers seem to have brothers and sisters who have slightly more information available. The most common seem to be the ST Smooth controllers; apart from spindle motor control, they also have several analog/digital channels.
This is a flash memory with a serial interface. It is also easy to use and can be from 64KB to 256KB in size. It seems to be used to store the program from which the hard disk controller starts. Some hard disks do not have this chip, but instead it has flash memory inside the disk controller chip.
These small devices are not chips, but piezoelectronic vibration sensors. They can be used to move the heads to a safe place when HD is subjected to mechanical vibration or shock, but most likely they simply record a bit somewhere indicating that the user has dropped HD and the warranty no longer applies.
The most interesting thing about this element is the hard disk controller. They are made by Marvell, ST and some other large integrated circuit manufacturers. Some hard disk drive manufacturers also make their own controllers: I noticed that Samsung and Western Digital do that. Almost everything else is simple, so I was interested in this device.
Unfortunately, these details have almost no documentation. If I were to say that controller manufacturers are not very willing to share information about them, it would be an understatement: they don’t even mention the existence of these details on their websites! Unfortunately, the rest of the Internet was also not very useful: searching for specifications leads only to specification sites where there is no PDF itself, as well as to mysterious Chinese sellers who claim to have these integrated circuits.
So, we do not have specifications for the most important integrated circuits, which means that we are at a dead end… or not?
We’re hooking up JTAG
Fortunately, besides the specifications, there are other ways to find information about these IPs. As a result of one web request, I found something useful.
On the HDDGuru forums I found a Dejan user thread. He somehow managed to damage the internal flash memory of his hard drive and he wanted to understand if there was a way to boot the controller from the external flash memory or a way to overwrite the flash memory. He couldn’t find an answer within five days, but the guy was inventive: the next post he published was a message that he was able to find a schematic of the JTAG port pinouts.
This is an important finding: the JTAG port can be used to control the controller as a puppet. You can stop it, restart it, modify the memory, set control points, etc. Then Dejan figured out how to create a dump of the controller’s boot ROM, realized that one of the hard disk sockets has a serial port, and was able to recover the flash ROM of his drive. Then he dumped a few more bits and pointers to the process of updating the flash, and then finally disappeared in the Internet mug.
All this proved to be quite useful information: at least it made me realize that all Western Digital controllers seem to have an ARM kernel that can be accessed via the JTAG port. I also learned from this information that hard drives usually have a serial port which is not used most often, but can be useful for debugging my hack. Now I should have enough information to start the hack.
My system looks like this:
The red piece is the FT2232H, a cheap board that you can buy for about $30, it has JTAG and serial port and SPI interface. It is connected to the JTAG interface of the hard disk, as well as to the slot on which the hard disk has a serial port. The hard disk is directly connected to the SATA port on the motherboard of my computers, as well as to an external ATX power supply. I used OpenOCD software to manage my JTAG.
Now the question is: will it really work? Dejan did this with a 2.5 inch 250GB hard drive with the 88i6745 controller and found out that the arm9 kernel was being used. I took a 3.5 inch 2TB drive with the 88i9146 controller and it has a different form factor and a bit newer. Fortunately, OpenOCD is able to recognize for itself what is in the JTAG chain. Here is what I found:
It confused me a bit… I expected one tap corresponding to one ARM core… but found three tap. Does that mean this chip has three ARM cores?
Based on my research, I found out that it looks like the chip has three cores. Two Feroceon cores are quite powerful like arm9 and the Cortex-M3 core is a weaker core, more like a microcontroller. After a bit of experimentation (and further research) I found out that all controllers have their own functions:
- The Feroceon 1 handles the physical reading and writing from/to the hard disk drive plate;
- Feroceon 2 manages the SATA interface;
- Feroceon 2 also works with cache and converts from LBA to CHS;
- Cortex-M3 does… nothing? I managed to stop it while keeping all the functions of my hard drive.
What kernel to start the hack from? My task was to compromise the system protection by modifying the hard disk firmware. The easiest way to implement this, which is probably the hardest to detect, is on the fly data modification. In this case, the data on the disk will not need to be changed, and the firmware can simply make itself invisible. To do this I need to find a kernel suitable for this kind of interception: I need a kernel that has access to the data that is in the process of being transferred from disk to SATA cable, which can be falsified to modify the data while it is between these two points.
How does this data get from HD plates to the SATA interface?
If processors used standard memory copying at 150MHz, they could reach 150*23/2=2.4Gb/s, but in practice they would probably be much less. According to the specification the hard drive has a 6Gb/s speed, so it probably uses some kind of hardware acceleration. The most likely hardware acceleration is DMA. This means that data is copied directly from the head read logic to memory without active CPU involvement. The same is true for the SATA port: the processor just needs to specify where the data is located and DMA logic reads the data directly from memory.
If this were the case, where would the DMA engine point to the memory? A good candidate is the hard disk cache: the data read from the disk still falls into the cache, so it would be logical to copy it there immediately after reading from the disk. Earlier I found out that Feroceon 2 is responsible for working with the cache, so it became the main target of my hacking attempt.
So, I concluded that the data is read and written using DMA, without the processor involved. Now the next question is: even if the processors should not touch the data during normal operation, can they access it? To answer that question, I first used a JTAG connection and disassembling to figure out how to bind a second Feroceon to memory:
As you can see, the memory bind is a bit fragmented. Small pieces of RAM are scattered, there is IO (I/O) and IRQ (interrupts) space, and a bit of internal loading ROM. There is also a large 64 MB segment of what I think is a DRAM chip with a cache on it. Let’s find out if this is the case. First I mounted the disk on my machine and wrote “Hello world!” to a file on it. Will I be able to find this line in the 64 MB memory area?
Yes, here she is. It looks like Feroceon processors have cache access and the cache is tied to a 64 MB DRAM area.
Of course, if I wanted to change something in the cache, I wouldn’t have scanned the entire 64 MB RAM each time: I needed to understand how the cache works. To do that, I needed to dump the firmware on the hard drive, disassemble it, and understand how the cache works.
Disassembling this firmware is a non-trivial task. First, the ARM instructions and thumb style instructions are mixed in the code. It’s annoying and I don’t have a disassembler that can automatically switch between these two sets. Also, there’s something in the code that makes disassembling software very easy: usually, the procedures are encoded to display messages like “Couldn’t open logfile!” when something goes wrong. These messages are very helpful when you know what the procedure is for.
However, there were no such lines in this firmware: you only had to know the assignment of the procedure by its code. However, the code base seems to be a bit outdated, and sometimes the disassembled code looks as if some of the functions were “screwed” to it later, which made the analysis a bit more complicated.
However, there were some aspects that made disassembling a bit easier. First of all, it seems that Western Digital didn’t deliberately obfuscate the code: no tricks like switching to the middle of the manual were used. In addition, thanks to the JTAG interface, you can experiment with the code, set checkpoints and change it on the fly, which made it much easier to determine the purpose of the procedure.
After studying the code for a long time and running the debugger to test my guess, I was able to get to the core of the caching system: a table in the RAM, which I called the “table of cache descriptors”:
Each element in the cache descriptor table describes a cache block. It contains the initial LBA sectors of the disk, which are or should be cached, the cache full with disk data, flags describing the state of the cache element and a number indicating where the cached data is in memory.
Now that I have solved the mystery of the table of cache descriptors, can I intercept the record on the disk before it goes from a SATA port to the computer? To do this, I will need to be able to execute my own code in my hard disk controller. What’s more, I need to ensure that the code runs at the right time: if I change the cache too early, the data won’t get into it yet; if I change it too late, the data will already be sent to the computer.
I implemented this by connecting it to an already existing procedure. My hack will be on Feroceon 2, and this processor will do all the SATA transmissions, so there must be some procedure in it responsible for setting up the SATA hardware to retrieve data from the cache. If I find this procedure, I will probably be able to execute my own code before it.
After a lot of research, checkpoint installation, trial and error, I finally found a procedure that meets my requirements. I modified it to execute my own code first. Here is the original code:
This is what happens when a call to my code is added to the code:
As you can see, some of the original instructions have been replaced with a new code in the previously unused RAM area at 0xFFE3F000 and an additional word to ensure that the code area checksum is correct. If I hadn’t done this, HD would have tried to load a backup copy from its plates, which we don’t need. The code to which the transition is made executes a procedure called changeThingsInCache and then does what the replaced code would do. It then continues executing the original procedure as if nothing had happened.
All I have to do now is write a procedure to change the cached data. For the first test, I decided to write a procedure that would do something similar in a pseudo code:
This small piece of code every time it is called replaces the first 4 bytes of each sector with 0x12345678, so if I downloaded it all to my hard drive, I would have to see this number at the beginning of each sector I read. I downloaded the code snippets via JTAG…
And here is the result:
Security of code
Of course, I could turn this into a complete hack, but because of the need to connect JTAG to the RAM it becomes quite useless every time the hard drive starts up. I had to make it save, i.e. store my modifications in a place where I could take them every time I turn on my hard drive.
For this purpose I chose flash ROM. Probably, I could put it somewhere in the reserved sectors of the hard disk itself, but if I got something wrong, I would not be able to recover the disk. The flash memory chip is just a standard piece with eight leads, so I can easily remove it, flash it and insert it again. To do this I unsolder it and connect it to a circuit board so that I can easily switch between the programmer and the hard disk:
What should we write in flash memory? Fortunately, the format stored in the chip has already been disassembled: it consists of many blocks of data, and at the beginning there is a table describing them. This table describes the location of the block in the flash memory, the way it is compressed (if it is compressed), the place where the block should be placed in the RAM, and for the last address – the execution point to which the loader should go for the program execution.
Unfortunately, I was unable to modify the code in the flash memory; the bits containing the parts I wanted to write my hooks into were compressed by an unknown compression algorithm, so it was impossible to modify them. However, I could add an extra block and change the execution address so that this block could be executed before the others. This made things much easier: when I started executing “my” block, I could just encode it so that it would insert the hookies into the bits of code that had already been unpacked.
Of course, to do that, I had to disassemble and reassemble the flash binary file. I created a tool for that and gave it the boring name “fwtool”. This tool can dump different flash memory blocks and can also translate the title into a text file for easy editing. You can then modify, delete or add a block, and then reassemble everything into a single firmware file ready to be written to the flash memory. I used it to add my own piece of code to the image, flashed it all back into the chip, solder the chip to the hard drive, run it again and get this as a result:
This is not surprising: this is exactly the result I have received before. The trick is that it doesn’t need to be modified via JTAG now.
We are flashing the software
Although modifying flash memory was a good step forward, I still couldn’t implement my imaginary hacking scenario: I don’t think any server company accepts “donations” in the form of hard drives with soldered and re-solder flash chips. I had to find a way to reflash the chip when it is soldered to the hard drive, preferably from the computer to which the drive is connected.
Western Digital’s firmware upgrade tools prove that this is possible: in fact, it’s a tool that runs under DOS to write new firmware to the flash memory and the service area, i.e. to reserved sectors of the hard drive. According to information from the Internet, the tool uses so-called Vendor Specific Commands for this purpose.
There are other tools that can modify firmware: for example, there is an experimental code that can use unused reserved sectors to hide data. There is also a set of tools called idle3-tools which can be used to modify the byte in the firmware to change the behavior of the hard disk in standby mode. This code also uses VSC using the “official” method with the Linux ioctls scsi driver. I decided to “borrow” this code, change it a bit and integrate it into fwtool. After experimenting with the code and selecting the VSC parameters my fwtool program suddenly learned to read and write the flash memory of the hard disk connected to the computer where it was running.
That was the end of my attack. If a blackhat hacker somehow got root access to a server with this disk, he would be able to use fwtool to remotely dump the flash memory to the disk, modify it and flush it back into memory. Sooner or later the owner of the computer would have found out that I was using his machine for malicious purposes and probably would have reinstalled the system, closing the loophole through which the hacker had originally entered.
However, after installing the firmware, the attacker could have ordered the hard drive to do something malicious with the newly installed system. First, he would need to run this behavior, and this could be done with a certain magic string that the hacker would look for on the disk. The magic string could be any file; for example an attacker could upload a .jpeg file with a line in it to the server. He could also request a file from a web server through a magic string attached to the URL. It would then appear in the machine logs, which would trigger the exploit.
Then the hard disk firmware hack would do something malicious. For example, it could wait until the machine starts reading the /etc/shadow file where Unix/Linux stores all passwords, and modify its contents on the fly, replacing it with something that the hacker prescribes. The attacker could then try to log on with his own password and the machine would compare this password to the modified /etc/shadow, allowing the attacker to log on again.
Here is a demonstration that I made for the presentation. You can see me trying unsuccessfully to log in to the root account of the computer. Then I turn on the hack and give him a new password hash, namely “test123”. Since Linux caches the shadow file (like all files recently accessed) I have to generate a strong disk activity to “force” the file out of the cache; thus, when I try to login again Linux will have to retrieve the shadow file from the disk. Finally, by clearing the cache I will be able to simply login to the root account with the false test123 password.
Of course, restoring access to servers that have had secret logon methods closed is not the only useful way to apply my reverse engineering work. It can also be used for protection purposes.
For example, you can create a non-clone hard disk: the hard disk will function normally if the sector access pattern is random, in the same way as the OS accesses the file system. If the disk is accessed only sequentially, as the disk cloning utility usually does, the hard disk may distort the data, making the clone different from the original.
The disk controller is interesting and simple as a controller card. It has three quite powerful processor cores, to which is connected a fairly large amount of RAM. It also has a uart for a serial port and at least two SPI interfaces; one for flash ROM and one for spindle motor controllers. It is possible to download code for the CPU by updating the data on the external flash chip or even using the serial in the bootloader. To demonstrate the power of the chip, I ported quite popular software to my hard drive. The demo is just a concept check, the only working peripheral is the serial port and there is no user space yet. However I am still a bit proud of what I have installed on my Linux hard disk.
I will tell you a little bit more about what is going on here: the kernel and init are each packed in pieces of exactly one sector and at the beginning there is a magic line and an ordinal number added. When reading a file from the disk, it finds itself in the disk cache. Writing the magic line “HD, lnx!” makes the modified firmware look for all sectors in the cache, rebuild the kernel image and load it. The kernel is built for a processor without a MMU (the disk controller does not have one) and it has only a driver for the serial port. Unfortunately, a kernel without MMU also requires specially formatted user space. I was not able to compile it, so the kernel eventually produces a kernel panic, because it can’t find a fit to execute it.