Let’s Build a Hexdump Utility in C
Let’s build a hexdump utility in C — this is a good beginner-level project if you’re learning the language for the first time.
A hexdump utility lets you look at the individual byte values that make up a file. Here’s some sample output from the utility we’re going to build, using its own source file as input:
The value of each 8-bit byte is displayed as a pair of hexadecimal (i.e. base-16) digits. Why hexadecimal? The number of distinct values any sequence of digits can represent is given by the base to the power of the number of digits in the sequence. The 8 binary digits in a byte can represent 2 8 or 256 distinct values, usually interpreted as the numbers 0 to 255. Similarly, a two-digit hexadecimal number can represent 16 2 distinct values — also 256. This convenient correspondance is why programmers like hexadecimal notation so much — it gives us a compact way of representing all possible byte values with just two digits. 1
Setup
I’m going to assume for this tutorial that you’re relatively new to programming in C. I’m not going to try to teach you the language itself, I’m assuming you know the basics already, but I am going to explain each step of the build process in detail, including how to set up and compile the project files. If you already have your own preferred way of doing things you should go ahead and use it — these instructions are intended for beginners.
I’m also going to assume that you’re working on the command line on some kind of unixy system — Linux, Mac, or any flavour of BSD for example. If you’re using Windows you can follow along by using the WSL.
Note that you can find working demonstration code for this tutorial on Github. This code has been placed in the public domain.
First Steps
First let’s make sure we can get something (anything!) compiling. Create a new directory for the project, cd into it, and create a new file called hexdump.c .
(You don’t need to type the $ symbols — I’m using them to indicate that this is input we’re typing in at a shell prompt.)
Next, add some simple hello world code to the file so we can verify that our toolchain is working as expected:
On my system I can compile this file by typing the following command:
This produces an executable binary called hexdump in the same directory as the source file. I can run this binary with the following command:
cc is the traditional name for the system’s C compiler. On my system it’s actually an alias for clang ; on yours it might be an alias for gcc , or your system might not ship with a builtin C compiler and you’ll have to install one yourself.
You should make sure that you can run this code (or something very like it) on your system before going any further!
If you haven’t used a command-line C compiler before, don’t worry — it looks intimidating at first but it’s actually quite straightforward once you get used to it. I remember when I was learning C for the first time finding it frustratingly difficult to get my hands on a clear, simple guide to how the compiler worked, but you’re in luck as I’ve written up an introductory cheatsheet which should help you get started.
Organising the Project Files
Okay, now we know we have a working toolchain that can compile C code. Next we’re going to organise our project files and set up a makefile to run the compiler for us.
First, create a src directory and move the hexdump.c file inside it:
Next, create a file called makefile and add the following lines:
Don’t just copy and paste the code above — it won’t work! The indent in a makefile needs to be an actual tab character and my website replaces tabs in code samples with spaces. 2
Now we can compile our project by running the make command from the project’s root directory:
This will compile the src/hexdump.c file and put the output binary in a directory called bin . We can run the binary by typing:
(I’m assuming that make is already installed on your system. It usually is, but if it isn’t you can download a copy from your operating system’s package manager.)
make is an extremely useful tool — it’s really designed for managing the dependencies between files in complex projects and we’re only going to scratch the surface of its capabilities here. If you haven’t used it before this tutorial is a good overview for beginners. (I’d also recommend adding the manual to your reading list.)
So what just happened? Looking at our makefile, the line:
is called a target. 3 A makefile can have multiple targets — we can tell make to run the recipe below a particular target by specifying its name as a command line argument:
If we don’t specify a target make will default to running the first target in the file, which is why we can now compile our project just by typing make .
The target’s recipe is a set of shell commands — we can put any commands here that we could type in at a shell prompt. The first command:
simply creates the bin directory if it doesn’t already exist. make normally prints each command to the shell before running it; I’ve put an @ symbol at the beginning of this line to tell make not to print it as I don’t want this kind of boring housekeeping code cluttering up my shell.
The next line is the actual compiler command:
We could have typed this line in the shell ourselves and achieved exactly the same result. One advantage of using make is that it’s shorter to type. A more important advantage is that when we come back in six months and want to recompile our project we won’t have to remember whatever complicated set of flags and commands we ended up using last time — it will all be written down for us in our makefile.
Adding a Library
Our hexdump utility is going to support a handful of command line options, e.g. a —num <int> option for specifying the number of bytes to read. Parsing command line options in C is a pain — there’s no builtin way to do it in the standard library — so we’re going to use a simple library I’ve written for the purpose called (imaginatively) Args.
Using the library is easy. You need to download two files, args.h and args.c , and add them to your project’s src folder. You can download these files from the tutorial repository on Github.
Next, change the compiler command in your makefile to the following:
This tells the compiler to compile the hexdump.c and args.c files individually and link the two resulting object files together into a single executable.
We can make sure the library is working properly by adding support for —help/-h and —version/-v flags to our executable.
Open the hexdump.c file and change its contents to the following:
I wouldn’t normally advocate cutting and pasting sample code but that helptext literal is an exception. The C language developers haven’t gotten around to supporting multi-line strings yet (any decade now) but we can hack it, kinda, sorta, by using the fact that C concatenates adjacent string literals.
If you recompile the code and run the binary with a -h or —help flag:
you should see the help text printed. Similarly if you use a -v or —version flag you should see the version number printed.
I’m not going to explain how the argument-parsing library works in detail here — you can read the documentation if you’re interested. The important point for us is that the library will handle the messy process of parsing the command line arguments, checking if they’re valid, and converting any option values into integers.
Writing the Code
It’s been a long road but we’re finally ready to begin writing our application code! Here are all the standard library #include statements we’re going to need, you should add them to the top of your hexdump.c file:
And here’s our finished main() function:
Most of this code is concerned with setting up and then processing our various command line options.
We default to reading from stdin if a filename hasn’t been specified by the user. If a filename has been specified we try to open the file, exiting with an error message if anything goes wrong.
If the user has specified an offset using the —offset <int> option we try to seek to that offset in the file. If this fails for any reason we exit with an error message. (This code as I’ve written it only supports seeking forward to a positive offset from the beginning of the file. If you wanted to enhance it, you could interpret a negative offset value as meaning the user wants to seek backward from the end of the file.)
We hand the actual job of hexdumping the file off to a dump_file() function which we’ll look at next. The bytes_to_read argument specifies the number of bytes we want this function to read — I’m using the ‘magic’ value of -1 to indicate that we want to read all the way to the end of the file.
Here’s the code for the dump_file() function:
We begin by allocating a buffer to hold a single line of input from the file. The loop then reads a single line of input per iteration into this buffer and hands it off to a print_line() function to display.
We have to do an elaborate little dance to figure out the maximum number of bytes we want to read per iteration. Generally we’ll want to read up to one full line of bytes, but we may want to read fewer on the last iteration if the user has specified a particular number of bytes to read with the —num <int> option.
The fread() function returns the number of bytes read. If this value is zero we’ve reached the end of the file (or the end of the block the user wanted to read) so we break from the loop.
Here’s the code for the print_line() function that displays the output:
We begin by printing the line number which is given by the offset variable. We then loop over the buffer and print each byte value formatted as a two-digit hexadecimal number (or a spacer if we’ve run out of bytes). We add an extra space before each group of four bytes to make the output easier to read.
The second loop prints the ASCII character corresponding to each byte value if it’s in the printable range, otherwise it prints a dot.
That’s it, we’re done! If you run make one more time you should have a working hexdump utility in your bin folder.
Final Thoughts
I’m sure you can think of ways to improve and expand on this code.
I’ve used ASCII lines and dots for maximum compatibility but you could use unicode dots and box-drawing characters instead to make the output look prettier.
Adding support for negative offsets lets you use an option like —offset -128 to easily view the last 128 bytes of a file. This capability can sometimes be very useful.
I’ve built a slightly more sophisticated hexdump utility of my own called Hexbomb which might give you some ideas to work from.
Links
You can find working demonstration code for this tutorial on Github. This code has been placed in the public domain.
- Demonstration Code
- C Compiler Cheatsheet
- Hexadecimal Numbers
- The Make Manual
- Windows Subsystem for Linux
- Arg Parser Documentation
- Hexbomb
Notes
Actually, it’s even better than you might suspect at first. Each hexadecimal digit aligns cleanly with four bits of the corresponding byte so the hexadecimal number 0x12 corresponds to the byte 0001_0010 and the hexadecimal number 0x34 corresponds to the byte 0011_0100 . This makes it really easy to read bit patterns directly from hex notation — at least after you’ve had a little practice!
If you haven’t met it before, the 0x prefix is used to indicate that a number is written in hexadecimal base. Similarly, 0o can be used to indicate octal (base-8) and 0b to indicate binary (base-2). If you want to play around with different number bases you might find a little utility I’ve written called Intspector useful.
Make’s reliance on hard tabs has been annoying programmers for more than forty years at this point; it will probably continue annoying us for another forty years at least. It’s been famously described (by Eric S. Raymond in The Art of Unix Programming) as «one of the worst design botches in the history of Unix».
Technically this is called a phony target as it doesn’t correspond to a file name. (In general a make target is a filename and the recipe that follows is a set of instructions for building that file.) Phony targets are useful for handling project management tasks — common examples include make check for running a project’s test suite, make clean for deleting temporary build files, and make install for building a binary and installing it on a user’s system.
Hexdump в c что это
Команда hexdump предназначена для вызова одноименной утилиты, осуществляющей вывод содержимого бинарных файлов. При этом помимо стандартного формата вывода, знакомого по hex-редакторам, утилита поддерживает различные экзотические форматы вывода, а также позволяет пользователю самому описывать необходимый ему формат вывода. При установке пакета с утилитой в системе создается символьная ссылка с именем hd, позволяющая вывести содержимое бинарного файла в классическом формате.
Базовый синтаксис команды выглядит следующим образом:
$ hexdump [параметры] имя-бинарного-файла
Утилита позволяет задать длину исследуемого фрагмента файла в байтах с помощью параметра -n, сдвиг от начала файла в байтах с помощью параметра -s, задать строку форматирования с помощью параметра -e, а также выбрать формат вывода с помощью одного из описанных ниже параметров. Чтобы было понятнее, для начала создадим текстовый файл со строкой «linux-faq.ru» с помощью следующей команды:
$ echo «linux-faq.ru» > hexdump.txt
Проверим содержимое этого файла:
$ cat hexdump.txt
linux-faq.ru
Теперь используем команду hexdump без каких-либо параметров для исследования этого файла:
$ hexdump hexdump.txt
0000000 696c 756e 2d78 6166 2e71 7572 000a
000000d
Это используемый по умолчанию, неудобный формат вывода. Очевидно, что в столбце слева приведены 32-битные адреса, а после них следуют шестнадцатеричные представления пар байтов. Эквивалентный результат будет выведен при использовании параметра -x, правда с немного измененным форматированием:
$ hexdump -x hexdump.txt
0000000 696c 756e 2d78 6166 2e71 7572 000a
000000d
Используем параметр -b для исследования файла:
$ hexdump -b hexdump.txt
0000000 154 151 156 165 170 055 146 141 161 056 162 165 012
000000d
В этом формате после шестнадцатеричных значений сдвигов выводятся восьмеричные значения каждого из байтов файла.
Используем параметр -c для исследования файла:
$ hexdump -c hexdump.txt
0000000 l i n u x — f a q . r u n
000000d
В этом формате после шестнадцатеричных значений выводятся непосредственно символы.
Используем параметр -d для исследования файла:
$ hexdump -d hexdump.txt
0000000 26988 30062 11640 24934 11889 30066 00010
000000d
В этом формате после шестнадцатеричных значений сдвигов приводятся десятичные представления пар байтов.
Используем параметр -o для исследования файла:
$ hexdump -o hexdump.txt
0000000 064554 072556 026570 060546 027161 072562 000012
000000d
В этом формате после шестнадцатеричных значений сдвигов приводятся восьмеричные представления пар байтов.
Наконец, используем параметр -C для исследования файла:
$ hexdump -C hexdump.txt
00000000 6c 69 6e 75 78 2d 66 61 71 2e 72 75 0a |linux-faq.ru.|
0000000d
Это классический формат hex-редакторов, который предусматривает вывод шестнадцатеричных значений всех байтов файла с соответствующими им символами.
Примечание: существует команда hd, реализованная в виде символьной ссылки на бинарный файл утилиты hexdump с параметром -C. Ее удобно использовать для исследования бинарных файлов, так как не приходится запоминать необходимый параметр утилиты hexdump.
Примеры использования
Исследование заголовка бинарного файла
Для исследования заголовка бинарного файла следует использовать параметр -n, позволяющий задать длину этого заголовка, а также параметр -C, активирующий удобный формат вывода:
$ hexdump -n 41 -C hexdump.zip
00000000 50 4b 03 04 14 00 08 00 08 00 55 b0 4d 4d 00 00 |PK. U.MM..|
00000010 00 00 00 00 00 00 0d 00 00 00 0b 00 20 00 68 65 |. .he|
00000020 78 64 75 6d 70 2e 74 78 74 |xdump.txt|
00000029
В выводе несложно обнаружить имя находящегося в архиве файла hexdump.txt.
Исследование завершающей секции бинарного файла
Для исследования завершающей секции бинарного файла следует использовать параметр -s, позволяющий указать длину сдвига от начала файла в байтах, а также параметр -C, активирующий удобный формат вывода:
DWise1’s Programming Pages
This is one of those computer things that you either know about and know is important, or you have no idea whatsoever. Kind of like my brother-in-law with decades of experience writing on a Mac being unable to understand what a text editor is for and why anyone would possibly want to create a plain-text file.
I’ve been working with computer dumps ever since I started my computer science classes in 1978, so the concept is perfectly natural for me. At that time, when our job on the school’s IBM S/370 would ABEND («end abnormally», AKA «fail» or «bomb»), our printout would include a «post-mortem» dump, AKA a «core dump». This would be a printout of the computer’s state at the time of the ABEND, including the register contents and a hexadecimal dump (AKA «hex dump») of our program’s portion of memory. By reading that printout we could see what our program was doing and so figure out what had gone wrong and how to fix it. In fact, I read so many postmortem dumps in school that I became fluent in reading the EBCDIC character set, which is unfortunate because I’ve had to use ASCII ever since leaving school and I’ve never been able to achieve the same level of reading fluency.
As the years have gone by, I’ve encountered many applications that need to present binary (unprintable) data and the standard format they use is the «hex dump». I have frequently needed to view a hex dump of a binary file in order to see exactly what was being written to it.
Hex dumps are a vitally important tool for programmers and a feature that some programmers would want to add to their programs. Even though there’s nothing really complicated about it, I would like to present the C functions I had written to generate a hex dump for one project.
General Format of a Hex Dump
- Offset/Address — The location of the data. In a file or a buffer, this would be the offset from the beginning of the file/buffer. In a memory dump, this would be the memory address. Since each line of a dump displays multiple bytes — usually a nice, round number like 16 (this is hexadecimal, after all)— the offset/address corresponds to the first byte in that line.
If the buffer being dumped is small enough (eg, when dumping the data contents of a TCP/IP packet), this section may be omitted.
- Hex Data — The data itself. The 256 possible values of each 8-bit byte are represented by two hexadecimal digits. The actual formatting of the line can vary with each byte separated from the others by spaces, paired together as 16-bit words (with bytes reversed or not), with or without additional spacing and/or punctuation between the 8th and 9th bytes, etc.
Since this section is the raison-d’être of a hex dump, it should never be omitted.
- Interpretation of Character Data — This section provides a convenient display of any text data that may be embedded in the dump. Whether the bytes are interpreted as ASCII or EBCDIC will depend on the system, but in most cases that you encounter it should be ASCII. Basically, if the byte is a printable code, then it will be displayed. Otherwise, unprintable codes will be replaced by a standard character, usually a period («.»).
A perennial problem stems from the characteristics of the character sets. For example, ASCII is only defined for 7-bit characters; there is no universal standard for ASCII codes 128 to 255. Now, a number of manufacturers have defined «extended ASCII character sets», but each one is different. For example, for DOS Microsoft defined its OEM font set, more properly refered to as Code page 437. However, the extended OEM characters are different from Microsoft’s «ANSI» code page for Windows (misnamed, since it’s not an ANSI standard). If the extended characters are displayed, then different characters will be displayed in a DOS app than in a Windows app. Some hex dumps will only print 7-bit ASCII codes while some will print 8-bit codes, and some will allow you to choose between 7-bit and 8-bit displays.
Also, ASCII codes 0 to 31 are control characters, the actual display of which could prove chaotic. To see why, try this experiment: from the command line, use the TYPE command to display a binary file, such as an .EXE file. You will see «garbage» flow past you and you will hear some beeping noises (that’s BEL, ASCII code 7) and you may even see some tabbing, backspacing, form feeds, etc. And if you do this in a Linux terminal, then the control codes could set the terminal into a completely hosed-up state — we’ve done that a few times. Not exactly how you would want your application to behave. Since control characters are unprintable, most hex dumps will just replace them with a period. However, since Code page 437 (AKA «OEM») did define printable counterparts to each control character, some DOS dumps will display them.
This section may be omitted in some dumps; eg, in a special-purpose dump that’s guaranteed to not contain ASCII data.
Hex Dump Examples
Vern Buerg’s List Utility
Anyone who remembers the early days of MS-DOS will also remember that many of the basic utilities that we take for granted today, like a screen editor or a file lister, were only available from third-party vendors. When this list utility first appeared in 1983, it was just such a third-party product and it quickly became an indispensible tool.
List includes a hex-dump mode, such that with a single key-stroke you can switch between viewing the file as text or as a hex dump. This becomes especially handy because you can open a binary file in List, in which case you just get «garbage», but then with an alt-H you go into hex-dump mode. For example, an EXE file produces this hex dump:
Note that the first two bytes are the ASCII codes for «MZ», which are the initials of Microsoft programmer, Mark Zbikowski, one of the developers of MS-DOS. That’s the EXE file signature and every EXE file starts with those two letters.
Note also that the OEM characters (AKA «Code page 437») are displayed for the byte values below 32 and above 127. List refers to this as the «8-bit mode». It also has a «7-bit mode» which only displays character for the byte values 32 through 126 (0x20 to 0x7E); for any byte value outside that range it displays a period. Here’s the same portion of the same file, only this time in 7-bit mode:
UNIX/Linux od Utility
You can see that it has the address/offset section and the hex dump itself, but not the ASCII interpretation section. Actually, you can have it output in ASCII and embedded binary codes, but I will leave that exercise to the reader. Read od’s man page for details.
The other thing you’ll see is that it doesn’t display the data one byte at a time, but rather one 16-bit word at a time. Furthermore, you’ll notice that the bytes are reversed, in Intel «little-endian» order, least significant byte first. Remember from above that «MZ» is 0x4D followed by 0x5A? Look at the first word in the dump and you will see that those bytes are reversed. So if you use od to produce hex dumps, you need to bear that in mind. Personally, I prefer not to use it.
And if you look more closely, you see that the location counter section appears odd. Each location looks like it’s twice as much as it should be — compare it with List above or xxd below. The reason is that, even though it’s displaying the data in hexadecimal, it’s still displaying the location counter in octal. Just something else for you to keep in mind if you decide to use od.
Several years ago, I found another Linux utility, xxd, which produces a much cleaner hex dump than od does. I was also able to find a Win32 port for it and it has become my standard command-line utility for generating hex dumps. Unfortunately, I completely forget where I got it from and Google’ing has failed to uncover that source again. Sorry.
Again using that same EXE file as the example, here is the dump produced by xxd:
Note that, even though the data is again displayed as 16-bit words, this time it’s displayed in «big-endian» format with the most significant byte first.
MS-DOS debug Utility
- Invoke the debugger by typing debug followed by the name of the file you want to dump.
- While you’re in the debugger, it prompts you with a hyphen.
- To do a dump, type d and press the ENTER key. A one-page dump will appear (see example below).
- To dump the next page, enter the d command again. Repeat for however long you need to.
- To exit the debugger, you quit by typing q at the hyphen prompt and pressing ENTER.
The first thing you’ll notice is the location section. Instead of telling you the offset into the file, it tells you the memory location that it was loaded into. Keep in mind that this is a debugger that we’re subverting into being a hex dumper. As a debugger, it will load an executable file into memory and run it step-by-step while allowing you to examine the registers and memory locations. It can even take machine code and unassemble it. It’s a powerful little tool, even if it is one of the more difficult debuggers to use.
Another thing you should have noticed when comparing it to the other dumps is that the first 4 lines are missing. That’s 64 bytes (0x40). Well, every EXE file starts with an EXE file header. That header contains information that the loader needs to load the program into memory and prepare it for execution. The header entry at location 8 gives the size of the header in units of 16-byte paragraphs. Go up to the xxd dump and you will see that location 8 contains a 0x04. 4 times 16 is 64, so the header is 64 bytes long, or 0x40 hex. And that’s all that happened; the 64-byte header was stripped off when the program was loaded for execution under the control of the debugger.
Does the same thing happen to a non-executable file? No, it doesn’t. For example, here are debug dumps of a Java source file, Test.java, and its compiled class file, Test.class:
You will have to trust me when I tell you that the very first line of the Java file is that import statement that you read in the ASCII section. But you won’t have to trust me about the class file; the first four bytes are of a Java class file is its file signature, which is 0xCAFEBABE. Hoo-rah!
But if you look at the command-line invocations you will see something that you should have already realized from the location counters: debug is a 16-bit application. It does not support long filenames. It uses segmented-memory addressing. It is just the old 16-bit utility thrown in and bundled with XP. If you want to use it on a file with a long name, you will need to discover what its short name is; you can do that with the DIR command using the /X command switch:
OK, so debug is cumbersome and has limitations. But if you’re on a machine that doesn’t have any hex dump utilities on it, then knowing about debug will save the day for you.
Editors
First, the hex dump display is read-only; you cannot edit the file through this display.By the way, notice that it’s displaying in 8-bit mode, but it’s using the Windows ANSI code page to do it. In comparison with the OEM code page, that not only means different characters for ASCII codes greater than 126, but also that there are no characters defined for the control codes less than 32 (space).
- Hex editor, which describes what a hex editor is.
- Comparison of hex editors, which lists several hex editors that are available and compares their features.
Other Applications
- debuggers — at the very least, display of memory contents is usually as a hex dump
- network sniffers — packet sniffers such as Ethereal (reborn as «WireShark») will display packet contents in hex-dump format
- binary editors — in particular, I’m remembering the old Norton Utilities’ DiskEdit which would display a disk sector in hex-dump format and allow you to edit it.
My Own Modest Contribution
At the time, I was still not certain of exactly what I would find in the response packet, particularly with regard to the byte order. So I wrote my hex dump functions to display exactly what I was receiving from the time server.
This is a sample run of my UDP Time Client, udptimec:
Because of the small size of the packet, 48 bytes, I left out an offset counter. However, for completeness, I have added that feature in the code below.
HexDump — this is the entry point of the routine that the application code calls. It controls the overall printing of the hex dump and calls WriteHexLine to print each individual line. It accepts from the application the location of the data to be dumped, how much data there is (in bytes), and the location/offset that this block of data starts from. This allows for HexDump to be called multiple times in order to dump a large amount of data one block at a time, but it also requires the calling function to keep track of the location/offset count.Sample Hex Dump Application
This program simply opens the file you provide in the command-line invocation, then reads it in one block at a time (2048 bytes) and hex-dumps that block. As you can see, it keeps track of the offset counter, ulOffset. If you change the block size, be sure to keep it a nice round figure; ie, a multiple of 16.
Share and enjoy!
First uploaded on 2007 November 21.
Updated on 2011 July 18.What does `*` mean using hexdump -C?
I’ve been doing an exersice in attempt to understanding some of what’s going on under the hood of a program. I wrote a small C program, and compiled it on i386 Linux (Ubuntu 12.04) using gcc . I then did a hexdump -C on the output to text file. I noticed that there were some gaps on the offset with an * :
My question is, how should I interpret the * ? I assume it means there’s a gap in the file, but then the question turns to why does a gap exist? Is this part of the standard format of an ELF file?
1 Answer 1
Like in the standard od command or hd , it means all the elided lines are the same as the preceding line. You can pass -v to make it display those lines anyway. From hexdump(1) :
The -v option causes hexdump to display all input data. Without the -v option, any number of groups of output lines, which would be identical to the immediately preceding group of output lines (except for the input offsets), are replaced with a line comprised of a single asterisk.
