Reverse engineering is about understanding something, in our case either a software binary, a fileformat, a network protocol or similar, with no other documentation than the actual binary. You do that using debuggers, disassemblers and tools that can monitor the environment in which the binary is running.
A great book to get you started with this disciplin is this one:
You should also visit the following urls:
I use the following software for reversing: