The Making of the Atomic Bomb by Richard Rhodes ($14 at Amazon) was a book I picked up for travel, though at 928 pages it's not a travel sized book. This starts almost at the beginning with discussions of radioactivity and the state of physics at the start of the 20th century, and introduces in turn all the people that end up being major players in the development of fission and the atomic bomb as well as nuclear policy (Fermi, Bohr, Oppenheimer, Szilard, Groves, Rutherford, and of course, Truman).
I won't begin to say that I understand atomic physics, but I enjoyed the standing on the shoulders approach that brought about great leaps in the early 1900's all the way through 1950. Fission was a dream, and the hurdles to it were considerable. I think one fun/scary part was the first atomic pile (reactor) that was built in Chicago. I remember it vaguely from some bit of history, but they built this in pretty low tech fashion, hiring football players to stack the carbon blocks needed. When they began testing they had a control rod and a meter, and they just kept pulling out the control rod until the pile went critical. They had a hacked together fail-safe system, but in truth it was one very smart guy managing a chain reaction by telling someone to raise/lower a control rod. Luckily he was smart enough, or Chicago could have become a bad place to live.
When it moves into the Manhattan project era the scale of the problems and solutions is staggering to me. Over the course of a couple years we built every type of process there was for creating weapons grade uranium and plutonium. The cost ended up being equivalent to the US auto industry in size and spend, more than $2 billion in 1940's dollars. They basically solved the problem by saying if you could get an ounce or whatever of material from a single million dollar plant, just build a 100 of them to increase production.
It's also here that you start to understand why nuclear material is such a big deal. The first bomb (Little Boy) was essentially a gun, they fired one chunk of uranium into another. I won't say it's trivial, but the engineering was doable and then and I suspect easily doable now, the hard part was getting the weapons grade material. Plutonium is a different story, not all is it hard to produce but they opted for an implosion mechanism that wasn't trivial then or now - imagine building explosives that will perfectly and evenly compress a sphere the size of a softball to the density of the sun in a lot less than a SQL millisecond. Note that a "shake" was a term coined during all of this that equates to 10 nanoseconds!
This also shows the challenge of science, sometimes it leads to things that can have negative consequences, but hard to stop discovery from happening. And it talks about the beginning of the arms race and from my reading of it, it seemed almost inevitable. Interesting but sad.
If you'd like to just learn more about the bomb from a simpler perspective try Sum of All Fears by Tom Clancy. In that novel terrorists obtain the material and set about building a bomb. It may not be perfectly accurate, but it seemed to me that he did a pretty good job of research to present the story.
I'm glad I read this, but it's a dense read, took me more than 4 weeks of stopping and starting to get through it.