with rocks in it
The answer is that the cement is not as strong as the other materials
Yes, individual rocks are much harder than cement and individual grains of sand (tiny rocks and crystal) are also much stronger. What happens when you mix in other materials is that a fracture of the cement must take a much longer route to break a piece of concrete apart per unit area of the concrete. It's easier to understand with a picture.
(rollover image - series of 4)
Obviously, if the bond from cement to rock is roughly equal to cement to cement but less than that of the internal rock bonds ... then ... for the resulting concrete to fracture it will have to go around the harder rocks meaning that many more cement-cement and cement-rock bonds will have to be broken per unit square of fracture surface. Therefore, a unit brick made of pure cement will be weaker than the same unit brick containing sand & rocks.
But you have to be careful about the size of the rocks so that to fracture it will require the most bonds. That's a mathematical problem ... but for sidewalks it must be about "thumb size" or smaller.
"Where the rubber meets the road"