Someone who is, figuratively speaking, "off the reservation" is beyond the control of his or her supposed leader.
The word "reservation" is used in two senses: animal preserves, where hunting of animals is forbidden, and reservations for Native American Indians, upon which these people are permitted by national governments (be they American or Canadian) to live largely according to traditional tribal laws rather than under the authority of state or provincial governments.
In either case, to be "off the reservation" is to be beyond a well-defined boundary.
Example: "Richard was speaking to a reporter about his division's new product. Either his boss knows about it and this was planned, or Richard is completely off the reservation and will get in very serious trouble. Unless he had approval, he could lose his job over this."
In this case, Richard is either part of a plan to quietly announce the product to the news media, or he has leaked in violation of his company contract. The latter would definitely be beyond a well-defined boundary!