This is a modern, informal version of "You must be joking." It is an expression of disbelief regarding something another person has said.
Example: "I heard that Mary-Ann is going to marry Frank in two months." "You've gotta be kidding me! There's no way she'd get together with a boring man like that!" This is an expression of complete disbelief.
Although idioms in general are not "formal English," expressions like "gotta be" ("have got to be") are informal; they are slang, and should never be used in formal situations. This particular idiom/ expression is nonetheless extremely common in American English.