
One of the trademarks of a dysfunctional family is that they tend to put each other down, and living in the family has a tendency to resemble a 'survival of the fittest' sort of scenario. Parents will often pit children against each other / compare children to each other , very rarely in a positive way.
" Your brother would know. " is sort of short for " your *brother* would know, because unlike *you* he isn't a stupid freak, and why the hell can't you do anything right and Jimmy always knows the answer so why don't you? "
... mostly. It's an attack against one child, implying they aren't as smart/good/knowledgeable as the brother.