The most compelling evidence against Kogoro is the fact that he has been oblivious in his internal monologues. With enough imagination, ideas could be come up with to explain everything else, but I agree that Aoyama-sensei wouldn't troll us with a deceitful internal monologue.