It's only tracking logged-in people who listen to the podcast right now. And just like Youtube, there's an anti-spamming feature to prevent noise in the data. So basically, someone cannot just come and refresh the page again and again to boost the count (it's actually pretty standard procedure now when making websites of all kinds). But if you listen to it every 12 hours, it will go up.
As for the downloads, CC has the numbers via mediafire, but I could add a counter as well on top of the link (thus, no need to link straight from this website Shuuichi ). Is it a feature you guys would like to see as well?