I suppose so, but then, most sites you link to will have referrer tracking enabled. For example, all WP.com blogs have referrer tracking possible. In general, if you really want to keep your blog private in this sense, then, you need to not make links.