Think about it like this. The average visitor to your site is going to speak Slovak, right? If you link to a page in English, or Russian, or Urdu, or whatever, the search engines have to make a judgment call about the likelihood that a visitor to your page will find that linked page valuable. My guess is that they would value a link to a page written in a similar language or English more than a link to a page written in a very different language because the average visitor to your site speaks Slovak, probably has some knowledge of related languages, and might also be able to read English, but they probably aren't able to read Kazakh or Spanish.

The reason I say that is that Google knows that my site, for example, is written in English, but that I talk a lot about French and German, so there's a good chance that people that visit my site have at least a passing knowledge of those languages, so if I write a blog post about a new German movie and then link to a review of it in German, I would guess that that link would pass more equity than if I linked to a review in Chinese because a link to a Chinese review holds no value for my average visitor.

This is really just speculation...

It sounds like you're solid on the on-page, site architecture side. I would assume that crawling and indexation will slow down though if your offsite signals don't keep up though. By this, I mean that Google might see that you're doing everything right on your end, but that over time you're not creating content that very many people care to link to, share, etc, so they'll stop wasting resources on you.

Why wouldn't you just implement rel=publisher? If it's a brand responsible for the content, my understanding is that that would be the appropriate thing to do.

Think about it like this. The average visitor to your site is going to speak Slovak, right? If you link to a page in English, or Russian, or Urdu, or whatever, the search engines have to make a judgment call about the likelihood that a visitor to your page will find that linked page valuable. My guess is that they would value a link to a page written in a similar language or English more than a link to a page written in a very different language because the average visitor to your site speaks Slovak, probably has some knowledge of related languages, and might also be able to read English, but they probably aren't able to read Kazakh or Spanish.

The reason I say that is that Google knows that my site, for example, is written in English, but that I talk a lot about French and German, so there's a good chance that people that visit my site have at least a passing knowledge of those languages, so if I write a blog post about a new German movie and then link to a review of it in German, I would guess that that link would pass more equity than if I linked to a review in Chinese because a link to a Chinese review holds no value for my average visitor.