While I think that the IHRA definition is a good working start, I wanted to put down some of my ideas about anti-Semitism which might help clarify how I understand the term.
First, the word was coined specifically to mean "against Jews." Look it up. It was never meant to be "against anyone who is called a semite".
Second, the Jews who are those who are defined as Jews by Jews and the canon of Jewish law. Deciding that a group is or isn't the "real" Jews because of some other set of beliefs is not in the scope of this piece of writing.
Anti-semitism (and yes, I will be wonderfully inconsistent in my capitalization and hyphenation practices -- sorry, not sorry) is speech or actions which posit inherent negativity in Jews or Judaism (or both or a combination of aspects of each). If a guy kills a bird, and the guy happens to be Jewish, identifying him as Jewish in order to add detail to the story is already problematic. Would one identify the culprit by any other religious marker? Why does that fact become appropriate or relevant? Adding it in in an effort to connect the Jewish aspect of identity with the behavior is to operate negatively against Jews.
So simply pointing out the otherwise irrelevant religious affiliation is a problem. Pointing out the actions of a person as a function of his religion (when that is neither accurate nor in evidence) is anti-Semitism. This sometimes happens in conjunction with stereotypes but can happen in most any case. Trying to represent Jewish law in a negative light is inherently anti-Semitic.
A Jew can be equally as anti-Semitic as a non-Jew. If the intent is to criticize the religion without understanding the religion, or the religious practitioners by associating actions which are not based in religion with the religion, that is problematic.
Zionism is the movement to ensure Jewish autonomy in an ancestral homeland. Criticizing that (in order not to be anti-Semitic) would have to be done with a consistency that would criticize any nationalist movement. Claiming that Israel's government acts in a certain way against non-Jews as a homogeneous group then puts the government's behavior in the context of "Jewish action" even though the Israeli government is full of non-Jews. So isolating the jewish identity piece and claiming that it is what drives Zionist actions can be anti-Semitic. Holding Israel to a standard that is not applied elsewhere is clearly problematic and possibly anti-Semitic.
So ascribing a behavior to someone's Jewish identity (unless you can prove that it was done as a function of that identity/belief, as defined by the Jew, himself) is anti-Semitic. Ascribing beliefs and actions to Judaism as a religion (unless you can confirm that they are, as defined by Jews who practice the religion) is anti-Semitic. Denying Judaism the right to its own identity and autonomy as a valid belief system is also a problem, as is denying Israel a right to exist as a Jewish state by applying a standard which is not applied to other contemporary political creations.
Thoughts, as long as they stay inside a person's head, might or might not be anti-Semitic, depending on whether they are ever expressed in speech or action.
Just some ideas off the top of my head. I'm sure there is more.