Если надо игнорировать знаки препинания, то:
import re
text = "Around the new position a circle, somewhat larger than in the former instance, was now described, and we again set to work with the spades. I was dreadfully weary, but, scarcely understanding what had occasioned the change in my thoughts, I felt no longer any great aversion from the labor imposed. I had become most unaccountably interested nay, even excited."
matches = re.findall(r"\b\w+\b", text) # Находим все слова
freqs = {word: matches.count(word) for word in set(matches)} # Составляем требуемый словарь
import json # Фишки для красивого вывода
print(json.dumps(freqs, indent=4))
Вывод:
{
"larger": 1,
"new": 1,
"great": 1,
"labor": 1,
"longer": 1,
"dreadfully": 1,
"scarcely": 1,
"set": 1,
"spades": 1,
"to": 1,
"understanding": 1,
"from": 1,
"most": 1,
"a": 1,
"we": 1,
"former": 1,
"work": 1,
"in": 2,
"the": 5,
"was": 2,
"and": 1,
"imposed": 1,
"interested": 1,
"no": 1,
"felt": 1,
"I": 3,
"had": 2,
"become": 1,
"again": 1,
"my": 1,
"thoughts": 1,
"now": 1,
"Around": 1,
"than": 1,
"weary": 1,
"described": 1,
"circle": 1,
"what": 1,
"but": 1,
"occasioned": 1,
"position": 1,
"somewhat": 1,
"even": 1,
"unaccountably": 1,
"change": 1,
"aversion": 1,
"instance": 1,
"with": 1,
"any": 1,
"excited": 1,
"nay": 1
}
P.S. Стоит учесть, что слова вида
I'm
будут интерпретированы как 2 слова
["I", "m"]
.