-
- Token is a technical term given to a sequence of characters. Note that “:)” is also a token.
- Word type is a unique item of the vocabulary.
- For ex., in a phrase of “this is a sample text for this sample test”, word types are [“this”, “is”, “a”, “sample”, “text”, “for”, “test”]