Hızlı yanıt: kod örneği
Your file almost certainly has actual U+00AD soft-hyphen characters in it.These are characters that mark places where a word could be split when fitting lines to a page. The idea is that the soft hyphen is invisible if the word doesn't need to be split, but printed the same as a U+2010 normal hyphen if it does.Since you don't care about rendering this text in a book with nicely flowing text, you're never going to hyphenate anything, so you just want to remove these characters.The way to do this is not to fiddle with the encoding. Just remove them from the Unicode text, using whichever of these you find most readable:Notice the single backslash. We're not replacing a literal backslash, , , , we're replacing a literal soft-hyphen character, that is, the character whose code point is hex 0xad.You can either do this to the whole file before splitting into words, or do it once per word after splitting.Meanwhile, you seem to be confused about what encodings are and what to do with them:I've tried encoding the .txt file as iso8859-15No, you've tried decoding the file as ISO-8859-15. It's not clear why you tried ISO-8859-15 in the first place. But, since the ISO-8859-15 encoding for the character is the byte , maybe that's correct.Ideally I'd like to encode this data as UTF-8 to avail of the nltk libraryBut NLTK doesn't want UTF-8 bytes, it wants Unicode strings. You don't need to encode it for that.Plus, you're not trying to encode your Unicode text to UTF-8, you're trying to decode your bytes from UTF-8. If that's not what those bytes are… if you're lucky, you'll get an error like this one; if not, you'll get mojibake that you don't notice until you've screwed up a 500GB corpus and thrown away the original data.11. UTF-8 is specifically designed so you'll get early errors whenever possible. In this case, reading ISO-8859-15 text with soft hyphens as if it were UTF-8 raises exactly the error you're seeing, but reading UTF-8 text with soft hyphens as if it were ISO-8859-15 will silently succeed, but with an extra character before each soft hyphen. The error is usually more helpful.
data = data.replace('\xad', '')data = data.replace('\u00ad', '')data = data.replace('\N{SOFT HYPHEN}', '')xad'\xad'b'\xad''Â'