Based on the following stop words I tokenized obama’s most recent state of the union. Below is a portion of the results.<\/p>\n
Stop words:<\/p>\n
(\"a\", \"an\", \"the\",\"and\",\".\",\",\",\" \",\"because\",\r\n\r\n\"why\",\"this\",\"is\",\"of\",\"in\",\"if\",\"that\",\"that's\",\"it\",\"then\",\"than\",\"when\",\r\n\r\n\"we\",\"as\",\"from\",\"to\",\"our\",\"s\",\"have\",\"they\",\"have\",\"?\",\r\n\r\n\"all\",\"must\",\"who\",\"you\",\"on\",\"for\",\"may\",\"be\",\"\/\",\"\\\"\\'\",\"\\\"\",\"get\",\"are\",\"i\",\"am\",\"not\",\r\n\r\n\"m\",\"make\",\"makes\",\"for\",\"into\",\"but\",\"can\",\"only\",\"happen\",\"don\",\"same\",\"against\",\"nearly\",\r\n\r\n\"entire\",\"sure\", \"u\", \"!\", \"was\", \"has\", \"its\", \"through\", \"me\", \"his\",\"once\",\"carry\",\r\n\r\n\"anew\",\"'\", \"t\",\"let\", \"us\", \"new\", \"before\", \"come\",\"two\", \"one\", \"ve\", \"go\", \"8\",\r\n\r\n\"she\", \"her\", \"he\",\"none\",\"at\",\"been\",\"these\",\"what\",\"up\",\"were\",\"them\",\"some\",\"had\",\r\n\r\n\"their\",\"do\",\"by\",\"or\",\"re\",\"aren\",\"so\",\"with\",\"will\",\"my\",\"no\",\"there\",\"here\",\"went\",\r\n\r\n\"much\",\"out\",\"other\",\"each\");<\/pre>\nResults: (portion)<\/pre>\n\r\n41835 41841\u00a0 |people|\r\n41845 41850\u00a0 |haiti|\r\n41858 41863\u00a0 |lives|\r\n41878 41887\u00a0 |americans|\r\n41896 41903\u00a0 |dropped|\r\n41904 41914\u00a0 |everything|\r\n41921 41930\u00a0 |someplace|\r\n41940 41945\u00a0 |never|\r\n41955 41959\u00a0 |pull|\r\n41960 41966\u00a0 |people|\r\n41976 41981\u00a0 |never|\r\n41982 41987\u00a0 |known|\r\n41997 42003\u00a0 |rubble|\r\n42005 42014\u00a0 |prompting|\r\n42015 42021\u00a0 |chants|\r\n42057 42064\u00a0 |another|\r\n42065 42069\u00a0 |life|\r\n42074 42079\u00a0 |saved|\r\n42084 42090\u00a0 |spirit|\r\n42100 42109\u00a0 |sustained|\r\n42115 42121\u00a0 |nation|\r\n42126 42130\u00a0 |more|\r\n42140 42149\u00a0 |centuries|\r\n42150 42155\u00a0 |lives|\r\n42171 42177\u00a0 |people|\r\n42187 42195\u00a0 |finished|\r\n42198 42207\u00a0 |difficult|\r\n42208 42212\u00a0 |year|\r\n42237 42246\u00a0 |difficult|<\/pre>\n\r\n42247 42253\u00a0 |decade|\r\n42265 42269\u00a0 |year|\r\n42286 42292\u00a0 |decade|\r\n42293 42302\u00a0 |stretches|\r\n42324 42328\u00a0 |quit|\r\n42339 42343\u00a0 |quit|\r\n42357 42362\u00a0 |seize|\r\n42368 42374\u00a0 |moment|\r\n42380 42385\u00a0 |start|\r\n42405 42410\u00a0 |dream|\r\n42411 42418\u00a0 |forward|\r\n42427 42437\u00a0 |strengthen|\r\n42442 42447\u00a0 |union|\r\n42453 42457\u00a0 |more|<\/pre>\n42458 42463\u00a0 |thank|\r\n42469 42472\u00a0 |god|\r\n42473 42478\u00a0 |bless|\r\n42488 42491\u00a0 |god|\r\n42492 42497\u00a0 |bless|\r\n\r\n42502 42508\u00a0 |united|\r\n42509 42515\u00a0 |states|\r\n42519 42526\u00a0 |america|\r\n<\/pre>\nI added stop words based on the unnecessary words that were appearing in the previous parsing. Doing this reinforced the idea that picking stop tokenizers is largely based on biases. To me it was clear which words were not meaningful, so I kept taking them out. At some point though it became more complicated. For example the use of words like “own”, which have different meanings, verb and noun, could be interesting, or things like “come”, but I decided they were not important.<\/p>\n
It seems to me that google, yahoo, and Bing all have very similar stop tokenizers. One article said that Yahoo! search is now powered by Bing in the US and google in Japan. (http:\/\/www.seobook.com\/relevancy\/)<\/p>\n
This article states:<\/p>\n
“You can differentiate by having product information. But Google scrapes it. You can differentiate through consumer & editorial reviews. But Google scrapes it. You can differentiate by brand, but Google sells branded keywords to competitors. No matter what you do, Google competes against you. You can opt out of being scraped, but then you get no search traffic (& the ecosystem is set up to pay someone else to scrape your content + wrap it in ads).” <\/span><\/p>\n
\n<\/span><\/span><\/p>\nFor google, there are so many other more complicated and more important algorithms for text parsing, and the ad \/proprietary \/ information ownership motives in addition to the need to define the user’s intentions are. <\/span><\/span><\/p>\n
When stop tokenization is used alone, the search has no way of ordering the results in a meaningful way. Google looks for things like “natural citations” to further break down results, presenting the ones that include the stop words as well as more relevant. It also has really interesting web based approaches like looking for natural link growth, depth and quality of links to and from a page, and a page’s age.<\/p>\n
Yahoo apparently, from what I understand, offers more search priority through ad sales than google, and has a poorer distinction of link quality and depth, therefore constantly presenting us with tangentially connected information, which I have to say has its place. I love when I go to check my yahoo spam account, that I keep mostly for the purpose of distraction, and I come across a link to a cat doing something cute, which brings me to some advertisement, and then another exciting link, and helps me pass some time while I get ready to get back to work. It has its place as a totally different kind of search, and as some search engines work to meet people’s needs, others are working to reduce people’s attention spans to meet their shifty search strategies and criteria.<\/p>\n
The most important thing is the slow convergence \/ homogenization of search strategies, following google mostly, limiting web recommendations and search result more and more, as often this is based on “link growth”, so “viral” things are almost self fulfilling prophecies.<\/p>\n
As search engines become personalized, they get to limit the recommendations for each person even more. On our end, we seem to develop a relationship with our preferred search engines as well, learning their stop tokenizers, and attempting to match their parsing style to fit our needs. We begin to develop a relationship, a two way relationship, with our search engine that becomes more and more specific, isolated, and limited, and we begin to learn from the search engine’s biases, further limiting our use of them as searching for something outside of their biases. Maybe the key is for different search engines to continue to evolve in different directions, so the recommendations and search results on each would not reply to our behaviors in the same way.<\/p>\n","protected":false},"excerpt":{"rendered":"
Based on the following stop words I tokenized obama’s most recent state of the union. Below is a portion of the results. Stop words: (“a”, “an”, “the”,”and”,”.”,”,”,” “,”because”, “why”,”this”,”is”,”of”,”in”,”if”,”that”,”that’s”,”it”,”then”,”than”,”when”, “we”,”as”,”from”,”to”,”our”,”s”,”have”,”they”,”have”,”?”, “all”,”must”,”who”,”you”,”on”,”for”,”may”,”be”,”\/”,”\\”\\'”,”\\””,”get”,”are”,”i”,”am”,”not”, “m”,”make”,”makes”,”for”,”into”,”but”,”can”,”only”,”happen”,”don”,”same”,”against”,”nearly”, “entire”,”sure”, “u”, “!”, “was”, “has”, “its”, “through”, “me”, “his”,”once”,”carry”, “anew”,”‘”, “t”,”let”, “us”, “new”, “before”, “come”,”two”, “one”, “ve”, “go”, “8”, “she”, “her”, “he”,”none”,”at”,”been”,”these”,”what”,”up”,”were”,”them”,”some”,”had”, “their”,”do”,”by”,”or”,”re”,”aren”,”so”,”with”,”will”,”my”,”no”,”there”,”here”,”went”, […]<\/p>\n","protected":false},"author":3,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[25,26,1],"tags":[],"class_list":["post-672","post","type-post","status-publish","format-standard","hentry","category-learning-bit-x-bit","category-notes","category-uncategorized"],"_links":{"self":[{"href":"http:\/\/www.mariarabinovich.com\/blog\/wp-json\/wp\/v2\/posts\/672","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/www.mariarabinovich.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/www.mariarabinovich.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/www.mariarabinovich.com\/blog\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"http:\/\/www.mariarabinovich.com\/blog\/wp-json\/wp\/v2\/comments?post=672"}],"version-history":[{"count":7,"href":"http:\/\/www.mariarabinovich.com\/blog\/wp-json\/wp\/v2\/posts\/672\/revisions"}],"predecessor-version":[{"id":675,"href":"http:\/\/www.mariarabinovich.com\/blog\/wp-json\/wp\/v2\/posts\/672\/revisions\/675"}],"wp:attachment":[{"href":"http:\/\/www.mariarabinovich.com\/blog\/wp-json\/wp\/v2\/media?parent=672"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/www.mariarabinovich.com\/blog\/wp-json\/wp\/v2\/categories?post=672"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/www.mariarabinovich.com\/blog\/wp-json\/wp\/v2\/tags?post=672"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}