How to Insert unique column data into elasticsearch? -

i have webcrawler scrapes business names , business contact numbers , inserts elasticsearch

the problem dot not want duplicate phone numbers inserted column

is there anyway can achieve internally in elasticsearch?

if not, there better solution below steps

fetch record -> check duplicate phone number -> insert

you need unique token filter this: https://www.elastic.co/guide/en/elasticsearch/reference/1.6/analysis-unique-tokenfilter.html

if phone numbers put in same field of 1 document, create tokenizer should output tokens each phone number.

and tokenizer define filter of type unique keep unique tokens. around these lines:

  "analyzer": {     "numbers": {       "tokenizer": "[whatever]",       "filter": [         ...,         "unique"       ]     }   }