Elasticsearch, docker, django-elastic_dsl, rest_framework로 n-gram 검색엔진 만들기 - 1차 트러블슈팅

Django로 API 서버를 구축하고, Elasticsearch와 Docker를 사용하여 n-gram 검색과 `analysis-nori` 형태소 분석기를 도입

 

1. Django 프로젝트 및 앱 생성

   django-admin startproject myproject
   cd myproject
   python3 manage.py startapp search

2. Django 앱 설정
   - `settings.py` 파일에서 앱과 REST framework를 설정

   # myproject/settings.py

   INSTALLED_APPS = [
       # ...
       'rest_framework',
       'search',
   ]

   REST_FRAMEWORK = {
       'DEFAULT_PAGINATION_CLASS': 'rest_framework.pagination.PageNumberPagination',
       'PAGE_SIZE': 10,
   }

3. Django 모델 생성
   - `models.py` 파일에 데이터 모델을 생성

   # search/models.py

   from django.db import models

   class Book(models.Model):
       title = models.CharField(max_length=255)

       def __str__(self):
           return self.title

4. Django 마이그레이션 및 데이터베이스 적용
   - 모델 변경을 적용하여 데이터베이스를 생성

python3 manage.py makemigrations
python3 manage.py migrate

5. Docker Compose 파일 작성
   - Elasticsearch를 Docker로 실행하기 위한 `docker-compose.yml` 파일을 작성

vim docker-compose.yml

 # docker-compose.yml

   version: '3'
   services:
     elasticsearch:
       image: docker.elastic.co/elasticsearch/elasticsearch:7.10.0
       environment:
         - discovery.type=single-node
       ports:
         - "9200:9200"

6. Docker Compose 실행
   - Docker Compose를 사용하여 Elasticsearch를 실행

docker-compose up

 

7. Elasticsearch Plugin 설치
   - Docker 컨테이너 내부에서 `analysis-nori` 플러그인을 설치

   ```bash
   docker exec -it [elasticsearch_container_id] /bin/bash

   # 내부에서 실행
   ./bin/elasticsearch-plugin install analysis-nori
   exit
   ```

container_id는 docker ps 명령어로 확인 가능

8. Django 앱에 n-gram 및 analysis-nori 설정 추가
   - `search` 앱 내에 `utils.py` 파일을 생성하고, Elasticsearch 색인 및 설정을 위한 코드를 추가

# search/utils.py

from elasticsearch import Elasticsearch
from elasticsearch.helpers import bulk
from .models import Book

es = Elasticsearch()

def index_books():
    bulk(client=es, actions=[
        {
            '_op_type': 'index',
            '_index': 'books',
            '_id': str(book.id),
            '_source': {'title': book.title}
        }
        for book in Book.objects.all()
    ])

index_books()

9. Django REST framework Serializer 생성
   - `serializers.py` 파일에 직렬화를 위한 Serializer를 생성

# search/serializers.py

from rest_framework import serializers
from .models import Book

class BookSerializer(serializers.ModelSerializer):
    class Meta:
        model = Book
        fields = ['id', 'title']

10. Django REST framework API 뷰 생성
    - `views.py` 파일에 REST framework를 사용하여 API 뷰를 생성

# search/views.py

from rest_framework import generics
from elasticsearch_dsl import Search
from .models import Book
from .serializers import BookSerializer

class BookSearchView(generics.ListAPIView):
    serializer_class = BookSerializer

    def get_queryset(self):
        q = self.request.query_params.get('q', '')
        s = Search(using='elasticsearch_dsl').query('match', title={'query': q, 'analyzer': 'ngram_analyzer'})
        response = s.execute()
        book_ids = [hit.id for hit in response]
        return Book.objects.filter(id__in=book_ids)

 

에러 : Import "elasticsearch_dsl" could not be resolved 

pip install elasticsearch-dsl

로 해결

11. Django REST framework API URL 설정
    - `urls.py` 파일에 API 뷰에 대한 URL을 설정

# search/urls.py

from django.urls import path
from .views import BookSearchView

urlpatterns = [
    path('search/', BookSearchView.as_view(), name='book-search'),
]

    - 메인 프로젝트의 `urls.py`에 앱의 URL을 포함시키기

# myproject/urls.py

from django.contrib import admin
from django.urls import include, path

urlpatterns = [
    path('admin/', admin.site.urls),
    path('api/', include('search.urls')),
]

 

 

12. 서버 실행

python3 manage.py runserver

 

에러 : http://127.0.0.1:8000/에 접속하면 Page not found (404) 오류, Using the URLconf defined in myproject.urls, Django tried these URL patterns, in this order:

 

방법1 - url 파일들 코드 수정

# myproject/urls.py

from django.contrib import admin
from django.urls import path, include

urlpatterns = [
    path('admin/', admin.site.urls),
    path('api/', include('search.urls')),  # 예시로 search 앱의 URL 패턴을 추가
]

# search/urls.py

from django.urls import path
from .views import BookSearchView

urlpatterns = [
    path('search/', BookSearchView.as_view(), name='book-search'),
]

 

settings 에 debug도 바꿔봄

# myproject/settings.py

DEBUG = False

>> CommandError: You must set settings.ALLOWED_HOSTS if DEBUG is False. 에러

로 또 생기길래 

# myproject/settings.py

ALLOWED_HOSTS = ['localhost', '127.0.0.1']

이걸로 수정해줌

 

다시 서버 실행해도 에러!

이번에는 페이지에 Not Found The requested resource was not found on this server. 에러

로 뜸

# search/views.py

from django.views.generic import TemplateView

class BookSearchView(TemplateView):
    template_name = 'book_search.html'

 이렇게 추가해줬다 

최종 views.py 코드는

# search/views.py
from django.shortcuts import render
from django.views import View
from rest_framework import generics
from elasticsearch_dsl import Search
from .models import Book
from .serializers import BookSerializer
from django.views.generic import TemplateView

class BookSearchView(View):
    template_name = 'book_search.html'

    def get(self, request, *args, **kwargs):
        query = request.GET.get('query', '')
        results = Book.objects.filter(title__icontains=query)
        return render(request, self.template_name, {'results': results})
    
class BookSearchAPIView(generics.ListAPIView):
    serializer_class = BookSerializer

    def get_queryset(self):
        q = self.request.query_params.get('q', '')
        s = Search(using='elasticsearch_dsl').query('match', title={'query': q, 'analyzer': 'ngram_analyzer'})
        response = s.execute()
        book_ids = [hit.id for hit in response]
        return Book.objects.filter(id__in=book_ids)

요런식이 되었음

그래도 안됨

 

13. 데이터베이스에 샘플 데이터 추가
    - 책 제목 샘플 데이터를 Django 데이터베이스에 추가

python manage.py shell

#!/usr/bin/env python
#myproject/manage.py

"""Django's command-line utility for administrative tasks."""
import os
import sys


def main():
    """Run administrative tasks."""
    os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'myproject.settings')
    try:
        from django.core.management import execute_from_command_line
    except ImportError as exc:
        raise ImportError(
            "Couldn't import Django. Are you sure it's installed and "
            "available on your PYTHONPATH environment variable? Did you "
            "forget to activate a virtual environment?"
        ) from exc
    execute_from_command_line(sys.argv)


if __name__ == '__main__':
    main()

from search.models import Book

books_data = ["aws 입문", "awthe 기초", "구의 증명", "쿠우쿠우 맛있게 먹기", "노리분석기", "데이터 구축하기", "데마분 정복하기", "라쇼몬", "뉴턴의 아틀리에", "싯다르타"]

for title in books_data:
    Book.objects.create(title=title)

 

 

 

 

 

elastic search 연결 다시 해보기

from elasticsearch import Elasticsearch

# Elasticsearch에 사용자 인증 정보를 제공합니다.
es = Elasticsearch(
    ['your_elasticsearch_host'],
    http_auth=('your_username', 'your_password'),
    port=9200,
)

# 예제 쿼리
response = es.search(index='your_index_name', body={
    "query": {
        "match_all": {}
    }
})

# 결과 출력
print(response)

 

 

추가 삽질기록 및 레퍼런스

---

설치 삽질

 

https://velog.io/@soyeon207/ES-2.-docker-%EC%97%90%EC%84%9C-ES-%EC%8B%9C%EC%9E%91%ED%95%98%EA%B8%B0

[ES] 2. docker 에서 ES 시작하기

yangdayeon@yangdayeon-ui-MacBookPro ~ % docker pull elasticsearch:7.9.3
7.9.3: Pulling from library/elasticsearch
75f829a71a1c: Pull complete 
2dd8aabff665: Pull complete 
fd17121b3976: Pull complete 
a19cf707b4fd: Pull complete 
4ccdd8a52dc0: Pull complete 
d018d3fc07a4: Pull complete 
70f1e3a1960a: Pull complete 
8f58f7e426fa: Pull complete 
817feb91b55c: Pull complete 
Digest: sha256:a13cd87cbf139fadbca64972ef2c8777222236887d303e4177c1ab7cff1b52f6
Status: Downloaded newer image for elasticsearch:7.9.3
docker.io/library/elasticsearch:7.9.3

yangdayeon@yangdayeon-ui-MacBookPro ~ % docker images
REPOSITORY                                                         TAG       IMAGE ID       CREATED        SIZE
925676828058.dkr.ecr.ap-northeast-2.amazonaws.com/asc-demo-repo2   latest    fa7f699ff65e   7 months ago   137MB
elasticsearch                                                      7.9.3     1ab13f928dc8   3 years ago    742MB

yangdayeon@yangdayeon-ui-MacBookPro ~ % docker run -d -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" --name elasticsearchTest elasticsearch:7.9.3

f32237ed21b54f18bbbecfcd2be6529561c8c8a7ddebe13662821e54eb5e7a90

yangdayeon@yangdayeon-ui-MacBookPro ~ % docker ps
CONTAINER ID   IMAGE                 COMMAND                   CREATED          STATUS          PORTS                                            NAMES
f32237ed21b5   elasticsearch:7.9.3   "/tini -- /usr/local…"   16 seconds ago   Up 15 seconds   0.0.0.0:9200->9200/tcp, 0.0.0.0:9300->9300/tcp   elasticsearchTest

yangdayeon@yangdayeon-ui-MacBookPro ~ % curl localhost:9200
{
  "name" : "f32237ed21b5",
  "cluster_name" : "docker-cluster",
  "cluster_uuid" : "-OdxG6xrTRaDF_dKee7iyw",
  "version" : {
    "number" : "7.9.3",
    "build_flavor" : "default",
    "build_type" : "docker",
    "build_hash" : "c4138e51121ef06a6404866cddc601906fe5c868",
    "build_date" : "2020-10-16T10:36:16.141335Z",
    "build_snapshot" : false,
    "lucene_version" : "8.6.2",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

f32237ed21b5

 

docker exec -it f32237ed21b5 /bin/bash

 

 

https://soyoung-new-challenge.tistory.com/77

[Elasticsearch] Plugin 설치 및 적용

bin/elasticsearch-plugin install file:///fast-elasticsearch-vector-scoring/target/releases/elasticsearch-binary-vector-scoring-7.5.1.zip

 

 

 

sudo /usr/share/elasticsearch/bin/elasticsearch-plugin install analysis-nori

 

 

---

python - django - es 삽질

 

https://soyoung-new-challenge.tistory.com/72

[Elasticsearch] python에서 엘라스틱 사용하기

https://python-jamo.readthedocs.io/en/latest/

A Guide to using Python-Jamo — jamo 0.4-beta documentation

---

연결 삽질

 

https://blog.yevgnenll.me/elk/install-plugin-to-elastic-search-and-check-install-or-not

Elastic Search 에 plugin install 하기, file plugin 설치 방법, plugin 설치 확인하기

잘 설치 되는데... 

문제가 우리팀 es 계정이 아니고 

애먼 걸 연결했다는 거임. 새로파서...? 

험...

 

 

https://happiestmemories.tistory.com/45

[AWS] Elasticsearch 구축하기

 

 

---

https://esbook.kimjmin.net/06-text-analysis/6.7-stemming/6.7.2-nori

6.7.2 노리 (nori) 한글 형태소 분석기 - Elastic 가이드북