Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

quize1小作業 #1

Open
wants to merge 38 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
38 commits
Select commit Hold shift + click to select a range
24c5a83
<new> Establishing the basic environment
Mar 18, 2023
039e1e0
<fead> DRF's Serializers & Model
Mar 18, 2023
b92df2b
<feat> DRF's Routers
Mar 18, 2023
2d8def3
<chore> Revise the Rate limiting package
Mar 19, 2023
bfa1797
<feat> The header field displays the rate limit judgment
Mar 19, 2023
b0efae7
<test> requirement test && Concurrently test
Mar 21, 2023
7f01ce7
<feat> add sqlite datebase
Mar 22, 2023
957a816
<refactor> view ratelimit_tracking info
Mar 22, 2023
2237f7a
<docs> create README.md && add image
Mar 22, 2023
34eee2b
Update README.md
dirtyrabbit Mar 22, 2023
c6caa76
Update README.md
dirtyrabbit Mar 22, 2023
d99b5d1
Add files via upload
dirtyrabbit Mar 22, 2023
edfb479
Add files via upload
dirtyrabbit Mar 22, 2023
ee4e702
update gitignore
dirtyrabbit Mar 24, 2023
f94670f
Delete webpage.png:Zone.Identifier
dirtyrabbit Mar 24, 2023
f7d9866
update gitgnore
dirtyrabbit Mar 24, 2023
b6694a6
<remove> all file from repo
dirtyrabbit Mar 24, 2023
57ebc56
commit file with gitignore
dirtyrabbit Mar 24, 2023
f744ef6
remove all file
dirtyrabbit Mar 24, 2023
a16c0fa
<fix> commit with gitignore
dirtyrabbit Mar 24, 2023
4bf12a1
<remove> file under data/
dirtyrabbit Mar 24, 2023
8671382
<fix> the count of A mistake
dirtyrabbit Mar 26, 2023
87cb136
<feat> rate limit reset api
dirtyrabbit Mar 26, 2023
8e4f150
<test> add execution time
dirtyrabbit Mar 27, 2023
f237d4f
<feat> add response function and remove tracking function
dirtyrabbit Mar 27, 2023
2b36e0e
Update README.md
dirtyrabbit Mar 27, 2023
fdc7270
<refactor> adjust test about request time of GET
dirtyrabbit Mar 27, 2023
f3bb7c6
<feat> test time greate 1s loop
dirtyrabbit Mar 28, 2023
1703ed7
<test> verify the response header
dirtyrabbit Mar 28, 2023
5f8e886
<fix> git push ignore file
dirtyrabbit Mar 28, 2023
ba9c220
<fix> migrations error
dirtyrabbit Mar 28, 2023
9bba224
<docs> restore files
dirtyrabbit Apr 13, 2023
5e24494
<refactor> change dockerfile copy to add
dirtyrabbit Apr 13, 2023
71b31f6
<refactor> modify tests print
dirtyrabbit Apr 13, 2023
c3fb900
<chore> python 3 to 3.9
dirtyrabbit Apr 13, 2023
39bcb11
<refactor> api reset
dirtyrabbit Apr 13, 2023
c8208e3
<feat> add path field to db
dirtyrabbit Apr 13, 2023
2c4e084
<refactor> change dockerfile ADD to COPY
dirtyrabbit Apr 13, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
190 changes: 190 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,190 @@
# Created by https://www.toptal.com/developers/gitignore/api/linux,django
# Edit at https://www.toptal.com/developers/gitignore?templates=linux,django

data/
### Django ###
*.log
*.pot
*.pyc
__pycache__/
local_settings.py
db.sqlite3
db.sqlite3-journal
media

# If your build process includes running collectstatic, then you probably don't need or want to include staticfiles/
# in your Git repository. Update and uncomment the following line accordingly.
# <django-project-name>/staticfiles/

### Django.Python Stack ###
# Byte-compiled / optimized / DLL files
*.py[cod]
*$py.class

# C extensions
*.so

# Distribution / packaging
.Python
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
share/python-wheels/
*.egg-info/
.installed.cfg
*.egg
MANIFEST

# PyInstaller
# Usually these files are written by a python script from a template
# before PyInstaller builds the exe, so as to inject date/other infos into it.
*.manifest
*.spec

# Installer logs
pip-log.txt
pip-delete-this-directory.txt

# Unit test / coverage reports
htmlcov/
.tox/
.nox/
.coverage
.coverage.*
.cache
nosetests.xml
coverage.xml
*.cover
*.py,cover
.hypothesis/
.pytest_cache/
cover/

# Translations
*.mo

# Django stuff:

# Flask stuff:
instance/
.webassets-cache

# Scrapy stuff:
.scrapy

# Sphinx documentation
docs/_build/

# PyBuilder
.pybuilder/
target/

# Jupyter Notebook
.ipynb_checkpoints

# IPython
profile_default/
ipython_config.py

# pyenv
# For a library or package, you might want to ignore these files since the code is
# intended to run in multiple environments; otherwise, check them in:
# .python-version

# pipenv
# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
# However, in case of collaboration, if having platform-specific dependencies or dependencies
# having no cross-platform support, pipenv may install dependencies that don't work, or not
# install all needed dependencies.
#Pipfile.lock

# poetry
# Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
# This is especially recommended for binary packages to ensure reproducibility, and is more
# commonly ignored for libraries.
# https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
#poetry.lock

# pdm
# Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
#pdm.lock
# pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it
# in version control.
# https://pdm.fming.dev/#use-with-ide
.pdm.toml

# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
__pypackages__/

# Celery stuff
celerybeat-schedule
celerybeat.pid

# SageMath parsed files
*.sage.py

# Environments
.env
.venv
env/
venv/
ENV/
env.bak/
venv.bak/

# Spyder project settings
.spyderproject
.spyproject

# Rope project settings
.ropeproject

# mkdocs documentation
/site

# mypy
.mypy_cache/
.dmypy.json
dmypy.json

# Pyre type checker
.pyre/

# pytype static type analyzer
.pytype/

# Cython debug symbols
cython_debug/

# PyCharm
# JetBrains specific template is maintained in a separate JetBrains.gitignore that can
# be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
# and can be added to the global gitignore or merged into this file. For a more nuclear
# option (not recommended) you can uncomment the following to ignore the entire idea folder.
#.idea/

### Linux ###
*~

# temporary files which can be created if a process still has a handle open of a deleted file
.fuse_hidden*

# KDE directory preferences
.directory

# Linux trash folder which might appear on any partition or disk
.Trash-*

# .nfs files are created when an open file is removed but is still being accessed
.nfs*

# End of https://www.toptal.com/developers/gitignore/api/linux,django
9 changes: 9 additions & 0 deletions quiz/01-rate-limit/quiz01/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
# syntax=docker/dockerfile:1
FROM python:3.9
ENV PYTHONDONTWRITEBYTECODE=1
ENV PYTHONUNBUFFERED=1
WORKDIR /code
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what we would do is

ADD requirements.txt /code/
RUN pip install -r requirements.txt

ADD . /code/

making requirements.txt to be the top layer of the container image would leverage image cache when you are re-built it.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the previous version, it was stated that according to the Dockerfile best practices guide, we should always prefer COPY over ADD unless we specifically need one of the two additional features of ADD. However, based on our requirements, we have changed it to ADD.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that's why we need a code review, seniority can NOT surpass the best practice.
Thanks for pointing that out. And yes, "Add" brings so many features and could be damaged as the behavior are more implicit rather than explicit. we should use 'COPY' instead.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, and thank you for bringing this up.

COPY requirements.txt /code/
COPY locustfile.py /code/
RUN pip install -r requirements.txt
COPY . /code/
71 changes: 71 additions & 0 deletions quiz/01-rate-limit/quiz01/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
# 01-rate-limit
>Rate limiting is used to protect resources from being over-using or abused by users/bots/applications. It is commonly implemented by social media platforms such as Facebook or Instagram.
## The project components
用Django做應用框架搭配 [django-ratelimit](https://django-ratelimit.readthedocs.io/en/stable/)做速率限制,在資料庫上用Sqlite做每個使用者的header file的紀錄。


## Usage
1. 建立Image,tag加入"v3.3.1",或者自行至`docker-compose.yml`修改web的image的賦予名稱

```
docker build -t="v3.3.1" .
```
2. 載入[locustio/locust](https://hub.docker.com/r/locustio/locust),Load testing 的工具
```
docker pull locustio/locust
```

3. Migration(資料遷移)
```
docker-compose run web python3 manage.py makemigrations
```
```
docker-compose run web python3 manage.py migrate
```

4. 啟動container
```
docker-compose up --scale worker=4
```
5. 開啟網頁,導向 http://localhost:8000/api/

![image](/quiz/01-rate-limit/quiz01/image/webpage.png)


## Test
使用Postman和撰寫Django Test做基本ratelimit測試,也加入[Locust](https://locust.io/)做Load testing 和 Profiling 評估。
### Postman
Postman 是常用的api測試工具,我們可以透過Postman簡單的進行Request,也可以透過該工具清楚的看到response的header field。

**Get**
![image](/quiz/01-rate-limit/quiz01//image/postman_get.png "This is a sample image.")

**Post**
![image](/quiz/01-rate-limit/quiz01//image/postman_post.png "This is a sample image.")

**429 Too Many Requests**
![image](/quiz/01-rate-limit/quiz01//image/postman_429.png "This is a sample image.")

### Django Test
透過Django Test 可以自訂義test的方法,查看response header內容,並顯示執行時間,如果超過呼叫次數顯示"Over Rating"。
```
docker-compose run web python3 manage.py test
```
* 自訂義Django Test 4種測試:
1. 在1秒內Request Get 100個請求(Get Not Over)
2. 在1秒內Request Post 1個請求(Post Not Over)
3. 在1秒內Request Get 101個請求(Get Over)
4. 在1秒內Request Post 2個請求(Post Over)

![image](/quiz/01-rate-limit/quiz01//image/django_test.png "This is a sample image.")

### Locust
> [Locust](https://locust.io/)
Define user behaviour with Python code, and swarm your system with millions of simultaneous users.
可以透過Locust簡易的Loading test設定,並觀察測試數據。

![image](/quiz/01-rate-limit/quiz01//image/locust_index.png "This is a sample image.")
![image](/quiz/01-rate-limit/quiz01//image/locust_statistics.png "This is a sample image.")
![image](/quiz/01-rate-limit/quiz01//image/locust_failures.png "This is a sample image.")


Empty file.
3 changes: 3 additions & 0 deletions quiz/01-rate-limit/quiz01/core/admin.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
from django.contrib import admin

# Register your models here.
6 changes: 6 additions & 0 deletions quiz/01-rate-limit/quiz01/core/apps.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
from django.apps import AppConfig


class CoreConfig(AppConfig):
default_auto_field = 'django.db.models.BigAutoField'
name = 'core'
12 changes: 12 additions & 0 deletions quiz/01-rate-limit/quiz01/core/migrations/0001_initial.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
# Generated by Django 3.2.18 on 2023-03-21 18:23

from django.db import migrations


class Migration(migrations.Migration):

dependencies = [
]

operations = [
]
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
# Generated by Django 3.2.18 on 2023-03-21 18:28

from django.db import migrations, models
import django.db.models.deletion


class Migration(migrations.Migration):

initial = True

dependencies = [
('core', '0001_initial'),
]

operations = [
migrations.CreateModel(
name='Customer_ID_Model',
fields=[
('customer_ID', models.TextField(primary_key=True, serialize=False)),
],
options={
'db_table': 'Customer_ID_Model',
},
),
migrations.CreateModel(
name='HeaderField_Model',
fields=[
('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
('request_type', models.TextField()),
('Limit', models.IntegerField(default=100)),
('Remaining', models.IntegerField(default=100)),
('Reset', models.IntegerField(default=1)),
('RetryAt', models.IntegerField(default=0)),
('upData', models.DateTimeField(auto_now=True)),
('ID', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to='core.customer_id_model')),
],
options={
'db_table': 'HeaderField_Model',
},
),
]
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
# Generated by Django 3.2.18 on 2023-03-22 04:44

from django.db import migrations, models


class Migration(migrations.Migration):

dependencies = [
('core', '0002_customer_id_model_headerfield_model'),
]

operations = [
migrations.CreateModel(
name='GET_Model',
fields=[
('customer_ID', models.TextField(primary_key=True, serialize=False)),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add a path field, this would help you record what paths are queried.
and the rate-limited module can be used to guard method + path

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, it has been added.

path = models.TextField(default = 'n/a')

('Limit', models.IntegerField(default=100)),
('Remaining', models.IntegerField(default=100)),
('Reset', models.IntegerField(default=1)),
('RetryAt', models.IntegerField(default=0)),
('upData', models.DateTimeField(auto_now=True)),
],
options={
'db_table': 'GET_Model',
},
),
migrations.CreateModel(
name='POST_Model',
fields=[
('customer_ID', models.TextField(primary_key=True, serialize=False)),
('Limit', models.IntegerField(default=1)),
('Remaining', models.IntegerField(default=1)),
('Reset', models.IntegerField(default=1)),
('RetryAt', models.IntegerField(default=0)),
('upData', models.DateTimeField(auto_now=True)),
],
options={
'db_table': 'POST_Model',
},
),
migrations.RemoveField(
model_name='headerfield_model',
name='ID',
),
migrations.DeleteModel(
name='Customer_ID_Model',
),
migrations.DeleteModel(
name='HeaderField_Model',
),
]
Loading