性能测试: Python3 利用asynico协程系统构建生产消费模型

今天研究了下python3的新特性 asynico ，试了试 aiohttp 协程效果，单核QPS在500~600之间，性能还可以。

ab测试工具

ab测试工具全名叫Apache Bench，主要用来做网站压力和性能测试。

ab命令：

$ ab -h
Usage: ab [options] [http[s]://]hostname[:port]/path
Options are:
    -n requests     Number of requests to perform
    -c concurrency  Number of multiple requests to make at a time
    -t timelimit    Seconds to max. to spend on benchmarking
                    This implies -n 50000
    -s timeout      Seconds to max. wait for each response
                    Default is 30 seconds
    -b windowsize   Size of TCP send/receive buffer, in bytes
    -B address      Address to bind to when making outgoing connections
    -p postfile     File containing data to POST. Remember also to set -T
    -u putfile      File containing data to PUT. Remember also to set -T
    -T content-type Content-type header to use for POST/PUT data, eg.
                    'application/x-www-form-urlencoded'
                    Default is 'text/plain'
    -v verbosity    How much troubleshooting info to print
    -w              Print out results in HTML tables
    -i              Use HEAD instead of GET
    -x attributes   String to insert as table attributes
    -y attributes   String to insert as tr attributes
    -z attributes   String to insert as td or th attributes
    -C attribute    Add cookie, eg. 'Apache=1234'. (repeatable)
    -H attribute    Add Arbitrary header line, eg. 'Accept-Encoding: gzip'
                    Inserted after all normal header lines. (repeatable)
    -A attribute    Add Basic WWW Authentication, the attributes
                    are a colon separated username and password.
    -P attribute    Add Basic Proxy Authentication, the attributes
                    are a colon separated username and password.
    -X proxy:port   Proxyserver and port number to use
    -V              Print version number and exit
    -k              Use HTTP KeepAlive feature
    -d              Do not show percentiles served table.
    -S              Do not show confidence estimators and warnings.
    -q              Do not show progress when doing more than 150 requests
    -g filename     Output collected data to gnuplot format file.
    -e filename     Output CSV file with percentages served
    -r              Don't exit on socket receive errors.
    -h              Display usage information (this message)
    -Z ciphersuite  Specify SSL/TLS cipher suite (See openssl ciphers)
    -f protocol     Specify SSL/TLS protocol
                    (SSL3, TLS1, TLS1.1, TLS1.2 or ALL)

比如开启一个10并发的1000次请求：

ab -c 10 -n 1000 http://www.tensorbytes.com/

添加头信息：

ab -c 1 -n 1 -w \
  -H 'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.105 Safari/537.36' \
  -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9' \
  -H 'Accept-Language: zh-CN,zh;q=0.9,en;q=0.8' \
 http://www.tensorbytes.com/

案例代码

代码：

import aiohttp
import asyncio
import hashlib
import time
from asyncio import Queue


class Fetch:
    def __init__(self):
        self.work_queue = Queue()
        self.max_loop = 10000
        self.host = "http://14.29.5.29/XXXX"
        self.payload = {"planId": 10000007, "activityId": 1002, "label": 1,
                        "key": "98214ecfe6b9ae8855e3ac6509ad940f", "keyType": "imei",
                        "batchId": 1, "token": "395La7f9x9x"}

    async def get_url(self, host, payload):
        async with aiohttp.ClientSession() as session:
            async with session.post(host, data=payload) as resp:
                text = await resp.text()
                if "1" in text:
                    print(text, payload["key"])

    async def consumer(self):
        while True:
            param = await self.work_queue.get()
            if param:
                await self.get_url(self.host, param)
                self.work_queue.task_done()
            else:
                break

    async def producer(self):
        i = 0
        string = '866260035710238'
        while 1:
            if i:
                md5_str = hashlib.md5(string.encode('utf-8'))
                self.payload["key"] = md5_str.hexdigest()
                string = str(int(string) + 1)
            await self.work_queue.put(self.payload.copy()) #必须要
            i += 1
            if i > self.max_loop:
                break

    async def run(self):
        await self.producer()
        print('start consumer...')

        tasks = [
            loop.create_task(self.consumer())
            for i in range(10)
        ]

        await self.work_queue.join()
        print('end join')
        for task in tasks:
            task.cancel()

t1 = time.time()
loop = asyncio.get_event_loop()
test = Fetch()
loop.run_until_complete(test.run())
loop.close()
print(time.time() - t1)