2016-06-30

Tableを使わずuiとli要素で表を記述する

リスト要素でテーブルに代えること

テーブルは推奨されない。リスト要素を使ってテーブルを表現してみる。 di (description item) を除く ui (unorderd list), oi (orderd list), li (list item) を使う。
どうするべきか。
行はリストのタグで代用する。tdタグではさむかわりに、spanクラスではさむ。列がずれないよう、CSSでpaddingを指定する。

ここでは触れなかったが、divを使う方法もある。

Bibliography


Generator is pretty faster than list

 Generator iterates just once. List stores values. You choose one by your own case.

>>> timeit.timeit('(i for i in range(1000))',number=10000)
0.009893075991520561
>>> timeit.timeit('[i for i in range(1000)]',number=10000)
0.4327252675073794
>>> timeit.timeit('(i for i in range(10000))',number=10000)
0.01080568787727998
>>> timeit.timeit('[i for i in range(10000)]',number=10000)
4.4710091782229995
>>> timeit.timeit('(i for i in range(100000))',number=10000)
0.008112851248995412
>>> timeit.timeit('[i for i in range(100000)]',number=10000)
50.3096059506083



python - Generator Expressions vs. List Comprehension - Stack Overflow

6/30 株取引 +900

  • シグナルおよび、ラブライブのアニメ放送開始と、今期最大売上高に注目した。少し儲けた。
  • 売買のスタイルがデイトレーダーのようになってきた。どうすれば張り付かずに済むか。利益確定と損切りを効率的に行いたい。
  • 購買余力が足りなくなった。注文の仕方にも注意していく。

3675 クロスマーケティング 100株@386 買付
3656 KLab 100株@650 買付
3675 クロスマーケティング 100株@384 買付
3675 クロスマーケティング 100株@385 売付
3656 KLab 100株@671 買付
3656 KLab 200株@668 売付

実現損益
3656 KLab +1,200円
3675 クロスマーケティング -300円

Total  +900円

2016-06-29

Flask リクエスト後の処理 (after_request)

サーバーからのレスポンスがスーパースローだった。処理の途中でいったんレスポンスを返すことを考えた。どうすればいいのか。以下に説明した。

まず、
from flask import g

def after_this_request(func):
    if not hasattr(g, 'call_after_request'):
        g.call_after_request = []
    g.call_after_request.append(func)
    return func


@app.after_request
def per_request_callbacks(response):
    for func in getattr(g, 'call_after_request', ()):
        response = func(response)
    return response
をコピペしソースコードにベタ書きしておき、次のようにレスポンスを受け取って返せばよいらしい。
@app.route('/')
def index():
    @after_this_request
    def do_this_after_request(response):
        # Do something:
        return response

具体例は
def invalidate_username_cache():
    @after_this_request
    def delete_username_cookie(response):
        response.delete_cookie('username')
        return response
のようにして使う。

Bibliography




6/29 株取引 +10,450

6074, ジェイエスエス, 100株@870円で買付
6314, 石井工作研究所, 100株@393円で売付 (+8,800円)
6074, ジェイエスエス, 100株@888円で100株売付 (+1,650円)

損益 +10,450円

保有銘柄: なし

2016-06-28

6/28 株取引 +750

春からちょっとずつやっている株取引ですが、最近になって勝てるようになってきた、戦績を記録します。
運用資金は10万円です。

6/27
6314, 石井工作研究所, 305円, 100株,買付

6/28
9980, マルコ, 181円, 100株, 買付
9980, マルコ, 190円, 100株, 売付

6314, 石井工作研究所, 340円, 100株, 保有

損益
9980, マルコ, +750円

Total +750円

2016-06-27

generator in flask

Found a way to yield values in flask. I wanted to send a large data, so this is useful. Here is a basic example.


from flask import Response

@app.route('/large.csv')
@app.route('/test')
def test():
    def generate():
        app.logger.info('request started')
        for i in range(5):
            time.sleep(1)
            yield str(i)
        app.logger.info('request finished')
        yield ''
    return Response(generate(), mimetype='text/plain')

To understand generator, yield and streaming, go over following materials

Bigliography

2016-06-23

Install Python34 and libraries on Amazon EC2

 Access to EC2

Default username is ec2-user. Check your public-dns of your ec2 and login by
ssh -i path/to/your/key.pem ec2-user@public-dns

If you are in Cygwin terminal and Windows, you have to do a little more at first because of Cygwin7s bug. Try
chgrp Users key.pem
chmod 400 key.pem

Install Python3.4

You have python2 by default, but no python3. You will find Python34, which is easily installed
sudo yum list | grep python3
install it
sudo yum install python34

Virtual Environment

I created myenv in /usr/bin
virtualenv -p python3.4 /usr/bin/myenv
check if it works by
[ec2-usr@ip...]$ source /usr/bin/myenv/bin/activate
then your terminal has (myenv) in the beggining of current path. See which versions your python command runs 
[ec2-usr@ip...]$ python
>>> import sys; sys.version 
, which should run with python 34. Now exit python by
>>> exit()
You can come out of virtual environment by
(myenv)[ec2-usr@ip...]$ deactivate
, but for now keep staying in the environment for further steps.

Install Development Tools

(myenv)[ec2-usr@ip...]$ sudo yum groupinstall "Development Tools"
(myenv)[ec2-usr@ip...]$ sudo yum -y install gcc-c++ python27-devel atlas-sse3-devel lapack-devel
(myenv)[ec2-usr@ip...]$ sudo yum install python3.4-devel
To install lxml
(myenv)[ec2-usr@ip...]$ sudo yum install libxml2-devel libxslt-devel python-devel
Some of them may not needed to install python libraries. I'm not sure.

Install python libraries

upgrade pip
(myenv)[ec2-usr@ip...]$ sudo /usr/bin/myenv/bin/pip3 install --upgrade pip
. Pip and pip3 links the same file (You can check by "which pip" and "which pip3"), but they seem not the same sometimes confusingly.

Install numpy

Even in virtual environment, "sudo pip install numpy" installs not in virtual. I do not know why. Fortunately full path works. Also /path/to/pip3 seems better than /path/to/pip.
(myenv)[ec2-usr@ip-...]$ sudo /usr/bin/myenv/bin/pip3 install numpy
. Take a cup of coffee. I do not know why it requires sudo even it installs in the virtual environment. Then see if you can import numpy in the virtual environment
(myenv)[ec2-usr@ip...]$ python
>>>import numpy
>>>numpy.version

Install scipy
(myenv)[ec2-usr@ip-...]$ sudo /usr/bin/myenv/bin/pip3 install scipy

Install matplotlib
(myenv)[ec2-usr@ip-...]$ sudo yum install libpng-devel
(myenv)[ec2-usr@ip-...]$ sudo /usr/bin/myenv/bin/pip3 install matplotlib

Install lxml

(myenv)[ec2-usr@ip-...]$ sudo yum install libxml2-devel libxslt-devel python34-devel
(myenv)[ec2-usr@ip-...]$ sudo /usr/bin/myenv/bin/pip3 install lxml

You may get  an error while installing lxml:
 {standard input}: Assembler messages:
    {standard input}:519888: Error: no such instruction: `c'
    {standard input}: Error: open CFI at the end of file; missing .cfi_endproc directive
    gcc: internal compiler error: Killed (program cc1)
    Please submit a full bug report,
    with preprocessed source if appropriate.
    See <http://bugzilla.redhat.com/bugzilla> for instructions.
    Compile failed: command 'gcc' failed with exit status 4
    creating tmp
    cc -I/usr/include/libxml2 -I/usr/include/libxml2 -c /tmp/xmlXPathInitVml43f.c -o tmp/xmlXPathInitVml43f.o
    cc tmp/xmlXPathInitVml43f.o -L/usr/lib64 -lxml2 -o a.out
    error: command 'gcc' failed with exit status 4

This is possibly because of the lack of memory. You can add swap file (abbreviated current path to copy easily)
sudo dd if=/dev/zero of=/swapfile bs=1024 count=524288
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
Then do again installing lxml as above. You can turn off the swap file.
sudo swapoff /swapfile


Install pandas

(myenv)[ec2-usr@ip-...]$ sudo /usr/bin/myenv/bin/pip3 install pandas

Install scikit-learn

(myenv)[ec2-usr@ip-...]$ sudo /usr/bin/myenv/bin/pip3 install scikit-learn

Install Jupyter

(myenv)[ec2-usr@ip-...]$ sudo /usr/bin/myenv/bin/pip3 install jupyter

Install flask

(myenv)[ec2-usr@ip-...]$ sudo /usr/bin/myenv/bin/pip3 install flask

Install jinja2

(myenv)[ec2-usr@ip-...]$ sudo /usr/bin/myenv/bin/pip3 install jinja2

Bibliography

2016-06-18

読書メーターに登録した読みたい本からISBNをPythonでスクレイピング (Scraping bookmeter.com with Python)

読書メーターから「読みたい本」のISBNコードをスクレイピングした。読みたい本として登録した一覧は下記のURLでアクセスできる。
http://bookmeter.com/u/<bookmeterID>/booklistpre&p=<int:page_number>
xpath使ってそれぞれのISBNにアクセスする。読書メーター側に負担をかけないよう、ページの読み込みごとに1秒待たせている。

# main code

def get_isbns(bookmeterID):
    """
    bookmeterID: Access your bookmeter page. Check the URL. You will see bookmeter.com/u/. Let number be an argument.
    The function returns a list of ISBNs
    """
    ISBN10s = []
    page=1
    while True:
        url = 'http://bookmeter.com/u/{id}/booklistpre&p={page}'.format(**{'id':bookmeterID, 'page':page})
        nextISBN10s = getISBNfromURL(url)
#         print(page, nextISBN10s)
        if nextISBN10s:
            ISBN10s.extend(nextISBN10s)
            page+=1
            time.sleep(1)
        else:
            break
    return ISBN10s

def getISBNfromURL(url):
    """
    used in get_isbns(bookmeterID)
    """
    page = requests.get(url)
    tree = lxml.html.fromstring(page.content)
    href_list = tree.xpath('//div[@class="book_box_book_title"]/a/@href')
    ISBN10 = [href[3:] for href in href_list if href.startswith('/b')]
    return ISBN10

Run/Execute python file on Apache2 with local server, ubuntu14.04LTS

Summary

 I tried to run my python3 scripts on my local server with apache2 and ubuntu14.04LTS. I had permission problems and addressed it.

Problem


My firefox shows

Internal Server Error

The server encountered an internal error or misconfiguration and was unable to complete your request.
Please contact the server administrator at webmaster@localhost to inform them of the time this error occurred, and the actions you performed just before this error.
More information about this error may be available in the server error log.

Apache/2.4.7 (Ubuntu) Server at localhost Port 80
 
 
also, in log file (/var/log/apache2/error.log), it says "Permission denied".

[Sun Jun 05 14:33:58.930599 2016] [core:notice] [pid 18049] AH00094: Command line: '/usr/sbin/apache2'
[Sun Jun 05 14:34:02.586809 2016] [cgi:error] [pid 18052] [client 127.0.0.1:38168] AH01215: (13)Permission denied: exec of '/var/www/html/cgi-bin/first.py' failed
[Sun Jun 05 14:34:02.588022 2016] [cgi:error] [pid 18052] [client 127.0.0.1:38168] End of script output before headers: first.py

Solution

Allow people to execute file.
sudo chmod 755 filename



Bibliography

Amazon Product API ItemLookup example with python3

This function returns a request about products. load_aws_keys() is not written here.

def get_amazon_xml(*asins):
    access_key, secret_key = load_aws_keys()
    verb = 'GET'
    endpoint = 'http://webservices.amazon.co.jp/onca/xml' # .co.jp in Japan / .com in US
    if access_key is None or secret_key is None:
        print('No access key is available.')
        sys.exit()
    asin = ','.join([str(i) for i in asins])
    query_dict = {'AWSAccessKeyId':access_key,
                 'AssociateTag':'asterisk37n-22',
                 'IdType':'ASIN',
                 'ItemId':asin,
                 'Operation':'ItemLookup',
                 'ResponseGroup':'Images,ItemAttributes,Offers',
                 'Service':'AWSECommerceService',
                 'Timestamp':datetime.datetime.utcnow().isoformat(),
                 'Version':'2013-08-01'}
    sorted_query = sorted(query_dict.items()) # order matters
    query_url = urlencode(sorted_query)
#     calcurate signature
    msg_HMAC_SHA256 = verb + '\n' + 'webservices.amazon.co.jp' + '\n' + '/onca/xml' +'\n' + query_url
    dig = hmac.new(secret_key.encode('utf-8'), msg=msg_HMAC_SHA256.encode('utf-8'), digestmod=hashlib.sha256).digest()
    signature = base64.b64encode(dig).decode()      # py3k-mode
    request_url = endpoint + '?' + query_url+'&Signature='+quote(signature)
    print('\nBEGIN REQUEST++++++++++++++++++++++++++++++++++++')
    print('Request URL = ' + request_url)
    r = requests.get(request_url)
    print('\nRESPONSE++++++++++++++++++++++++++++++++++++')
    print('Response code: %d\n' % r.status_code)
    print(r.text)
    return r 

2016-06-14

get xml's elements with namespace from Amazon Product API with lxml and Python3

You have to add namespace before every prefix like below.

import xml.etree.ElementTree as ET
root = ET.fromstring(xml) # prepare xml
nspace = '{http://webservices.amazon.com/AWSECommerceService/2013-08-01}'
for item in root.findall('.//'+nspace+'Items/'+nspace+'Item/'+nspace+'ItemAttributes/'+nspace+'Title'):
    print(item.text, item.attrib)

# or simpler way
for item in root.findall('.//ns:Items/ns:Item/ns:ItemAttributes/ns:Title', namespaces={'ns':'http://webservices.amazon.com/AWSECommerceService/2013-08-01'}):
    print(item.text, item.attrib)


Also, beware that findall() only finds direct children like parent.findall(children).
findall() returns a list, find() not.

cf)
XPath and XSLT with lxml

飲食店への商品持ち込みを批判する人への反論

持ち込み商品の反対意見について反反論がある。持ち込み商品については、それはマナー違反だからと言う人が多い。商品を持ち込んでいる客がいると、注意したところでバイト代の上がらないバイト、給料の上がらない正社員は注意をし、さらに周りの客でも注意をする人がいる。彼らにその理由を聞くと、「...