分类目录归档:Python

阿里云升级freebsd10.1到11.0-release-p8记录

阿里云升级freebsd10.1到11.0-release-p8记录

总体是一次顺利的升级体验!

主要用到的命令是:

查看版本

945  22:20   freebsd-version -k -u

设置环境变量,10.3版本,好像不需要弄这个:

946  22:20   setenv UNAME_r “10.3-RELEASE”

更新,这个在国内如果没有镜像的话时间超长!

 

为了装jupyter,使用了其它一下命令:

947  22:20   freebsd-update fetch

 

920  20:59   whereis jupyterhub

921  21:00   whereis jupyter

926  21:00   locate notejs

927  21:21   df

928  21:25   whereis nodejs

929  21:25   locate node

930  21:27   pkg install node

931  21:27   pkg-static install node

932  21:55   pkg_stick install python36

933  21:56   pkg-static install python36

934  22:02   whereis py-pyzmq

935  22:03   make PYTHON_VERSION=python3.6 install clean

936  22:03   cd /usr/ports/net/py-pyzmq

937  22:03   make PYTHON_VERSION=python3.6 install clean

938  22:07   pkg-static install py-setuptools36

939  22:07   pkg-static install devel/py-setuptools36

940  22:08   cd /usr/ports/net/py-pyzmq

943  22:09   make -V PYTHON_VERSION=python3.6 install clean

944  22:19   freebsd-version

945  22:20   freebsd-version -k -u

946  22:20   setenv UNAME_r “10.3-RELEASE”

后来知道,这里不需要设这个环境变量。

 

956  22:29   /usr/local/bin/python3.6

957  22:31   ps -aux

958  22:31   nameserver

959  22:32   nslookup

960  22:32   vi /etc/resolv.conf

 

983  22:36   freebsd-update fetch

984  22:36   freebsd-update fetch &

 

1000  22:43   freebsd-update -s ‘freebsd-updates.mirrors.163.com’ fetch

可惜163的镜像没有了。

参考的这篇文章:

升级FreeBSD 10.2-STABLE 到 11.0-RELEASE

https://bbs.aliyun.com/read/297189.html?spm=5176.bbsr296915.0.0.z411Uy

发现那篇文章中有误,

不过那个错误不影响大局,因为到了后面会有提示

freebsd-update fetch 之后,就可以先升级到10.3了

freebsd-update upgrade -r 10.3-RELEASE

 

root@iZ25alqsdzzZ:~ # freebsd-update upgrade -r 10.3-RELEASE

Looking up update.FreeBSD.org mirrors… 4 mirrors found.

Fetching metadata signature for 10.1-RELEASE from update5.freebsd.org… done.

Fetching metadata index… done.

Fetching 2 metadata files… done.

Inspecting system… done.

 

The following components of FreeBSD seem to be installed:

kernel/generic world/base world/lib32

 

The following components of FreeBSD do not seem to be installed:

src/src world/doc world/games

 

Does this look reasonable (y/n)? y

 

Fetching metadata signature for 10.3-RELEASE from update5.freebsd.org… done.

Fetching metadata index… done.

Fetching 1 metadata patches. done.

Applying metadata patches… done.

Fetching 1 metadata files…

done.

Inspecting system…

 

done.

Fetching files from 10.1-RELEASE for merging… done.

Preparing to download files… done.

Fetching 11045 patches…..10….20….30….40….50….60….70….80….90….100….110….120….130….140….150….160….170….180….190….200….210….220….230….240….250….260….270..

 

本来以为要3个小时呢,后来很快:

….10010….10020….10030….10040….10050……..11030….11040.. done.

Applying patches… done.

Fetching 393 files… done.

Attempting to automatically merge changes in files… done.

 

The following file could not be merged automatically: /etc/ntp.conf

Press Enter to edit this file in vi and resolve the conflicts

manually…

 

说/etc/ntp.conf无法自动合并,只能手工上!

 

回答了一大堆yes

然后运行安装:

/usr/sbin/freebsd-update install

 

root@iZ25alqsdzzZ:~ #/usr/sbin/freebsd-update install

Installing updates…

Kernel updates have been installed.  Please reboot and run

“/usr/sbin/freebsd-update install” again to finish installing updates.

 

一年多没重启了,重启一下

启动后看下:

root@rich:~ # freebsd-version -k -u

10.3-RELEASE-p11

10.1-RELEASE

 

Ok,成功从10.1升级到10.3,现在开始主版本升级,从10升级到11,输入如下命令:

# : > /usr/bin/bspatch

# freebsd-update upgrade -r 11.0-RELEASE

# freebsd-update install

<reboot the system>

# freebsd-update install

<rebuild third-party software>

# freebsd-update install

 

root@rich:~ # freebsd-update upgrade -r 11.0-RELEASE

src component not installed, skipped

Looking up update.FreeBSD.org mirrors… 4 mirrors found.

Fetching metadata signature for 10.3-RELEASE from update5.freebsd.org… done.

Fetching metadata index… done.

Fetching 1 metadata patches. done.

Applying metadata patches… done.

Fetching 1 metadata files… done.

Inspecting system… done.

 

The following components of FreeBSD seem to be installed:

kernel/generic world/base world/lib32

 

The following components of FreeBSD do not seem to be installed:

world/doc world/games

 

Does this look reasonable (y/n)? y

 

Fetching metadata signature for 11.0-RELEASE from update5.freebsd.org… done.

Fetching metadata index… done.

Fetching 1 metadata patches. done.

Applying metadata patches… done.

Fetching 1 metadata files… done.

Inspecting system… done.

Fetching files from 10.3-RELEASE for merging… done.

Preparing to download files… done.

Fetching 11218 patches…..10….20….30….40….50….60….70….80….90….100….110….120….130….140….150….160….170….180….190….200….210….220….230….240….250….260….270….280….290….300….310….320….330….340….350….360….370….380….390….400….410….420….430….440….450….460….470….480….490….500……….11200….11210…. done.

Applying patches… done.

Fetching 1645 files… done.

Attempting to automatically merge changes in files… done.

 

The following file could not be merged automatically: /etc/ntp.conf

Press Enter to edit this file in vi and resolve the conflicts

 

安装过程中,又出现ntp.conf,我直接确认退出,然后出现:

The following changes, which occurred between FreeBSD 10.3-RELEASE and

FreeBSD 11.0-RELEASE have been merged into /etc/group:

— current version

+++ new version

@@ -1,6 +1,6 @@

-# $FreeBSD: releng/10.3/etc/group 256366 2013-10-12 06:08:18Z rpaulo $

+# $FreeBSD: releng/11.0/etc/group 294896 2016-01-27 06:28:56Z araujo $

#

wheel:*:0:root,sky

daemon:*:1:

kmem:*:2:

sys:*:3:

@@ -15,10 +15,11 @@

staff:*:20:

sshd:*:22:

smmsp:*:25:

mailnull:*:26:

guest:*:31:

+video:*:44:

bind:*:53:

unbound:*:59:

proxy:*:62:

authpf:*:63:

_pflogd:*:64:

@@ -26,10 +27,11 @@

uucp:*:66:

dialer:*:68:

network:*:69:

audit:*:77:

www:*:80:

+_ypldap:*:160:

hast:*:845:

nogroup:*:65533:

nobody:*:65534:

mysql:*:88:

sky:*:1001:

Does this look reasonable (y/n)?

 

更新了一大堆东西,pkg这个是我真实目的,因为阿里云freebsd10.1的pkg挂了:

/usr/sbin/periodic

/usr/sbin/pkg

/usr/sbin/pmcannotate

 

最后出现:

/var/yp/Makefile.dist

To install the downloaded upgrades, run “/usr/sbin/freebsd-update install”.

 

按照提示运行

/usr/sbin/freebsd-update install

 

root@rich:~ #/usr/sbin/freebsd-update install

src component not installed, skipped

Installing updates…

Kernel updates have been installed.  Please reboot and run

“/usr/sbin/freebsd-update install” again to finish installing updates.

 

重启系统后,看一下:

root@rich:~ # uname -a

FreeBSD rich 11.0-RELEASE-p8 FreeBSD 11.0-RELEASE-p8 #0: Wed Feb 22 06:12:04 UTC 2017     root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC  amd64

root@rich:~ # freebsd-version -k -u

11.0-RELEASE-p8

10.3-RELEASE-p17

 

呵呵,已经是11啦!

再来一次/usr/sbin/freebsd-update install

root@rich:~ # /usr/sbin/freebsd-update install

src component not installed, skipped

Installing updates…

Completing this upgrade requires removing old shared object files.

Please rebuild all installed 3rd party software (e.g., programs

installed from the ports tree) and then run “/usr/sbin/freebsd-update install”

again to finish installing updates.

root@rich:~ #

 

这里让重新build所有的三方软件,天啊!

这个先不管它了

先看一下版本:

root@rich:~ # freebsd-version -k -u

11.0-RELEASE-p8

11.0-RELEASE-p8

 

嗯,这样就对了 !

 

总体来说,目前看算顺利,没有出什么内核无法启动,sshd无法启动,网站挂了等幺蛾子 !

python下大奖章量化接口 模拟交易测试成功

python3.4 大奖章量化接口1.10

先看看成功案例的样子:

Python 3.4.1 |Continuum Analytics, Inc.| (default, May 19 2014, 13:04:39) on Windows (32 bits).
This is the IEP interpreter with integrated event loop for PYSIDE.

Using IPython 2.1.0 — An enhanced Interactive Python.
?         -> Introduction and overview of IPython’s features.
%quickref -> Quick reference.
help      -> Python’s own help system.
object?   -> Details about ‘object’, use ‘object??’ for extra details.

 

In [3]:  from WindPy import *

In [4]: import datetime

In [5]: w.start()
Welcome to use Wind Quant API 1.0 for Python (WindPy)!
You can use w.menu to help yourself to create commands(WSD,WSS,WST,WSI,WSQ,…)!

COPYRIGHT (C) 2013 WIND HONGHUI INFORMATION & TECHKNOLEWDGE CO., LTD. ALL RIGHTS RESERVED.
IN NO CIRCUMSTANCE SHALL WIND BE RESPONSIBLE FOR ANY DAMAGES OR LOSSES CAUSED BY USING WIND QUANT API 1.0 FOR Python.
Out[5]:
.ErrorCode=0
.Data=[[[],[‘],[O],[K],[!],[‘],[]]]

In [6]: dct1=w.tlogon(“00000010″,”0″,”M:15853799XXX01″,”123456″,”SHSZ”)

 

In [9]: dct1.Data
Out[9]: [[2], [‘M:15853799XXX01′], [‘SZSHA’], [0], [”]]

 

In [11]: w.torder(“600030.SH”,”Buy”,”0″,”200″,”OrderType=B5TC;LogonID=2″)
Out[11]:
.ErrorCode=0
.Fields=[[,’,R,e,q,u,e,s,t,I,…]
.Data=[[[],[[],[5],[]],[,],[ ],[[],[‘],[6],[0],…]

In [12]: dct=w.torder(“600030.SH”,”Buy”,”0″,”200″,”OrderType=B5TC;LogonID=2″)

 

In [14]: dct.Data
Out[14]:
[[6],
[‘600030.SH’],
[‘1′],
[‘0′],
[‘200′],
[‘B5TC’],
[‘2′],
[0],
[‘Sending …’]]

In [15]:

然后查看模拟账户,发现400股中信证券静静的躺在那里:

证券代码 证券名称 证券数量 可卖数量 成本价 最新价 最新市值 浮动盈亏 盈亏率(%) 股东代码
600030 中信证券 400 0 359882 352500 14100 -295 -2.05% A000002215

回过头来讲讲苦难的历程,都说好事多磨,刚开始测试的时候报错:

In [16]: dct1=w.tlogon(“00000010″,”0″,”M:15853799xxx01″,”123456″,”SHSZ”)

 

In [17]: dct1.Data

Out[17]:

[[0],

[‘M:15853799XXX01′],

[‘SZSHA’],

[-40530102],

[‘200登录失败:资金账户M:15853799XXX01密码错误!’]]

出现这样的错误真的让人很无奈,根本不指导问题在哪里啊! 后来向大奖章求助,小疯猪提到了Wind密码问题,我在w.start()的时候一直用的用户名+密码,但是小疯猪提到的是手机号+密码,因此到了这里我就感觉有点眉目了。 后来重置了手机号+密码,在w.start()的时候,用手机号登陆,模拟交易的密码错误问题就解决了。

感谢大奖章,也希望上面的经历能帮助朋友们!

 

专门的python爬虫框架:scrapy

http://scrapy.org/

竟然有专门的爬虫框架,呵呵。

这里有中文例子:

http://www.cnblogs.com/txw1958/archive/2012/07/16/scrapy-tutorial.html

中文文档:

http://scrapy-chs.readthedocs.org/zh_CN/0.24/

Scrapy是一个为了爬取网站数据,提取结构性数据而编写的应用框架。 可以应用在包括数据挖掘,信息处理或存储历史数据等一系列的程序中。

其最初是为了 页面抓取 (更确切来说, 网络抓取 )所设计的, 也可以应用在获取API所返回的数据(例如 Amazon Associates Web Services ) 或者通用的网络爬虫。

当然,如果想更深入的解析,还是推荐使用lxml 完成那部分工作!

 

Requests: HTTP for Humans 更好使用的http python接口

http://cn.python-requests.org/zh_CN/latest/

在Python的世界里,事情不应该这么麻烦。

原来使用urllib,非常繁琐,现在改成Requests,非常简洁,符合人的操作习惯!

 

#!/usr/bin/env python
# -*- coding: utf-8 -*-

import requests

r = requests.get('https://api.github.com', auth=('user', 'pass'))

print r.status_code
print r.headers['content-type']

# ------
# 200
# 'application/json'

支持的功能
Requests 完全满足如今网络的需求。

国际化域名和 URLs
Keep-Alive & 连接池
持续性的 Cookie 会话
类浏览器式的 SSL 加密认证
基本/精简式的身份认证
优雅的键/值 Cookies
自动解压
Unicode 编码的响应主体
多段文件上传
连接超时
支持 .netrc
适用于 Python 2.6—3.3
安全的线程使用

 

 

pyramid url dispatch 学习

今天学习pyramid中增加路由

新创建工程:
pcreate -s starter mp
在不改变目录结构的情况下,
在__init__.py中添加:
config.add_route(‘test’,’/test/’)

在view.py文件中添加:
@view_config(route_name=’test’, renderer=’templates/mytemplate.pt’)
def test_view(request):
return {‘project': ‘mp of test’}
然后浏览器测试:http://a.zqfx.org:6543/test/

测试通过,学习结束。

碰到的问题是:
刚开始修改后,系统一直提示
The resource could not be found.

/test/

就好像没做修改一样。而事实上已经在__init__.py中加上了两句:
config.add_view(‘test_view’,route_name=’test’)
config.add_route(‘test’,’/test/’)

于是重启系统:
pserve mp/development.ini
发现报错,ImportError: No module named ‘test_view’
于是将原来写的这句注释掉: config.add_view(‘test_view’,route_name=’test’)

再启动,就ok了

另外为什么自己修改之后系统没显示呢? 我认为是启动时没有加上–reload
也就是学习时应该这样启动:
pserve mp/development.ini –reload

 

下午再进行测试的时候,发现http://a.zqfx.org:6543/test/ 也是404,这可太奇怪了,因为上午是好的呀!

后来检查代码,才发现:

@view_config(route_name=’test’, renderer=’templates/mytemplate.pt’)
def test_view(request):
return {‘project': ‘mp of test’}

@view_config(route_name=’test2′, renderer=’templates/mytemplate.pt’)
def test_view(request):
return {‘project': ‘mp of test’}

写了两个test_view,结果导致”test”的路由没有生效,于是修改:

@view_config(route_name=’test’, renderer=’templates/mytemplate.pt’)
def test_view(request):
return {‘project': ‘mp of test’}

@view_config(route_name=’test2′, renderer=’templates/mytemplate.pt’)
def test_view2(request):
return {‘project': ‘mp of test’}

再测试,test页面ok了。

python3.4 pyramid笔记

从python3.3开始,python内建支持easy_install和类似virtualenv的pyvenv,所以安装要比以前节省很多时间和步骤,比较快的安装方法是:

1 下载python3.4源代码,解压,进入该目录,configure 然后make install (我注意到是自动clean的)

wget https://www.python.org/ftp/python/3.4.2/Python-3.4.2.tgz

tar -xzvf Python-3.4.2.tgz

cd Python-3.4.2

./configure –prefix=$HOME/opt/py34
make install

这个步骤完成后,就已经有了easy_install、pip和pyvenv了

2 用pyvenv建立虚拟环境,然后激活该env,然后安装pyramid就ok了

注意我是将python安装到opt/py34目录里了。

在用户根目录下bash#:

opt/py34/bin/pyvenv env3
source env3/bin/activate
easy_install pyramid

需要注意的是bash acitvate没问题,tcsh则很难activate 。 另现在建议用pip安装,即:

pip install pyramid

安装中碰到的问题:因为是裸机器,没有spenssl,所以一开始easy_install那块是装不上,直接yum install openssl

后来有次安装,pip那里报错pip Ignoring ensurepip failure: pip 1.5.6 requires SSL/TLS,装了openssl也不行,需要 yum install openssl-devel

ps,正在用阿里云,速度果然比国外快啊!

总体而言,我建议root用户先装一个py3,然后进入个人账户创建env虚拟环境,再安装pyramid等。

或者就在个人账户里装py3,再创建env,然后安装pyramid等。

 

用scikits-learn对沪深300成份股做聚类分析

代码:

import datetime
import numpy
import sklearn.cluster
from matplotlib import finance
#1. Download price data

# 2011 to 2012
start = datetime.date(2013,5,15)
end = datetime.date(2014,5,15)
a='''0000001
0000002
0000009
0000012
0000024
0000039
0000046
0000060
0000061
0000063
0000069
0000100
0000156
0000157
0000333
0000338
0000400
0000401
0000402
0000423
0000425
0000528
0000536
0000538
0000562
0000568
0000581
0000596
0000598
0000623
0000625
0000629
0000630
0000651
0000656
0000686
0000703
0000709
0000718
0000725
0000728
0000729
0000750
0000758
0000768
0000776
0000778
0000783
0000792
0000793
0000800
0000826
0000831
0000839
0000858
0000869
0000876
0000878
0000895
0000933
0000937
0000960
0000961
0000963
0000970
0000983
0000999
0002001
0002007
0002024
0002038
0002051
0002065
0002069
0002081
0002106
0002129
0002142
0002146
0002155
0002202
0002230
0002236
0002241
0002269
0002294
0002299
0002304
0002310
0002344
0002353
0002375
0002385
0002399
0002415
0002422
0002431
0002450
0002456
0002500
0002570
0002594
0002603
0002653
0002673
1600000
1600008
1600009
1600010
1600011
1600015
1600016
1600018
1600019
1600027
1600028
1600029
1600030
1600031
1600036
1600048
1600050
1600058
1600060
1600062
1600066
1600068
1600079
1600085
1600089
1600096
1600100
1600104
1600108
1600109
1600111
1600115
1600118
1600123
1600143
1600150
1600153
1600157
1600160
1600166
1600170
1600177
1600188
1600196
1600208
1600216
1600219
1600221
1600252
1600256
1600259
1600266
1600267
1600271
1600276
1600309
1600315
1600316
1600332
1600340
1600348
1600352
1600362
1600369
1600372
1600376
1600383
1600395
1600403
1600406
1600415
1600436
1600489
1600497
1600498
1600516
1600518
1600519
1600528
1600535
1600546
1600547
1600549
1600582
1600583
1600585
1600588
1600597
1600598
1600600
1600637
1600642
1600648
1600649
1600655
1600660
1600663
1600664
1600674
1600688
1600690
1600694
1600703
1600705
1600718
1600739
1600741
1600783
1600795
1600804
1600809
1600811
1600827
1600832
1600837
1600839
1600859
1600863
1600873
1600875
1600886
1600887
1600893
1600895
1600900
1600970
1600971
1600997
1600999
1601001
1601006
1601009
1601018
1601088
1601098
1601099
1601101
1601106
1601111
1601117
1601118
1601139
1601158
1601166
1601168
1601169
1601186
1601231
1601238
1601258
1601288
1601299
1601318
1601328
1601333
1601336
1601369
1601377
1601390
1601398
1601555
1601600
1601601
1601607
1601618
1601628
1601633
1601666
1601668
1601669
1601688
1601699
1601717
1601766
1601800
1601808
1601818
1601857
1601866
1601888
1601898
1601899
1601901
1601918
1601928
1601933
1601939
1601958
1601988
1601989
1601991
1601992
1601998
1603000
1603993'''
#print(a)
aaa='''0000001
0000002
0000009
0000012
0000024
0000039
0000046
0000060
0000061
0000063
0000069
0000100
0000156
0000157
0000333
0000338
0000400
0000401
0000402
0000423
0000425
0000528
0000536
0000538
0000562
0000568
0000581
0000596
0000598
0000623
0000625
0000629
0000630
0000651
0000656
0000686
0000703
0000709
0000718
0000725
0000728
0000729
0000750
0000758
0000768
0000776
0000778
0000783
0000792
0000793
0000800
0000826
0000831
0000839
0000858
0000869
0000876
0000878
0000895
0000933
0000937
0000960
0000961
0000963
0000970
0000983
0000999
0002001
0002007
0002024
0002038
0002051
0002065
0002069
0002081
0002106
0002129
0002142
0002146
0002155
0002202
0002230
0002236
0002241
0002269
0002294
0002299
0002304
0002310
0002344
0002353
0002375
0002385
0002399
0002415
0002422
0002431
0002450
0002456
0002500
0002570
0002594
0002603
0002653
0002673'''

bbb="""1600000
1600008
1600009
1600010
1600011
1600015
1600016
1600018
1600019
1600027
1600028
1600029
1600030
1600031
1600036
1600048
1600050
1600058
1600060
1600062
1600066
1600068
1600079
1600085
1600089
1600096
1600100
1600104
1600108
1600109
1600111
1600115
1600118
1600123
1600143
1600150
1600153
1600157
1600160
1600166
1600170
1600177
1600188
1600196
1600208
1600216
1600219
1600221
1600252
1600256
1600259
1600266
1600267
1600271
1600276
1600309
1600315
1600316
1600332
1600340
1600348
1600352
1600362
1600369
1600372
1600376
1600383
1600395
1600403
1600406
1600415
1600436
1600489
1600497
1600498
1600516
1600518
1600519
1600528
1600535
1600546
1600547
1600549
1600582
1600583
1600585
1600588
1600597
1600598
1600600
1600637
1600642
1600648
1600649
1600655
1600660
1600663
1600664
1600674
1600688
1600690
1600694
1600703
1600705
1600718
1600739
1600741
1600783
1600795
1600804
1600809
1600811
1600827
1600832
1600837
1600839
1600859
1600863
1600873
1600875
1600886
1600887
1600893
1600895
1600900
1600970
1600971
1600997
1600999
1601001
1601006
1601009
1601018
1601088
1601098
1601099
1601101
1601106
1601111
1601117
1601118
1601139
1601158
1601166
1601168
1601169
1601186
1601231
1601238
1601258
1601288
1601299
1601318
1601328
1601333
1601336
1601369
1601377
1601390
1601398
1601555
1601600
1601601
1601607
1601618
1601628
1601633
1601666
1601668
1601669
1601688
1601699
1601717
1601766
1601800
1601808
1601818
1601857
1601866
1601888
1601898
1601899
1601901
1601918
1601928
1601933
1601939
1601958
1601988
1601989
1601991
1601992
1601998
1603000
1603993"""
ccc="""1600376
1600383
1600395
1600403
1600406
1600415
1600436
1600489
1600497
1600498
1600516
1600518
1600519
1600528
1600535
1600546
1600547
1600549
1600582
1600583
1600585
1600588
1600597
1600598
1600600
1600637
1600642
1600648
1600649
1600655
1600660
1600663
1600664
1600674
1600688
1600690
1600694
1600703
1600705
1600718
1600739
1600741
1600783
1600795
1600804
1600809
1600811
1600827
1600832
1600837
1600839
1600859
1600863
1600873
1600875
1600886
1600887
1600893
1600895
1600900
1600970
1600971
1600997
1600999"""
b=aaa.split('n')
#print(b)
c=[]
#print(b)
for x in b:
    #print (x,x[0],x[1:7])
    if x[0]=='0':
        temp=x[1:] + ".SZ"
        #print(temp)
        c.append(temp)
    if x[0]=="1":
        temp=x[1:]+".SS"
        c.append (temp)
#print(c)
#Dow Jones symbols
symbols = c

quotes = [finance.quotes_historical_yahoo(symbol, start, end, asobject=True)
          for symbol in symbols]

close = numpy.array([q.close for q in quotes]).astype(numpy.float)
print (close.shape)

#2. Calculate affinity matrix
logreturns = numpy.diff(numpy.log(close))
print (logreturns.shape)

logreturns_norms = numpy.sum(logreturns ** 2, axis=1)
S = - logreturns_norms[:, numpy.newaxis] - logreturns_norms[numpy.newaxis, :] + 2 * numpy.dot(logreturns, logreturns.T)

#3. Cluster using affinity propagation
aff_pro = sklearn.cluster.AffinityPropagation().fit(S)
labels = aff_pro.labels_
print(labels)
for i in range(len(labels)):
    print ('%s in Cluster %d' % (symbols[i], labels[i]))
temp=[]

输出:

[ 0  2 13  9  7  9  9  2 16 13  2  2  1  2  3  2 13  7  9  9  2  2  4  4  5
  9 10 16  9  2 13  2  2  9  7  6 13  2  7  9  2  2  9  2  9  2  2  2  2 16
 13  4 16  9  2  2  9  2  4  2  2  9  7  4  8  2  4  9  9 10 11 16 16 13 12
 13 10  2  7 16 13 14  4 15 16  4 13  9 17 18 11 16 19  9 20  9 13 13 21  2
 22 16 11 10  2]
000001.SZ in Cluster 0
000002.SZ in Cluster 2
000009.SZ in Cluster 13
000012.SZ in Cluster 9
000024.SZ in Cluster 7
000039.SZ in Cluster 9
000046.SZ in Cluster 9
000060.SZ in Cluster 2
000061.SZ in Cluster 16
000063.SZ in Cluster 13
000069.SZ in Cluster 2
000100.SZ in Cluster 2
000156.SZ in Cluster 1
000157.SZ in Cluster 2
000333.SZ in Cluster 3
000338.SZ in Cluster 2
000400.SZ in Cluster 13
000401.SZ in Cluster 7
000402.SZ in Cluster 9
000423.SZ in Cluster 9
000425.SZ in Cluster 2
000528.SZ in Cluster 2
000536.SZ in Cluster 4
000538.SZ in Cluster 4
000562.SZ in Cluster 5
000568.SZ in Cluster 9
000581.SZ in Cluster 10
000596.SZ in Cluster 16
000598.SZ in Cluster 9
000623.SZ in Cluster 2
000625.SZ in Cluster 13
000629.SZ in Cluster 2
000630.SZ in Cluster 2
000651.SZ in Cluster 9
000656.SZ in Cluster 7
000686.SZ in Cluster 6
000703.SZ in Cluster 13
000709.SZ in Cluster 2
000718.SZ in Cluster 7
000725.SZ in Cluster 9
000728.SZ in Cluster 2
000729.SZ in Cluster 2
000750.SZ in Cluster 9
000758.SZ in Cluster 2
000768.SZ in Cluster 9
000776.SZ in Cluster 2
000778.SZ in Cluster 2
000783.SZ in Cluster 2
000792.SZ in Cluster 2
000793.SZ in Cluster 16
000800.SZ in Cluster 13
000826.SZ in Cluster 4
000831.SZ in Cluster 16
000839.SZ in Cluster 9
000858.SZ in Cluster 2
000869.SZ in Cluster 2
000876.SZ in Cluster 9
000878.SZ in Cluster 2
000895.SZ in Cluster 4
000933.SZ in Cluster 2
000937.SZ in Cluster 2
000960.SZ in Cluster 9
000961.SZ in Cluster 7
000963.SZ in Cluster 4
000970.SZ in Cluster 8
000983.SZ in Cluster 2
000999.SZ in Cluster 4
002001.SZ in Cluster 9
002007.SZ in Cluster 9
002024.SZ in Cluster 10
002038.SZ in Cluster 11
002051.SZ in Cluster 16
002065.SZ in Cluster 16
002069.SZ in Cluster 13
002081.SZ in Cluster 12
002106.SZ in Cluster 13
002129.SZ in Cluster 10
002142.SZ in Cluster 2
002146.SZ in Cluster 7
002155.SZ in Cluster 16
002202.SZ in Cluster 13
002230.SZ in Cluster 14
002236.SZ in Cluster 4
002241.SZ in Cluster 15
002269.SZ in Cluster 16
002294.SZ in Cluster 4
002299.SZ in Cluster 13
002304.SZ in Cluster 9
002310.SZ in Cluster 17
002344.SZ in Cluster 18
002353.SZ in Cluster 11
002375.SZ in Cluster 16
002385.SZ in Cluster 19
002399.SZ in Cluster 9
002415.SZ in Cluster 20
002422.SZ in Cluster 9
002431.SZ in Cluster 13
002450.SZ in Cluster 13
002456.SZ in Cluster 21
002500.SZ in Cluster 2
002570.SZ in Cluster 22
002594.SZ in Cluster 16
002603.SZ in Cluster 11
002653.SZ in Cluster 10
002673.SZ in Cluster 2

现在的问题是:

1 不知道为什么,沪深300 的时候出错,而且是报错1600887 问题这个应该已经被转成600887.SS了啊! 单独换成上海的100多只股票,也是出错

2 对股票数太多的情况,应该修改输出形式,即一个Cluster 后面跟上所属的所有股票代码!

 

 

统计历史上某天的当日涨跌幅度

代码:

from datetime import date
import datetime
date1 = datetime.date( 2014, 4,28 )
date2 = datetime.date( 2014,5,4 )
#print(date1)
#rint(type(date1))
testd=datetime.date(1990,12,19)
testd=datetime.date(testd.year,testd.month,testd.day+1)
#print(testd)
lista=[]

# for line in open("D:/快盘/work/python/SH999999.csv"):
#     year1,so1,sh1,sl1,sc1,aa1,bbbb = line.split(",")
#     test=(year1,so1,sh1,sl1,sc1)
#     #print(test)
#     if year1==str(testd) or year1==str(date1):
#         pass
#         #print(year1,test)
da512=[]
for line in open("D:/快盘/work/python/SH999999.csv"):
    year1,so1,sh1,sl1,sc1,aa1,bbbb = line.split(",")
    test=(year1,so1,sh1,sl1,sc1,aa1,bbbb)
    #print(test)
    ii=0
    if year1.split('-')[1]=='05' and year1.split('-')[2]=='12':
        ii=ii+1
        #print(test)
        da512.append(test)

zs=0.0
zf=0.0
print(type(zs),type(zf))
for ld in da512:
#     if zs

 

输出:

>>> (executing lines 1 to 43 of “上证数据读.py”)

<class ‘float’> <class ‘float’>

1992-05-12 当日涨幅0.74%

1993-05-12 当日涨幅0.15%

1994-05-12 当日涨幅-0.68%

1995-05-12 当日涨幅2.72%

1997-05-12 当日涨幅0.64%

1998-05-12 当日涨幅-0.69%

1999-05-12 当日涨幅0.30%

2000-05-12 当日涨幅-0.30%

2003-05-12 当日涨幅0.58%

2004-05-12 当日涨幅2.25%

2005-05-12 当日涨幅-1.62%

2006-05-12 当日涨幅4.56%

2008-05-12 当日涨幅2.21%

2009-05-12 当日涨幅2.18%

2010-05-12 当日涨幅1.05%

2011-05-12 当日涨幅-0.73%

 

编程中一度想加上当日对前一日收盘价的涨跌,但是思路错了,变成了对去年的收盘价的涨跌了,同时编程中还碰到了一系列的问题:

1 浮点数==0问题,后来是用两种方法解决: int(浮点数)==0 或者 浮点数< 10 来解决。因为指数不会遇到< 10 的情况

2 最主要的就是逻辑错了,对前一日的涨跌,需要得到前一日的收盘价,这一点在yahoo上比较容易,可以直接取,在通达信数据里,则需要费点周折:

1) 把历史数据全部补上,休息日按无交易日处理

2) 取中间量,顺序遍历的时候,顺便用来存取前一日的收盘价。

3) 直接逻辑处理后,拿前一天的数据!

使用pyzo过程中,碰到输入时光标消失的问题,很讨厌

另外还碰到自动写的问题,比如输入abc,结果变成abcabc。。。。

通达信导出日K线数据测试

yahoo的上证指数日线数据有问题,没法用,所以转而选择通达信的数据。

刚开始还费力去编程转换通达信的存盘数据,后来发现,它竟然有导出功能,直接就能导出从1990.12.19日开始的日K线数据,真不错!

我倒出了csv格式,这样python处理起来就更简单了!选择的日线数据,精确复权(指数这里非必要,但个股一定要),文件名xxxxxx.csv,用逗号分隔,“生成导出头部”不勾选,日期格式yyyy-mm-dd .

导出后就是这样的格式:

2014-5-6,2024.26,2038.7,2021.49,2028.04,74609411,57548111872

2014-5-7,2023.15,2024.63,2008.45,2010.08,74360192,57558052864

2014-5-8,2006.85,2036.94,2005.69,2015.27,77865393,59529367552

2014-5-9,2016.5,2020.45,2011.78,2017.67,17104686,12630095872

分别是:日期,开盘,最高,最低,收盘,成交量,成交金额 (注意因为是今天导出,还没盘后维护,5.9日的数据只是盘中数据)

然后编程进行读取测试:

from datetime import date
import datetime
date1 = datetime.date( 2014, 4,28 )
date2 = datetime.date( 2014,5,4 )
#print(date1)
#rint(type(date1))
testd=datetime.date(1990,12,19)
testd=datetime.date(testd.year,testd.month,testd.day+1)
#print(testd)
lista=[]

for line in open("D:/快盘/work/python/SH999999.csv"):
    year1,so1,sh1,sl1,sc1,aa1,bbbb = line.split(",")
    test=(year1,so1,sh1,sl1,sc1)
    #print(test)
    if year1==str(testd) or year1==str(date1):
        print(year1)

输出:

>>> (executing lines 1 to 17 of “上证数据读.py”)

1990-12-20

2014-04-28

这就表明读取成功了!
但是中间也有波折,一直提示:

Traceback (most recent call last):

File “D:快盘workpython上证数据读.py”, line 13, in <module>

year1,so1,sh1,sl1,sc1,aa1,bbbb = line.split(“,”)

ValueError: need more than 1 value to unpack

后来通过debug,发现读完5.9日的数据之后才报错,仔细看,才发现数据文件最后一行写着:“数据来源通达信”

原来就是因为这句话报错的啊!浪费了我不少的时间…..

autopy初探

想让python帮自己干点力气活,于是找到了autopy

AutoPy Introduction and Tutorial

还有这篇:

用Python制作游戏外挂

 

这就是一个移动鼠标的例子:

import autopy
import math
import time
import random

TWO_PI = math.pi * 2.0
def sine_mouse_wave():
    """
    Moves the mouse in a sine wave from the left edge of
    the screen to the right.
    """
    width, height = autopy.screen.get_size()
    print(width,height)
    height /= 2
    height -= 10 # Stay in the screen bounds.

    for x in xrange(width):
        y = int(height * math.sin((TWO_PI * x) / width) + height)
        autopy.mouse.move(x, y)
        time.sleep(random.uniform(0.001, 0.003))

sine_mouse_wave()
autopy.mouse.smooth_move(1280-1,1024-1)

我的屏幕是1280×1024的。

 

晚上又看了一下,按键方面主要有如下几个可能的应用:

key.type_string(string, wpm=0)

例子:>>> autopy.key.type_string(“hello”,120)

 

key.tap(key, [modifiers])

Convenience wrapper around toggle() that holds down and then releases the given key and modifiers.

例子:>>> key.tap(‘a’)

>>> key.tap(key.K_META)

 

鼠标操作:

mouse.get_pos()

Returns a tuple (x, y) of the current mouse position.

例子:

>>> mouse.get_pos()
(688, 775)

移动鼠标:

mouse.move(x, y)

Moves the mouse to the given (x, y) coordinate.

Exceptions:
  • ValueError is thrown if the point is out of index.

mouse.smooth_move(x, y)

Smoothly moves the mouse to the given (x, y) coordinate in a straight line.

Exceptions:
  • ValueError is thrown if the point is out of index.

上面那个程序就是移动鼠标的例子

 

鼠标点击:mouse.click(button=LEFT_BUTTON)

Convenience wrapper around toggle() that holds down and then releases the given mouse button.

例子:

>>> mouse.click()

 

screen.get_size()

Returns a tuple (width, height) of the size of the main screen.

例子:

>>> screen.get_size()
(1440, 900)

 

screen.get_color(x, y)

Returns hexadecimal value describing the RGB color at the given point.

例子:

>>> screen.get_color(0,0)
15132899

>>> color.hex_to_rgb(15132899)
(230, 232, 227)

 

在系统中可以直接help:help(autopy) >>> help(autopy.bitmap.Bitmap)

图片操作方面:

To save the screen capture to a file, we can use:

>>> import autopy
>>> autopy.bitmap.capture_screen().save('screengrab.png')
>>> import autopy
>>> autopy.bitmap.Bitmap.open('i-am-a-monkey-and-i-like-it.png')
<Bitmap object at 0x1001d5378>

http://www.autopy.org/documentation/api-reference/