在centos7中分布式部署pyspider

1.搭建环境：

系统版本：Linux centos-linux.shared 3.10.0-123.el7.x86_64 #1 SMP Mon Jun 30 12:09:22 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

python版本：Python 3.5.1

1.1.搭建python3环境：

本人在尝试过后选择集成环境Anaconda

1.1.1.编译

# 下载依赖yum install -y ncurses-devel openssl openssl-devel zlib-devel gcc make glibc-devel libffi-devel glibc-static glibc-utils sqlite-devel readline-devel tk-devel gdbm-devel db4-devel libpcap-devel xz-deve# 下载python版本wget https://www.python.org/ftp/python/3.5.1/Python-3.5.1.tgz# 或者使用国内源wget http://mirrors.sohu.com/python/3.5.1/Python-3.5.1.tgzmv Python-3.5.1.tgz /usr/local/src;cd /usr/local/src# 解压tar -zxf Python-3.5.1.tgz;cd Python-3.5.1# 编译安装./configure --prefix=/usr/local/python3.5 --enable-sharedmake && make install# 建立软链接ln -s /usr/local/python3.5/bin/python3 /usr/bin/python3echo "/usr/local/python3.5/lib" > /etc/ld.so.conf.d/python3.5.confldconfig# 验证python3python3# Python 3.5.1 (default, Oct 9 2016, 11:44:24)# [GCC 4.8.5 20150623 (Red Hat 4.8.5-4)] on linux# Type "help", "copyright", "credits" or "license" for more information.# >>># pip/usr/local/python3.5/bin/pip3 install --upgrade pipln -s /usr/local/python3.5/bin/pip /usr/bin/pip# 本人在安装时出现问题 将pip重装wget https://bootstrap.pypa.io/get-pip.py --no-check-certificatepython get-pip.py

1.1.2.集成环境anaconda

# 集成环境anaconda(推荐)wget https://repo.continuum.io/archive/Anaconda3-4.2.0-Linux-x86_64.sh# 直接安装即可./Anaconda3-4.2.0-Linux-x86_64.sh# 若出错，可能是解压失败yum install bzip2

1.2.安装mariaDB

# 安装yum -y install mariadb mariadb-server# 启动systemctl start mariadb# 设置为开机启动systemctl enable mariadb# 配置密码 默认为空mysql_secure_installation# 登录mysql -u root -p# 创建一个用户 自己设定账户密码CREATE USER 'user_name'@'localhost' IDENTIFIED BY 'user_pass';GRANT ALL PRIVILEGES ON *.* TO 'user_name'@'localhost' WITH GRANT OPTION;CREATE USER 'user_name'@'%' IDENTIFIED BY 'user_pass';GRANT ALL PRIVILEGES ON *.* TO 'user_name'@'%' WITH GRANT OPTION;

1.3.安装pyspider

本人使用Anaconda

# 搭建虚拟环境sbird python版本3.*conda create -n sbird python=3*# 进入环境source activate sbird# 安装pyspiderpip install pyspider# 报错 # it does not exist. The exported locale is "en_US.UTF-8" but it is not supported# 执行 可写入.bashrcexport LC_ALL=en_US.utf-8export LANG=en_US.utf-8#ImportError: pycurl: libcurl link-time version (7.29.0) is older than compile-time version (7.49.0)conda install pycurl# 退出source deactivate sbird# 若在虚拟机内 出现无法访问localhost:5000 可关闭防火墙systemctl stop firewalld.service#########直接运行源码==============mkdir git;cd git# 下载git clone https://github.com/binux/pyspider.git# 安装/root/anaconda3/envs/sbird/bin/python /root/git/pyspider/run.py